One of the good source of inspiration to design a neural network is to study how nature built the human neocortex. The neocortex is about 30cm x 30cm if it was to be unfolded in a flat surface. 2.5mm thick. Nature apparently folded the cortex into many circumvolutions so that it fits inside the skull. Six layers could be observed in the cortex, with one layer containing only communication lines and no neurons.
When looking at the neocortex, its structure appears to be the same everywhere, whether in the language area, the memory, the motor control, or any of the seven senses of the humans. Being the same everywhere is probably the simplest way nature found of increasing our far ancestors capabilities. This has a huge implications. Being identical everywhere means that nature found an universal algorithm good for all the senses, including memory and motor control. One of the objectives of ‘E’ is to find different types of central algorithm able to do all the things human are able to do with its neocortex.
It is believed that some other brain structures are important for learning. The hypothalamus for example might be where delay lines are created for temporal events. The hypothalamus is also essential to long term memory.
Differences between the most popular neural network research and what the nature was able to create: