Tomaso Poggio is dedicated to neuroscience, convolutional neural networks, and machine learning …
Original Article: here
… , specifically deep and reinforcement learning. Poggio presented his ideas at the Machine Learning Prague 2019 conference in Solving the 3 Main Puzzles of Deep Learning. It has inspired us to talk about neural networks, the principle of the site or autonomous means of transport.
Tomaso Poggio is a recognized neuroscience expert. What do you think is his greatest asset?
Jaroslav: To begin with, Poggio has great experience and is one of the most cited scientists in the field of computational neuroscience, so he can compare the shift in this area. For example, it compared the progress of autonomous cars in the early 1990s and now. However, the real break with neural networks occurred when multi-layer work began. Although we are still talking about old known neural networks, only grouped in layers, we now call them deep neural networks.
Jiří: The big shift compared to the past is mainly in computing power. If they had it at that time, they’d probably be able to do what we did today.
Jaroslav: This is, of course, pure speculation (laughs).
Zdeňka: In the 1980s, it has been proven in the neural network area that it is enough to work with three layers: one more hidden layer is added to the input and output layers. Consequently, everyone experimented with this theory. But it did not work very well, so most researchers for a while have lost time to neural networks – until recently, when they began to have fantastic results from a deep neural network. Those who persisted in the classical neural networks despite their temporary unpopularity were rewarded this year with the prestigious Turing Award, known as the Nobel Prize for Informatics. It’s just to the edge. One of Poggi’s interesting results is that he showed why and under what conditions neural networks work in layers. Its explanation is that when we use one layer, exponential growth occurs based on the number of input dimensions, such as the number of pixels in the image. Thus, the number of input dimensions of a neural network processing 250 × 250 pixels will be 62,500. We can thus easily reach a K constant of 62,500. This would mean that the number of neurons in the hidden layer exceeds the number of atoms in the universe. Which is not feasible. Learning Functions: When Is Deep Better Than Shallow of 2016 showed that when layers are stacked, it grows linearly – not K to 62,500, but K × 62,500. Therefore, fewer parameters are enough to learn. The reason for this is that neural networks are actually compositional functions.
Jiří: It is mainly about the principle of the locality, where the neurons are between neighboring neurons. Therefore, not everything is connected to everything and there are not many parameters because the scales are connected locally. It could be said that it is to a certain extent similar to the biological structure of the brain, where the internal structures are also connected locally. Jaroslav: Poggio is very influenced by working in The Center for Brains, Minds and Machines, where he tries to combine research into real neural networks of humans and animals with artificial neural networks. He claims that one example is enough to learn. He argued that we did not show the child one picture a million times. But we do not fully agree with this view of the matter.
Zdeňka: Yes, but I will return to the original question of what makes Poggio special. Most experts are content with doing something, but he is looking for why it works mathematically. His next area of exploration is computational neuroscience (computational neuroscience). Hence, it seeks to understand how and why artificial neural networks work and to show, for example, how the human brain works by mathematical simulations.
What is computational neuroscience dedicated to?
Jaroslav: We could simply say that an artificial neural network is inspired by how the neuron works. Convolution networks are in turn inspired by the visual cortex. However, we need to understand that we are only talking about inspiration. As limited artificial neural networks work, it proves that it is not enough to have sufficient computing power and connect it to these networks to get brain or visual cortex. This is by no means the case. It is entirely about inspiration, we must not forget it. When we use terms like neurons or neural networks, we don’t mean that they work just like real ones.
Jiri: The inspiration is that neurons are also connected in some way, but that ends. Processing in the brain is completely different.
Zdeňka: Computational Neuroscience fully respects this and tries to simulate real neural networks (ie the brain). But I will return to the compositional functions: Poggio claims that the brain, but also the whole world, works in a compositional way. Some areas of machine learning, such as forecasting financial market developments, may be an exception. For them, human brain-inspired machine learning methods may not be as appropriate as for the problems solving the same thing as the human brain – for example, the aforementioned image recognition.
Jiri: Actually, we people live locally in the world, we do not have the opportunity to get at once, jumping for example from Brno to New York. Financial markets may not be compositional because they operate globally, the world is globalized and what happens will affect everything around.
Poggio, respectively. his students dealt with the subject of autonomous means of transport in detail. What was the shift in this area? When will we ride in self-driving cars?
Jaroslav: Of course we do not know. Autonomous means of transport are at a very high level: cars have infrared cameras, lidars, can accurately detect characters even under difficult conditions … The case when a car has run over humans has only happened because some systems have been shut down. The tested autonomous cars have very sensitive sensors, and because of this, the car has often stopped itself, so they shut it down. The machine itself, with the systems fully switched on, would not pass anyone, rather it caused a person in the car who was not paying attention.
Zdeňka: The second thing is that at the moment it is not so much in the autonomous management as in the management assistance, so the human factor plays an important role. Of course, one can very easily start to rely 100% on a machine, even if the machine is designed to support driving.
Jaroslav: The current trend is not that we buy a car that drives itself. The direction of thinking about autonomous means of transport has changed a bit. New taxi service is now being tested in very limited traffic. When I say limited, I think in an environment where it is not raining, it is not snowing, there are no multi-lane roads, it does not turn left and so on.
Jiri: We must not forget the psychological aspect where people can be afraid to ride in a vehicle without a driver or a pilot. This is customary, and I believe that the generation of our children or grandchildren, who will grow up in a different environment, will feel completely at ease.