Big Data

Deep Learning: The third neural network wave

DSCT Blog - February 2016 | Eric Postma - Cognitive Science and Artificial Intelligence group; TiCC, TSH


Facebook makes great strides in automatically recognising the contents of images. Thanks to a technique called deep learning, their systems are able to recognise the faces of your friends depicted on your account.

The data science revolution is partly fuelled by the third neural network wave. Neural networks (a shorthand for “artificial neural networks”) are learning algorithms that are coarsely modelled after neurons, the building blocks of the brain, and its adaptive connections. Although the name “neural networks” is merely a metaphor (biological neural networks are much more complex than their artificial abstractions), the latest wave of neural network research, called deep learning, caused impressive advances in image and speech recognition. Data Science research at Tilburg University benefits from these advances. In what follows, I briefly review the history of neural networks.

The first wave

In 1958, the psychologist Frank Rosenblatt proposed one of the first artificial neural networks, called the Perceptron. Rosenblatt developed a hardware implementation (the Mark I Perceptron), which consisted of a matrix of light sensitive sensors that were connected through adaptive connections (artificial synapses) to a threshold unit (artificial neuron). By providing the Perceptron with labelled example images, it could train itself to distinguish between two classes of simple images. Trained on a triangle-detection task with images of triangles and squares, the threshold unit would become active (“1”) in case of a triangle and inactive (“0”) in case of a square. The Perceptron gave rise to overoptimistic expectations about the ability of self-learning automata. On 8 July of 1958, the New York Times reported on the Mark I Perceptron that “The Navy revealed the embryo of an electronic computer today that it expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.”

The enthusiasm for the Perceptron waned quickly when the late Marvin Minsky  (together with Seymour Papert) published a devastating mathematical analysis of Perceptrons. One of the main limitations of the Perceptron was that learning could only be done for a single layer of adaptive connections. Minsky and Papert showed that the classification capacity of a single-layer Perceptron was severely limited. Stacking multiple Perceptrons on top of each other would enhance their learning abilities considerably. However, the learning algorithm proposed by Rosenblatt could not deal with multiple layers. Minky and Papert’s analysis effectively shut down neural network research for several decades.

The second wave

Almost thirty years later, around 1986, the psychologists David Rumelhart, Geoffrey Hinton and Ronald Williams proposed backpropagation, a learning algorithm for multilayer perceptrons. With the new learning algorithm, stacked Perceptrons could be trained to solve much more complex tasks than could be solved by the original single-layer Perceptron.  The typical multilayer perceptron consisted of an input layer, a hidden layer of neurons, and an output layer. The two-layer perceptron could handle quite complex recognition tasks. Just like its predecessor, the multilayer perceptron gave rise to overoptimistic expectations about brain-like machines. After a few years, when the limitations of multilayer perceptrons became apparent, the enthusiasm for neural networks faded again. They were replaced by more principled methods, such as support vector machines, that equal or even exceed multilayer perceptrons in performance, but have a more solid foundation in mathematics and optimisation theory.

The third wave

Currently, data science is in the middle of the third neural network wave. A special kind of multilayer perceptron, called a convolutional neural network, led to breakthroughs in many recognition and prediction tasks. Convolutional neural networks are multilayer perceptrons that are relatively insensitive to shifts of patterns in their input. Typically, the networks are more than two layers deep, hence their name “deep learning”. The first convolutional neural network was developed back in 1989 by Yann LeCun, founding Director of the NYU Center for Data Science and director of AI research at Facebook. The re-emergence of this 25-year old neural network is mainly due to two technological advances: the large amount of data available for training and the widespread availability of powerful (parallel) computer hardware. (Similar deep-learning advances are made in the closely-related domains of reinforcement learning and recurrent neural networks.) Deep learning leads to breakthroughs in a large variety of tasks where traditional data science and machine learning research got stuck in the past. Just last week, a new breakthrough was announced by the Google-acquired startup Deep Mind: Combining deep neural networks with efficient search algorithms, Deep Mind was able to beat the European champion of the game of Go.

Deep learning in Tilburg and ’s-Hertogenbosch

The insights obtained from deep learning contribute to theories and applications in many domains. In the context of the Data Science Center Tilburg the following examples are worth mentioning.

Our Cognitive Science and Artificial Intelligence group has several research lines focussing on deep learning. Together with my PhD students Nanne van Noord and Yu Gu, I study deep learning applied to visual, auditory and time-series data. Together with their co-workers, my colleagues Pieter Spronck and Grzegorz Chrupała investigate deep learning in the video-game and linguistic domains, respectively.

In cognitive neuroscience, deep learning offers a powerful tool for the analysis of fMRI and EEG data. In addition, convolutional neural networks trained on natural data have been shown to model cortical processing, for instance, in the inferotemporal (object recognition) pathway of the human visual system. As for business and economics, deep predictive analytics is still in its infancy but is generally expected to lead to major breakthroughs in the coming years. Especially for the joint master program Data Science and Entrepreneurship in ’s-Hertogenbosch, young data science entrepreneurs may benefit from the new business opportunities offered by deep learning. Finally, deep learning confronts society with another step towards extraction of information from data. This raises many questions that need to be addressed by legal and ethical scholars working at the Data Science Center Tilburg.


  • Chrupała, G., Kádár, A. & Alishahi, A. Learning language through pictures. 2015. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). Zong, C. & Strube, M. (eds.). Beijing, China: Association for Computational Linguistics, p. 112-118. http://www.aclweb.org/anthology/P15-2019.
  • Noord, N. van, Hendriks, Ella, & Postma, E. (2015). Toward discovery of the artist's style: learning to recognize artists by their artworks. IEEE Signal Processing Magazine, 32(4), 46-54.
  • Vries, M. de & Spronck, P. (2016, in preparation), Modeling play styles and game platforms.