Tilburg center for Cognition and Communication (TiCC)

We study how people communicate with each other and how computer systems can be taught to communicate with us.


TiCC Colloquium: Tal Linzen

What: Structure-sensitive dependency learning in recurrent neural networks
Where: CZ 8
When: Wednesday, 31 May 2017, 12:45 - 13:45 hours


Neural networks have recently become ubiquitous in natural language processing systems. Yet we typically have little understanding of these networks beyond their accuracy on a particular test set. The present work investigates the ability of recurrent neural networks (RNNs), which are not equipped with explicit syntactic representations, to learn structure-sensitive dependencies from a natural corpus; we use English subject-verb number agreement as our test case.

We examine the success of the RNNs (in particular LSTMs) in predicting whether an upcoming English verb should be plural or singular. We focus on specific sentence types that are indicative of the network's syntactic abilities, and use both naturally occurring sentences and constructed sentences from the experimental psycholinguistics literature. We analyze the internal representations of the network to explore the sources of its ability (or inability) to approximate sentence structure. Finally, we compare the errors made by the RNNs to agreement attraction errors made by humans.

RNNs were able to approximate certain aspects of syntactic structure very well, but only in common sentence types and only when trained specifically to predict the number of a verb (as opposed to a standard language modeling objective). In complex sentences their performance degraded substantially; they made many more errors than human participants. Our results suggest that stronger inductive biases are likely to be necessary to eliminate errors altogether and acquire syntax from a natural corpus. More broadly, they illustrate how methods from linguistics and psycholinguistics can help us understand the abilities and limitations of "black-box" neural network models.'

About Tal Linzen


When: 31 May 2017 12:45

End date: 31 May 2017 13:45