Wann? Freitag, 16.01.2026, 11:00 Uhr (CET)
Wo? Kolingasse 14-16, Raum Nr. 2.38 (live stream viewing)
Zoom-Link: univienna.zoom.us/j/66136752477

Titel: Comprehensively Evaluating Language in Language Models

Abstract

As Large Language Models are being increasingly used in high-stakes situations, it is vital that we accurately assess their strengths, but also their limitations. To this end, I ask: how can we ensure that we neither over- nor underestimate Language Models' linguistic capabilities? For this, evaluations must consider the full breadth of human language. In my talk, I will demonstrate how progress can be made towards this goal in two aspects: multilingual evaluation, and evaluation for the long tail of language. For multilingual evaluation, I will show how agreement evaluation can be scaled to over 100 languages. For the long tail of language, I will report results from two investigations of language models' understanding of the so-that construction, with which even state-of-the-art models struggle, even though rich distributional information is available in their training data. I will further demonstrate how LLMs themselves can be leveraged to annotate corpora for long-tail constructions, further stretching the boundaries of what we are able to test. All evaluations together paint a nuanced picture of the linguistic capabilities of large language models, showing achievements as well as remaining deficits.

Vortragende

Leonie Weissweiler is a postdoc at Uppsala University Computational Linguistics, working with Joakim Nivre and supported by the Walter-Benjamin-Programme of the DFG. From April 2026 on, she will be assistant professor for Natural Language Processing at Leipzig University. She is interested in the implications of LLMs for linguistic theory, as well as their uses for linguistic typology.
She received her PhD in 2024 from LMU Munich, where she worked with Hinrich Schütze on computational approaches to Construction Grammar and morphology, with visiting positions at Carnegie Mellon University and Princeton University. Before her PhD, she studied computational linguistics in Munich and Cambridge.



Wenn Sie über zukünftige Vorträge der Vienna Circuits-Reihe informiert werden möchten, können Sie sich in die » Mailingliste zur Reihe eintragen.