Understanding Abilities and Failures of Language Models

At the invitation of the current CAS cohort, Michael Hahn will be our guest. His talk on “Understanding Abilities and Failures of Language Models” is part of the cohort’s project entitled “Knowledge acquisition, representation and application in human minds and machines.”

Abstract:
The reasoning capabilities of LLMs have seen enormous progress, but it remains hard to predict when they fail, and how many reasoning tokens they need to solve different problems. I will present two lines of research aiming to make reasoning abilities more predictable via theoretical bounds on the abilities of the underlying architecture — the Transformer. First, I will present our recent work aiming to predict on which algorithmic tasks transformers can generalize to longer inputs, and compare to LLM performance. Second, I will describe our recent work bounding the reasoning cost needed to solve various algorithmic problems with transformers. I will close by discussing problems for further research. 

The lecture will be streamed on Zoom as well. Thus, a (virtual) participation is possible. Please use the following link to access the Zoom meeting: https://uni-mannheim.zoom-x.de/j/68235694831. You will first enter the so-called “waiting room”. The host will give you access once the event officially starts.