Seminar on Benchmarking AI (HWS 2026)

AI Systems, and Large Language Models in particular, have been used for an increasing variety of tasks. At the same time, new and improved models are released on a day to day basis.

If an AI system is to be used for a specific task, it is therefore interesting to know how good different AIs fare on that task. This is the purpose of AI benchmarks.

In this seminar, we will cover two aspects:

• Which benchmarks exist for different tasks and domains? How are they created? How do they differ?
• Since most of these benchmarks are public, benchmark leakage is a problem (i.e., large language models may memorize a benchmark and thereby get good ratings, despite worse performance on the task in general). How can it be detected? How can we mitigate benchmark leakage?

This semester runs as a dual seminar for bachelor and master students. All students are expected to attend all presentations. The focus, however, is different for both groups:

• Bachelor students are expected to pick a task and/or domain for AI benchmarks (e.g., mathematical problem solving, question answering, sentiment analysis, text summarization, …) and present and compare different benchmarks that exist for the task
• Master students are expected to pick a family of benchmark leakage detection or benchmark leakage avoidance techniques (e.g., white box, gray box, black box, LLM-as-a-judge, …) and present and compare them.

The first part of the seminar will be bachelor presentations, the latter will be master presentations.

This seminar is organized by Prof. Dr. Heiko Paulheim

Available for Bachelor Students (2 SWS, 5 ECTS) and master students (2 SWS, 4 ECTS)

Prerequisites: basic knowledge on LLMs

Additional resources:

• LaTeX template

Schedule:

• Wednesday September 23rd, 13:45–15:15: kick off meeting, room tba
• September 28th: topic assignment
• October 25th: paper draft due (only master students)
• November 1st: peer review due (only master students)
• Presentations Wednesdays, 13:45–15:15, room tba
  • October 14th, 21st, November 5th, 12th, 19th, 26th
• January 17th: final paper due