Does Copilot Really Understand my Code? Methods for Mechanistic Interpretability of LLMs

Does Copilot Really Understand my Code? Methods for Mechanistic Interpretability of LLMs

by Sven Apel, Youssef Abdelsalam, and Anna-Maria Maurer

Code LLMs have become ubiquitous in software understanding and development. However, despite their increasing integration into development workflows, they remain black-box models. What do these AI models actually understand about code, and what methods can we use to peek under the hood? What correlational and causal relationships or characteristics can be identified?

In this seminar, students will read and critically discuss papers about methods for mechanistic interpretability, as well as familiarize themselves with these methods in short demos. Throughout the semester, they will formulate research questions targeting assigned topics centering diverse aspects of code, design small experiments for these questions using these methods, and then apply them to analyze what code LLMs represent internally and how information flows through the transformer architecture. Each student will be supported by a dedicated advisor and receive feedback throughout the semester. The seminar concludes with a final report and research presentations in which each participant presents their results and reflects on their implications.

Kick-Off Meeting: Thursday, 22 October 2026

The seminar takes place during the semester on Thursdays from 12:00 - 14:00 (~12 sessions in sum) and in addition final presentation sessions during the lecture free period.

Participation in all sessions is mandatory, as is the submission of all assignments.

Requirements: 

This seminar is open to motivated Bachelor and Master students who are eager to understand the inner workings of LLMs while critically and empirically examining their behavior. Prior knowledge of LLMs and transformer architectures is recommended, along with general programming experience.

Further information will be provided via e-mail after registration.