Representation Learning: Foundations and Challenges

Representation Learning: Foundations and Challenges

 

Representation learning refers to a family of methods that enable models to automatically discover informative representations of data. Designing models that operate directly on raw inputs (such as pixels, waveforms, or text tokens) is often challenging: these inputs are high-dimensional, the underlying factors of variation are entangled, and the information relevant to a given task typically constitutes only a small fraction of the observed signal. Representation learning addresses this challenge by learning transformations that map raw inputs into a representation space where the relevant structure of the data becomes more explicit and easier to exploit. When successful, these learned representations can be more informative than the original inputs and support a wide range of downstream tasks, such as classification, retrieval, and generation.

This seminar is organized around the central question: how do training objectives shape learned representations? More specifically, we will study the types of structure that different learning objectives induce in the representation space. To explore this question, the seminar is divided into four thematic blocks:

• Block 1 – Generative Models

How latent variable models learn representations as a byproduct of the generative process.
Keywords: latent spaces, VAEs, multimodal VAEs, representation learning in diffusion models.

• Block 2 – Contrastive Learning

How representations can be learned by aligning data across modalities and the geometric properties induced by this alignment.
Keywords: InfoNCE, CLIP, cross-modal alignment, modality gap.

• Block 3 – Disentanglement

How to learn structured representations in which individual dimensions or subspaces correspond to meaningful factors of variation in the data.
Keywords: β-VAE, shared vs. private representations.

• Block 4 – Causal Representations

How to move beyond statistical correlations toward representations that capture the underlying causal structure of the data.
Keywords: causal representation learning, spurious correlations, interventions.

Students will be divided into four teams (one team per seminar block). Each team will work collaboratively throughout the seminar, guided by the instructors, to conduct a literature review on their assigned topic. The final goal of each team is to prepare and deliver a tutorial that introduces the topic, explains its evolution over time, and discusses the current state of the art.

Evaluation will be based on participation, the quality of the presentations, and the depth of the literature analysis.

 

Time and place

Thursdays, 12:15 to 13:45.

Room:  0.01 at building E1.7 

 

Attendance

The seminar will be held in person, and attendance is mandatory. 

 

General structure

The students will be divided into four teams (one team per seminar block), with three students per team. Each team will work collaboratively throughout the seminar, guided by the instructors, to conduct a literature review on their assigned topic. The final goal is for each group to prepare and deliver a tutorial that introduces the topic, explains how it has evolved over time, and discusses the current state of the art.

The seminar will include three types of sessions:

• Background sessions: Sessions led by the instructors to provide the necessary foundations in representation learning and guidance on how to approach and understand research papers.

• Private group meetings: Informal meetings with each team to discuss their progress, evaluate the current state of their presentation, address questions, identify missing elements or unclear aspects, and provide feedback on the content, structure, and their overall understanding of the research landscape.

• Public presentations: Sessions where each team presents their tutorial to the rest of the class, followed by questions and discussion.

Background sessions (2 sessions)

The seminar will start with two background sessions designed to give students the necessary tools to begin their literature review.

Building a tutorial I: Foundations, Definitions, and Research Questions (4 sessions)

The goal of this stage is for each team to build a strong understanding of the foundations of their topic. Their presentation should introduce the key definitions and concepts, present the main approaches in the literature, and discuss the limitations and open questions that motivated subsequent research.

• Private group meetings (2 sessions)

  • One private meeting per team (45 min).

• Public presentation (2 sessions)

  • One presentation per team (45 min for presentation + Q&A).

Building a Tutorial II: Evolution of the Topic and Current State of the Art (4 sessions)

During this stage, students should expand their presentation to cover the evolution of the field, including major developments, influential works, recent advances, and current open research directions and challenges. They should also incorporate their own perspective and draw their own conclusions. At this stage, each team is expected to discuss at least 9 papers (a minimum of 3 papers per team member).

• Private group meetings (2 sessions)

  • One private meeting per team (45 minutes).

• Public presentation (2 sessions)

  • One presentation per team (45 min for presentation + Q&A)

 

 

Calendar

Date	Session Type	Content
October 15	Background	Introduction to Representation Learning
October 22	Background	How to read a paper?
3 weeks without seminar sessions
November 19	Private meetings (GM, CL)	Building a tutorial I
November 26	Private meetings (D, CRL)	Building a tutorial I
1 week without seminar sessions
December 10	Public presentations (GM, CL)	Building a tutorial I
December 17	Public presentations (D, CRL)	Building a tutorial I
3 weeks without seminar sessions
January 14	Private meetings (GM, CL)	Building a tutorial II
January 21	Private meetings (D, CRL)	Building a tutorial II
1 week without seminar sessions
February 4	Public presentations (GM, CL)	Building a tutorial II
February 11	Public presentations (D, CRL)	Building a tutorial II

 

* Each team has 2 weeks between the private group meeting and the presentation, and 4 weeks between their first presentation and the next private meeting.

IMPORTANT NOTE: The session scheduled for December 10 will likely be rescheduled due to NeurIPS (December 9–13).

Selecting topic

We will share a form with the students to indicate their preferences.

• Release form to indicate preferences: October 15 (first background session)

• Release team assignment: October 22 (second background session, 3 weeks before the first private group meeting)

 

Deliverables and grading scheme

TBA

List of seminal papers

TBA