News

Exam & Sheet 6

Written on 26.07.23 by Johannes Hoffmann

The date for the exam has been fixed. All information regarding the exam can be found on the main page.

The full solution to sheet 6 will be available soon. There was a problem with ReLU in exercise 3, where the scaling factor of 1/sqrt(m) should not have been part of the network function, but… Read more

The date for the exam has been fixed. All information regarding the exam can be found on the main page.

The full solution to sheet 6 will be available soon. There was a problem with ReLU in exercise 3, where the scaling factor of 1/sqrt(m) should not have been part of the network function, but instead part of the initialization of the vector a. The exercise will be updated accordingly. Apologies for the inconvenience.

No more lectures

Written on 10.07.23 by Roland Speicher

  • Today we had the final lecture in the course.
  • There will be no lectures on Wednesday nor next week.
  • Instead you can look a bit more into free probability theory, either by reading the article of Roland Speicher in the Materials under Some fun reading, or by watching the following videos:
  • Read more

Lecture 20 - incomplete video

Written on 06.07.23 by Roland Speicher

The video of Lecture 20 has been uploaded. Note, however, that due to technical problems, the last 20 minutes of the lecture were not recorded.

Exercise sheet 6

Written on 03.07.23 by Roland Speicher

The new (and last) assignment has been uploaded.

Updates

Written on 27.06.23 by Johannes Hoffmann

  • There will be no lecture tomorrow. The next lecture will be on Monday, July 3.
  • Sheet 6 will be released later this week, the exercise date and the hand-in date will shift by one week as well (so the last tutorial slot will be in the last week of the lectures).
  • There have been no requests for… Read more
  • There will be no lecture tomorrow. The next lecture will be on Monday, July 3.
  • Sheet 6 will be released later this week, the exercise date and the hand-in date will shift by one week as well (so the last tutorial slot will be in the last week of the lectures).
  • There have been no requests for the tutorial tomorrow so far. You can still request it until 8pm today (20:00 Uhr), otherwise the tutorial tomorrow will be canceled.

Exam

Written on 23.06.23 by Johannes Hoffmann

There will be a written exam for the High Dimensional Analysis. After Sheet 6 is graded we will know who is admitted to the exam, so we will then try to find a date for the exam that suits everyone that is interested in participating.

Sheet 3

Written on 01.06.23 by Johannes Hoffmann

The feedback for sheet 3 should now be available.

Also, please refer to the bottom of the main page for updates regarding the status of the exercise classes.

Updates

Written on 30.05.23 by Johannes Hoffmann

Exercise sheet 4 has been released.

The date for the exercise class for sheet 3 has been changed from tomorrow to one week later (June 7, usual time slot). Since nobody showed up for the last exercise class, future exercise classes will only take place if at least one participant requests this in… Read more

Exercise sheet 4 has been released.

The date for the exercise class for sheet 3 has been changed from tomorrow to one week later (June 7, usual time slot). Since nobody showed up for the last exercise class, future exercise classes will only take place if at least one participant requests this in advance (just send me an email).

The feedback for sheet 3 will be available in the next few days.

New exercise sheet & tutorials

Written on 01.05.23 by Johannes Hoffmann

Exercise sheet 2 is now available.

The first tutorial will be on Wednesday, May 3, in SR 2 (E2.5) at 14:15.

Tutorial time slot

Written on 27.04.23 by Johannes Hoffmann

Please indicate your preference for the tutorial time slot via Doodle. The slot with the most votes wins. The deadline is defined as "when I look at the results on Monday, May 1", so try to vote Sunday at the latest.

Updates to sheet 1

Written on 23.04.23 by Johannes Hoffmann

Two changes were made to sheet 1: first, in exercise 4(c), mu and sigma were missing their normalization constants, which have been added. Second, the estimate in exercise 4(b) was replaced with one that works better with the task.

Please excuse the inconvenience.

Schedule for next week

Written on 20.04.23 by Johannes Hoffmann

There will be no regular lectures next week.

Instead you should watch an earlier video of Roland Speicher on the Wick Formula (up to 1:16:33). This explains and proves the Wick formula (which is a combinatorial formula for moments of Gaussian variables and of products of independent Gaussian… Read more

There will be no regular lectures next week.

Instead you should watch an earlier video of Roland Speicher on the Wick Formula (up to 1:16:33). This explains and proves the Wick formula (which is a combinatorial formula for moments of Gaussian variables and of products of independent Gaussian variables). This formula will be used later in our course; the main facts about the Wick formula will be provided then, but the video offers more details.

Additionally, instead of the lecture on Wednesday, April 26, there will be a Q&A session for the first exercise sheet.

As May 1 is a holiday, the next regular lecture will be on Wednesday, May 3.

First exercise sheet available

Written on 17.04.23 by Johannes Hoffmann

The first exercise sheet is now available under Materials. Submissions can be handed in via CMS starting from 22.04.2023. Please compile your submission into one pdf.

Up until the evening of the 21st, you can create or join teams to hand in the exercises in groups of up to three people (or you can… Read more

The first exercise sheet is now available under Materials. Submissions can be handed in via CMS starting from 22.04.2023. Please compile your submission into one pdf.

Up until the evening of the 21st, you can create or join teams to hand in the exercises in groups of up to three people (or you can hand them in as a singleton, then you don't have to create a team).

Starting time of the lectures

Written on 16.04.23 by Roland Speicher

All lectures will start c.t., i.e. at 10:15 a.m.

First Lecture

Written on 05.04.23 by Roland Speicher

The first lecture will be on Wednesday, April 12, at 10:15.

Registration

Written on 14.03.23 (last change on 14.03.23) by Roland Speicher

Registration is now open for the course.

 

Show all

High Dimensional Analysis: Random Matrices and Machine Learning

4h Topics Course (9 CP)

Analysis in high-dimensional spaces is at the basis of an understanding and rigorous treatment of dealing with large data sets. In a very rough sense one has to understand functions of many variables, in particular also their generic and statistical behaviour.

High-dimensional spaces are actually quite strange and exotic, when considered from our experience in small dimensions (remember that we live in three dimensions). On one side, high-dimensional spaces are VERY large, so it is easy to get lost there (a phenomenon often known under the name “curse of dimensionality”), but on the other hand, functions of many variables often concentrate to constant values and thus allow very precise answers to apparently intractable questions (a phenomenon going under the name of “concentration phenomenon” and also considered as the “blessing of dimensionality”).

Also statistics in high dimensions with many variables and many observations is different from what one is used to in a classical setting. In classical statistics the number of variables is small compared to the number of observations and essentially one can understand the distribution of the variables via the law of large numbers by doing many measurements. In modern statistics the number of variables is of the same order as the number of observations, so the relation between the real and the measured properties of the variables is more tricky and new mathematical tools are needed to unravel this. In particular, random matrices (which were originally introduced by the statistician Wishart) are such a tool.

The neural networks of modern deep learning are in some sense a special class of functions of many variables, built out of (random) matrices and also some entry-wise non-linear functions. Thus random matrices should help to say something about neural nets, but the latter also present new and interesting extensions of random matrices.

In this course we will take a trip through high dimensions. We hope not to get lost, but to get a feeling for its strange and beautiful behaviour. Much of the terrain is still unexplored, but there are some first interesting islands which have been discovered, waiting for us to explore them.

This course is intended for everybody with an interest in a mathematical rigorous discussion of high-dimensional phenomena. A basic mathematical background on the level of the MfI 1-3 courses should be sufficient (but also necessary). Be aware, however, our intention is not to develop better and faster algorithms for deep learning, but to touch upon some mathematical theories which might (or might not, who knows) be important for a sound mathematical theory of deep learning.

 

Tentative List of Topics

  • curse and blessing of dimensionality
  • concentration of vectors and matrices in high dimensions
  • Wishart matrices and Marchenko-Pastur law
  • signal-plus-noise models
  • neural networks, overparameterization, neural tangent kernel, feature learning and all that

 

Bibliography

Here are some references on which the lectures will partly rely or which might provide some further reading. In any case they should give an idea what the course will be about. We will not follow one source exclusively, but pick the best from each of them, according to the taste and interest of the lecturer.

  • R. Couillet and Z. Liao: Random Matrix Methods for Machine Learning, Cambridge University Press, 2022
  • D.A. Roberts and S. Yaida: The Principles of Deep Learning Theory, Cambridge University Press, 2022
  • R. Vershynin: High-Dimensional Probability, Cambridge University Press, 2018
  • M.J. Wainwright: High-Dimensional Statistics, Cambridge University Press, 2019
  • A. Zagidullina: High-Dimensional Covariance Matrix Estimation, Springer, 2021
  • the Tensor programs framework of Greg Yang
  • A. S. Bandeira, A. Singer, T. Strohmer: Mathematics of Data Science (draft)

This list is not yet final and will increase in due time.

See also the Semesterapparat/Course Reference Shelf provided by the Campus Library for Computer Science and Mathematics.

 

Time and Place

Lectures: Mondays and Wednesdays, 10-12 in HS IV, building E 2.4. The lectures start c.t., i.e., at 10:15 a.m.

Tutorials: Wednesdays, 14-16 in SR 2, building E2.5. The tutorials also start c.t., i.e., at 14:15.

  • Exercise class 1: May 3
  • Exercise class 2: May 17 (canceled)
  • Exercise class 3: June 7
  • Exercise class 4: June 14
  • Exercise class 5: June 28 (canceled)
  • Exercise class 6: July 19 (canceled)

Exam

The exam will be on Wednesday, August 9, in lecture hall III (HS III), from 14:15 to 16:15.

You are allowed to bring your own notes to the exam, as long as they are on sheets of paper (handwritten or printed is both fine). The content can be anything you like, but keep it to a reasonable length/amount and don't bring any books. In particular, any use of an electronic device (calculator, phone, laptop, ...) during the exam will be considered cheating.

Don't forget to register via HISPOS/LSF!

Exam inspection

The exam inspection will be on Friday, August 11, in lecture hall IV (HS IV) in building E2.4, from 13:30 to 14:00.

Privacy Policy | Legal Notice
If you encounter technical problems, please contact the administrators.