News

Lecture extended to 9 ECTS

Written on 02.10.23 by Eddy Ilg

Dear Students,

in the end of the lecture, there will be an exciting project on building your own SLAM pipeline. Due to the additional work, we decided to extend the lecture to 9 credit points similar to the lecture Neural Networks: Theory and Implementation.

Best,

Eddy … Read more

Dear Students,

Best,

Eddy

New 3D Computer Vision Lectures

Written on 25.07.23 (last change on 25.07.23) by Eddy Ilg

The CVMP lab will offer the new lectures “3D Computer Vision” and “3D Real World Modeling and Inference” in upcoming winter and summer semesters.

The 3D Computer Vision lecture will teach you the fundamentals of going from 2D images to 3D. After having given an intuition of the field and traditional methods, it will proceed to state-of-the-art approaches and provide you with a basis for working in 3D vision. The lecture will cover the mathematical fundamentals, feature matching, stereo, optical flow and scene flow estimation, rigid 3D reconstruction, non-rigid 3D reconstruction and SLAM.

The 3D Real World Modeling and Inference lecture will teach you how to design models in 3D. It will start with introducing the basic 3D representations ranging from point clouds to meshes, triplanes and voxel grids. Basic and modern deep learning techniques to train and perform inference on these representations will be introduced. Afterwards, the recent implicit representations such as NeRF and DeepSDF with MLPs and how to encode 3D scenes with them efficiently will be covered. Many modern models involve generative models and subsequently the basics of generative models from autodecoders to GANs and finally diffusion models will be presented.

3D Computer Vision

Summary

Computer vision has led to many recent technology breakthroughs and is today one of the most demanded fields. 3D computer vision is becoming increasingly important, and the field has recently shown remarkable progress.

This lecture will teach you the fundamentals of 3D computer vision. After having given an intuition of the field and traditional methods, it will proceed to state-of-the-art approaches and provide you with a basis for working on any 3D computer vision approaches.

The lecture starts with fundamentals on projective geometry and sensor devices. It will then dive into correspondence and depth estimation techniques that provide the foundation of 3D reconstruction. Starting with feature extraction and matching, the lecture will continue with stereo depth estimation, optical flow and scene flow estimation. Having covered the fundamentals, the lecture will proceed to complete reconstruction pipelines ranging from SfM, COLMAP and KinectFusion to the modern deep learning techniques NeRF and NeuS.

The lecture will then advance to online reconstruction with simultaneous localization and mapping (SLAM) approaches and show applications in autonomous driving and AR technology. Finally, the lecture will conclude with the most challenging discipline of generic non-rigid reconstruction.

The lecture will be accompanied by hands-on exercises comprised of sheets and coding tasks. Towards the end of the lecture, there will be and challenging and fun project to set up a simple complete SLAM pipeline.

The lecture is the prerequisite for the 3D Real World Modeling and Inference lecture that will be held in summer and presents the foundation of modern deep learning techniques in 3D.

The lecture is offered by the Computer Vision and Perception Lab that focuses on building the next generation machine perception algorithms. Please contact ilg@cs.uni-saarland.de if you are interested in working with the lab.

Requirements

As a prerequisite for this course, you must have taken either “High Level Computer Vision” or “Neural Networks: Theory and Implementation” with the computer vision project in the end. You must be familiar with CNNs and how to implement and train them with PyTorch.
Having taken “Computer Graphics” and “Image Processing and Computer Vision” is helpful, but not required.

Credit Points

9 ECTS (advanced lecture with project)

Lecturer

Prof. Eddy Ilg

Syllabus

Lecture 1: Acquisition Devices (RGB-, ToF-, Event-Cameras; LIDARs; IMUs) and Calibration
Lecture 2: Feature Extraction and Matching
Lecture 3: Projective Geometry and Image Formation, Part 1
Lecture 4: Projective Geometry and Image Formation, Part 2
Lecture 5: Rotations and Spherical Harmonics
Lecture 6: Stereo Depth Estimation Methods
Lecture 7: Optical Flow Estimation Methods
Lecture 8: Scene Flow Estimation Methods
Lecture 9: Rigid 3D Reconstruction: SfM, COMAP, KinectFusion, Sphere Tracing and Implicit Rep.
Lecture 10: Rigid 3D Reconstruction: Volumetric Representations (NeRF + NeuS)
Lecture 11: Visual Localization
Lecture 12: Simultaneous Localization and Mapping (SLAM) and Autonomous Driving
Lecture 13: Non-Rigid 3D Reconstruction (DynamicFusion, OcclusionFusion)
Lecture 14: Non-Rigid 3D Reconstruction (NeRFies, NeuS2, DynIBaR)

Exercises

Excercise 1: Geometry Fundamentals
Excercise 2: Estimating Poses with COLMAP and Visualizing Camera Rays
Excercise 3: Feature Matching with HoG and Deep-Learned Features
Excercise 4: Semi-Global Matching for Stereo Depth Estimation
Excercise 5: FlowNet for Optical Flow Estimation
Project: Implementing and Running a SLAM Pipeline