Fakultät Informatik

3D Scanning & Motion Capture


 Prof. Dr. Matthias Niessner


Master Informatics

Time, Place

Tuesdays, 14:00-16:00, Room:  02.13.010
Wednesdays, 10:00-12:00, Room:  02.13.010


October 17., 2017


Introduction to Informatics I, Analysis, Linear Algebra, Computer Graphics, C++


3D reconstruction, RGB-D scanning (Kinect, Tango, RealSense), ICP, camera tracking, sensor calibration, VolumetricFusion, Non-Rigid Registration, PoseTracking, Motion Capture, Body-, Face-, and Hand-Tracking, 3D DeepLearning, selected optimization techniques to solve the problem statements (GN, LM, gradient descent).



Lecture Content 

  • Requirements
    • C++ is a must
    • Profound knowledge of linear algebra
    • Basic concepts of graphics and 3D
  • Basic concepts of geometry
    • Meshes (polygonal), Point Clouds, Pixels & Voxels
    • RGB and Depth Cameras
    • Extrinsics and Intrinsics
    • Capture devices
      • RGB and Multi-view
      • RGB-D cameras
      • Stereo
      • Time of Flight (ToF)
      • Structured Light
      • Laser Scanner, Lidar
  • Surface Representations
    • Polygonal meshes (trimeshes, etc.)
    • Parametric surfaces: splines, nurbs
    • Implicit surfaces
      • Ridge-based surfaces
      • Radial basis functions 
      • Signed distance functions (volumetric, Curless & Levoy)
      • Indicator function (Poisson Surface Reconstruction)
      • More general: level sets
        • Marching cubes
  • High-level overview of reconstructions
    • Structure from Motion (SfM)
    • Multi-view Stereo (MVS)
    • SLAM
    • Bundle Adjustment
  • Optimization
    • Non-linear least squares
    • Gauss-Newton LM
    • Examples in Ceres
    • Symbolic diff vs auto-diff
    • Auto-diff with dual numbers
  • Rigid Surface Tracking & Reconstruction
    • Pose alignment
    • ICP (point cloud alignment; depth-to-model alignment; rigid ‘fitting’)
    • Online surface reconstruction pipeline: KinectFusion
    • Scalable surface representations: VoxelHashing, OctTrees
    • Loop closures and global optimization
    • Robust optimization
  • Non-rigid Surface Tracking & Reconstruction
    • Surface deformation for modeling
    • Regularizers: ARAP, ED, etc.
    • Non-rigid surface fitting: e.g., non-rigid ICP
    • Non-rigid reconstruction: DynamicFusion/VolumeDeform/KillingFusion
  • Body Tracking & Reconstruction
    • Skeleton Tracking and Inverse Kinematics
    • Learning-based approaches from RGB and RGB-D
  • Face Tracking & Reconstruction
    • Keypoint detection & tracking
    • Parametric / Statistical Models -> BlendShapes
  • Hand Tracking & Reconstruction
    • Parametric Models 
    • Some DeepLearning-based things
  • Discriminative vs generative tracking
    • Random forests
  • Motion Capture in Movies
    • Marker-based motion capture
    • LightStage -> movies
  • BRDF and Material Capture
  • Open research questions
  • Useful tools:
    • Eigen
    • Ceres


D. A. Forsyth and J. Ponce. Computer Vision: A Modern Approach (2nd Edition). Prentice Hall, 2011.

R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2003.

Botsch, Kobbelt, Pauly, Alliez, Levy: Polygon Mesh Processing, AK Peters, 2010




Bachelor and Master thesis in the following areas:
- A remote rendering system for point cloud data (in collaboration with industry)

- Deep learning for improved weather forecasting

- Learning trajectory clustering using neural network
- Learning Level-of-Detail representations for point clouds

- In collaboration with partners from industry, we have a number of thesis topics available in the area of point-based rendering, geo-localization using public data, scene fusion from different viewpoints. If you are interested, please contact  westermann(at)tum.de


- One PhD position on   Turbulence Visualization is available at the Computer Graphics & Visualization group.