ETH Zurich - D-INFK - IVC - CVG - Lectures - Mixed Reality

Mixed Reality


MR Logo
Course ID: 263-5905-00L (Autumn Semester 2021)
Instructors: Dr. Iro Armeni, Dr. Federica Bogo, Prof. Marc Pollefeys
Teaching assistants: Taein Kwon, Denys Rozumnyi, Jonas Hein, Isaak Hanimann
Lectures: Mondays 10:15-13:00 in Zoom Meeting (Link in Moodle)
Language:English
Credits:5KP

Quick Access

All course materials are provided in Moodle

Course Description

Brief Description The goal of this course is an introduction and hands-on experience on latest mixed reality technology at the cross-section of 3D computer graphics and vision, human machine interaction, as well as gaming technology.
Learning Goals After attending this course, students will:
1. Understand the foundations of 3D graphics, Computer Vision, and Human-Machine Interaction
2. Have a clear understanding on how to build mixed reality apps
3. Have a good overview of state-of-the-art Mixed Reality
4. Be able to critically analyze and assess current research in this area
Course Evaluation 1. 5 %: Course Participation (e.g., during the class in lectures and paper discussion or at the online forum)
2. 25%: Paper presentation (incl. discussion moderation)
3. 70%: Final project which includes a report and presentation/demo
Prerequisites / Notice Prerequisites include:
- Good programming skills (C# / C++ / Java etc.)
- Computer graphics/vision experience: Students should have taken, at a minimum, Visual Computing. Higher level courses are recommended, such as Introduction to Computer Graphics, 3D Vision, Computer Vision.
Related Courses 151-0306-00L "Visualization, Simulation and Interaction - Virtual Reality I"
151-0317-00L "Visualization, Simulation and Interaction - Virtual Reality II"
252-0570-00L "Game Programming Laboratory"

Course Schedule

No. Date Topic Presenter Paper Presentations
1. 27.09 MR Introduction Iro Armeni, Federical Bogo -
2. 04.10 Introduction to Unity, HoloLens, and ASA Patrick Misteli (Microsoft) -
3. 11.10 Project Proposal Presentation - -
4. 18.10 Guest Lecture Martin Oswald 2, 10, 16
5. 25.10 Mixed Reality with HoloLens 2 Marc Pollefeys 1, 8, 9
6. 01.11 Experience Design for Mixed Reality - how to create empowering experiences Stella Muehlhaus (Facebook) 5, 7, 22
7. 08.11 Augmented Virtuality, Diminished Reality, Substitutional Reality Andreas Fender (ETH) 4, 14, 15
8. 15.11 Mid-Term Project Presentation - -
9. 22.11 3D scene understanding for AR applications Federico Tombari (TUM/Google) 3, 20, 21
10. 29.11 Medical AR Esfandiari Hooman (Balgrist UniversitatsKlinik) 6, 13
11. 06.12 12, 17, 18
12. 13.12 Egocentric perception and anticipation for user-centric applications Antonino Furnari, Giovanni Maria Farinella (University of Catania) 11, 19
13. 20.12 Final Project Presentations - -

Paper list

1. Depth from motion for smartphone AR
2. CoolMoves: User Motion Accentuation in Virtual Reality
3. MEgATrack: Monochrome egocentric articulated hand-tracking for virtual reality
4. Human touch: Social touch increases the perceived human-likeness of agents in virtual reality
5. Egocentric pose estimation from human vision span
6. Passthrough+: Real-time stereoscopic view synthesis for mobile mixed reality
7. Evaluating grasping visualizations and control modes in a VR game
8. Driving signal aware full-body avatars
9. DepthLab: Real-time 3D interaction with depth maps for mobile augmented reality
10. Deep implicit volume compression
11. TapID: Rapid touch interaction in virtual reality using wearable sensing
12. SMPL: A Skinned Multi-Person Linear Model
13. MRTouch: Adding Touch Input to Head-Mounted Mixed Reality
14. HoloDesigner: A mixed reality tool for on-site design
15. LoBSTr: Real-time lower body pose prediction from sparse upper-body tracking signals
16. DeepFovea: Neural reconstruction of foveated rendering and video compression using learned statistics of natural videos
17. SuperGlue: Learning feature matching with graph neural networks
18. Real-time mixed reality teleconsulation for intensive care units in pandemic situations
19. Temporally adaptive shading reuse for real-time rendering and virtual reality
20. Mixed reality light fields for interactive remote assistance
21. LoFTR: Detector-Free Local Feature Matching with Transformers
22. H2O: two hands manipulating objects for first person interaction recognition

Deadlines

October 02: Group formation and project selection - Students select from a list of project proposals and we assign them to the topics.

October 10: Project proposal documents - Students submit their project proposal documents after discussing with their assigned supervisors.

October 11: Proposal presentations - Students present their project proposals during lecture.

November 12: Midterm project report - Students submit a written report with current updates and plan.

November 15: Midterm presentations - Students present their progress on their projects during lecture.

December 20: Final project presentations - Students present their projects in a joint session.

January 07: Final project reports - Students submit their final reports for the projects.

* All the submission deadlines are due by 23:59 on the specified date.

Student Projects

Over the semester, students will work on a project related to a topic in Mixed Reality in collaboration with a supervisor. Students are required to form groups of 4 and submit their preferred project topics first. For this purpose, we will provide a list of project suggestions, but you are free to propose your own project.

For your projects, we can make the following hardware available:
  • Microsoft HoloLens
  • Mobile phone devices

Students with project idea that require any other special equipment, please talk to us! You will also get access to a Windows machine that you can use for development purposes.

Each student group is then required to hand in and present a project proposal by the announced deadline. Make sure to talk to your assigned supervisor and discuss the project with him/her while planning your proposal. Each team will present their project proposal during a designated lecture. The template for the project proposal report can be found here.

The proposal should be 1-2 pages describing what you want to do in the project, and how you plan to achieve your envisioned results. A good idea is to identify the algorithmic and technical challenges within the project. Try to address each of them individually and explain your considered solutions; also make an attempt to think about alternatives if you believe a particular approach is unstable or likely to fail.

Midterm presentations have the purpose that you present what you did so far and that you get feedback. So you are encouraged to raise open questions. This is a possibility for us to steer the project and help you, if you got stuck. In addition, you are required to submit a 2-page report that identifies your current status, problems you have identified, and your next steps.

Final presentations will be held as a regular presentation session. In addition, you are required to hand in a technical report for your project. The report format should be in parallel with CVPR paper format. Latex and Word templates can be found here.

Student Paper Presentations

In each class, a (guest) lecture will be given first. After the lecture, students, together with their project group members, will give presentations on selected papers.

Other students are encouraged to engage in the paper presentations through active discussions. To organize the discussion in a more lively way, each project group will be assigned to lead the discussion of an other project group's presentation; i.e. this project group acts as an "opponent" or moderator and actively supports the discussion by asking relevant questions wrt. the presented paper and motivates other students to contribute.

We will assign each group a paper and a presentation date after the projects are assigned. The list of papers to be presented by students will be published soon.

Discussion Forum

There exists a discussion forum page in MOODLE for this course. Students should sign in using their ETHZ accounts and participate in the discussion forums. Please put all your discussions related to the lectures, paper presentations and projects there.

Some useful links

How to read and present scientific papers

© CVG, ETH Zürich webmaster: psarlin@inf.ethz.ch