CS291-I: Visual Computing and Interaction –
Extended Reality (XR)

Spring 2024

This home page and the CS291I-XR Slack Channel will be used as centers of communication for the class (invite on Canvas). Homework submission and general messaging will occur through the CS291-I Canvas, including announcement of remote class lectures and meetings. We may also make use of other tools (Gradescope, Discord) if there is demand.

While the webpage provides you with up-to-date information about assignments and what is currently going on in class, the Canvas serves as the main coordination and submission site, and Slack as an open communication forum: questions, answers, suggestions, etc.

General Information

Class Hours:    Tue/Thu, 03:00-4:50pm
Class Location: Phelps 3526 
                Announcements will be made on Canvas and Slack. 
                Potential crashers should come to the first classes! 

Instructor:     Tobias Höllerer
Office Hours:   Wed, 4:30pm-5:30pm, or by appointment
Office:         2155 Harold Frank Hall, (805) 284 9395
e-mail:         holl@cs...

Course Description

Mixed and Augmented Reality, now often subsumed under the overarching term XR: Extended Reality, has been an active research field since the 1990s. It has recently gained significant popularity because of the possibility of being implemented on smartphones, because of new emerging head-worn platforms (including the recent Apple "VisionPro" device), and because of its unique approach of offering context-based computing directly in a person's field of vision. Augmented Reality is the concept of overlaying computer-generated information on top of the physical world. Mixed Reality is a bit broader and subsumes the fields of Augmented Reality, Augmented Virtuality, and Virtual Reality. Games such as Niantic's Pokemon GO as well as various Face filter apps and Snap Lenses have popularized the concept, but much more is possible. In particular, applied machine learning concepts can play a strong part in this research area. Virtually all the major players in the technology sector today invest more or less heavily in this paradigm, as the expectation is that it may well be the future of mobile personal information access. This class provides a hands-on introduction to these novel interface technologies.

This is an advanced research-oriented course. Programming experience and some knowledge of computer graphics and computer vision concepts is expected. What is of utmost importance is the students' readiness to self-motivatedly explore novel research areas!

A special focus area this quarter, promoted by recent developments in the field of HCI, will be the integration of visual computing technologies with generative and discriminative AI techniques, such as Large Language Models, GANs or Diffusion models, or ML/CV for scene understanding and reconstruction.

Course Requirements and Grading

There will be a series of AR/VR space creation, ideation, coding, or research paper presentation assignments that lead up to individual or group class projects. We will continuously assign reading material from various book chapters and papers that we will make available. There may be one exam (in week 6 or 7) or a series of quizzes. Here is how your final grade will be determined:  

• Exam or quizzes (if held, cumulatively would be worth 30%)
• 3-5 homework assignments (if we hold an exam, these will be worth 30%, otherwise 45%)
• Final project implementation, documentation, and presentation (if we hold an exam, these will be worth 40%, otherwise 55%)
• Class participation and activities [paper presentations, pointers to relevant material] will be noted and might positively influence your grade

In case you disagree with any grade, submit your grievance in writing (email) to the grader responsible, explaining and documenting your case.

Books and Tutorials, Class Materials

• We will be handing out materials from the following textbook: Schmalstieg/Höllerer, Augmented Reality, Principles and Practice, Pearson Addison-Wesley, 2016
  
  
• Previous incarnations of this course posed some assignments using OpenCV and OpenGL, and these basics may be important for some projects, especially when it comes to coordinate transformations using any modern XR toolkit or SDK.
  
  • OpenCV: Learning OpenCV, Computer Vision with the OpenCV Library Gary Bradski, Adrian Kaehler, O'Reilly Media
    Available from Safari Online (UCSB IP) here, from outside ucsb via library proxy here
    OpenCV Tutorials
  • OpenGL: Unless you want to focus on real-time or highly realistic 3D graphics in your final project, basic OpenGL knowledge will suffice to complete the class assignments. You can find much information online, e.g. at the official OpenGL web site. If you are the type of programmer who needs reference books to be effective, the two essential documentation resources that every serious OpenGL developer should have are the latest releases of:
    • The OpenGL Reference Manual
    • The OpenGL Programming Guide(edition from May 2012)
    • Available online
    
    
• Also, this quarter, you can opt to work more proactively with Unity, WebVR, and Smartphone resources (Unity has opened up their Premium learning materials for free). You need to have access to your own suitable hardware (e.g. reasonably powerful graphics computer, recent iPhone for ARKit, or ARCore-capable Android phone)
  
  
  • WebXR technologies:
    • Mozilla Hubs: https://hubs.mozilla.com/
    • https://www.8thwall.com/products-web
    • ...
    
    
  • Smartphone-based AR:
    • ARCore (Google Android)
    • ARKit (Apple)
    • Niantic Lightship SDK
      
      
  • Headset-based XR:
    • Meta Quest-1/2/3 or Quest Pro
    • MagicLeap, HoloLens, ...
      
    • Apple VisionPro
    • VR headsets
  
  
• For additional class materials, please see handout column in schedule below

Lateness Policy

All assignments are due at midnight on the scheduled due date. To make the deadlines more manageable, each student will be allowed three ``late days'' during the quarter for which lateness will not be penalized. Late days may be applied to all assignments but not the final project! Your  late days may be used as you see fit -- one or multiple per assignment -- but once you used a late day it's good and gone, you cannot reapply it to another assignment. Anything turned in after 12:00:00am until midnight the next day is one day late. Every day thereafter that an assignment is late, including weekends and holidays, counts as an additional late day.

Absolutely no late work will be accepted after the deadline if you have used up all your late days. If you're not done on time you must turn in what you have to receive partial credit. There will be no exceptions from this rule. Please make sure you understand this policy.

When making use of your late days, the online submission provides the timestamp that counts.

Academic Misconduct

We will strictly enforce UCSB's academic misconduct policies. We use electronic tools to detect plagiarism among submitted homework solutions and sources from the internet. Read these guidelines before beginning each programming assignment. Any form of plagiarism, collusion, or cheating will result in an "F" in this course and may result in suspension from UCSB for two quarters. When in doubt about any forms of receiving help on your assignments, ask us!

Open Door Policy

I would like the course to be informative and enjoyable. Let me know what you find good and interesting about the course. Let me know sooner if you feel something could be improved. See me (in virtual office hours), message on Slack, or send an e-mail.

Schedule:

Assignments

Every student taking this class needs to fill in the Student Info Questionnaire

For HW assignment 1, discussed in class session 2, you have to submit an experience report on XR tool installation/testing three times, each for a different exploration! Your first questionnaire must be submitted by Mon, 04/08, 11:59:59pm (we will discuss first results in Tuesday's class), the second one by end of the day Wed, 04/10 (11:59:59pm) .

For HW assignment 2, please document a thorough literature search in a writeup (overall, I would expect it to be about 2 papges long, but it can vary), and submit it as a PDF on Canvas. Also submit a spreadsheet with all the papers you considered. Submission deadline is Wednesday, 04/17, 11:59:59pm

HW assignment 3, is a paper review exercise (loosely modeled after peer review in the field) that will be due Wed., April 24th, 11:59:59pm

HW assignment 4 is a two-part assignment that will get you right into the middle of your quarter project work, with due dates on Wed., May 1st (for parts 1a and 1b) and Wed., May 8, 11:59:59pm (for part 2)