CS291A - Mixed and Augmented Reality

Winter 2022

This home page and the CS291A-MAR Slack Channel will be used as centers of communication for the class (invite on GauchoSpace). Homework submission and general messaging will occur through the CS291A GauchoSpace, including announcement of remote class lectures and meetings. We may also make use of other tools (Piazza, Gradescope) if there is demand.

While the webpage provides you with up-to-date information about assignments and what is currently going on in class, the GauchoSpace serves as the main coordination and submission site, and Slack as an open forum: questions, answers, suggestions, etc.

General Information

Class Hours:    Tue/Thu, 1:00-2:50pm
Class Location: Virtually, to begin with. Phelps 2510 after return to in-person instruction. 
                Regular meeting information will be announced on GauchoSpace. 
                Potential crashers should contact the instructor for meeting access

Instructor:     Tobias Höllerer
Office Hours:   Wed, 4:30pm-5:30pm, or by appointment
Office:         2155 Harold Frank Hall, (805) 284 9395
e-mail:         holl@cs...

Course Description

Mixed and Augmented Reality, an active research field since the 1990s, has recently gained significant popularity because of the possibility of being implemented on smartphones and because of its unique approach of offering context-based computing directly in a person's field of vision. Augmented Reality is the concept of overlaying computer-generated information on top of the physical world. Mixed Reality is a bit broader and subsumes the fields of Augmented Reality, Augmented Virtuality, and Virtual Reality. Recent games such as Niantic's Pokemon GO as well as Facebook/Meta Spark AR and Snap Lenses have popularized the concept, but much more is possible. In particular, applied machine learning concepts can play a strong part in this research area. Virtually all the major players in the technology sector today invest more or less heavily in this paradigm, as the expectation is that it may well be the future of mobile personal information access. This class provides a hands-on introduction to these novel interface technologies.

This is an advanced research-oriented course. Programming experience and some knowledge of computer graphics and computer vision concepts is expected.

A special focus area this quarter, promoted by the circumstances, will be remote AR/VR collaboration. We will have plenty opportunity to think about these topics from a personal perspective!

Course Requirements and Grading

There will be a series of AR/VR space creation, ideation, coding, or research paper presentation assignments that lead up to individual or group class projects. We will continuously assign reading material from various papers that we will make available. There may be one exam (in week 6 or 7) or a series of quizzes. Here is how your final grade will be determined:  

• Exam or quizzes (if held, cumulatively would be worth 30%)
• 3-5 homework assignments (if we hold an exam, these will be worth 30%, otherwise 45%)
• Final project implementation, documentation, and presentation (if we hold an exam, these will be worth 40%, otherwise 55%)
• Class participation and activities [paper presentations, pointers to relevant material] will be noted and might positively influence your grade

In case you disagree with any grade, submit your grievance in writing (email) to the grader responsible, explaining and documenting your case.

Books and Tutorials, Class Materials

• We will be handing out materials of the following textbook: Schmalstieg/Höllerer, Augmented Reality, Principles and Practice, Pearson Addison-Wesley, 2016
  
  
• Previous incarnations of this course posed some assignments using OpenCV and OpenGL, and we will go over these topics again this time:
  
  • OpenCV: Learning OpenCV, Computer Vision with the OpenCV Library Gary Bradski, Adrian Kaehler, O'Reilly Media
    Available from Safari Online (UCSB IP) here, from outside ucsb via library proxy here
    OpenCV Tutorials
  • OpenGL: Unless you want to focus on real-time or highly realistic 3D graphics in your final project, basic OpenGL knowledge will suffice to complete the class assignments. You can find much information online, e.g. at the official OpenGL web site. If you are the type of programmer who needs reference books to be effective, the two essential documentation resources that every serious OpenGL developer should have are the latest releases of:
    • The OpenGL Reference Manual
    • The OpenGL Programming Guide(edition from May 2012)
    • Available online
    
    
• Also, this quarter, you can opt to work more proactively with Unity, WebVR, and Smartphone resources (Unity has opened up their Premium learning materials for free). You need to have access to your own suitable hardware (e.g. reasonably powerful graphics computer, recent iPhone for ARKit, or ARCore-capable Android phone):
  • Unity:
    • Unity Basics - Using the Unity Interface
    • VR in Unity - Getting Started with VR (20 min)
    • Design, Develop and Deploy for VR, Unity and Oculus
    • Setting Up AR Foundation (multi-platform AR support within Unity) - (Premium)
      
    • Introduction to XR: VR, AR, and MR Foundations - (Premium)
    • Setting up Development for AR Applications, for ARKit (iOS) or ARCore (Android) - (Premium)
    • Optimizing AR/VR Experiences - (Premium)
    • Recorded Video Session: Creating AR Content With Vuforia
      
      
  • Mozilla Hubs and other webXR technologies:
    • https://hubs.mozilla.com/#/
    • TBA
    
    
  • Smartphone-based AR
    • ARCore (Google Android)
    • ARKit (Apple)
  
  
• For additional class materials, please see handout column in schedule below

Lateness Policy

All assignments are due at midnight on the scheduled due date. To make the deadlines more manageable, each student will be allowed three ``late days'' during the quarter for which lateness will not be penalized. Late days may be applied to all assignments but not the final project! Your  late days may be used as you see fit -- one or multiple per assignment -- but once you used a late day it's good and gone, you cannot reapply it to another assignment. Anything turned in after 12:00:00am until midnight the next day is one day late. Every day thereafter that an assignment is late, including weekends and holidays, counts as an additional late day.

Absolutely no late work will be accepted after the deadline if you have used up all your late days. If you're not done on time you must turn in what you have to receive partial credit. There will be no exceptions from this rule. Please make sure you understand this policy.

When making use of your late days, the online submission provides the timestamp that counts.

Academic Misconduct

We will strictly enforce UCSB's academic misconduct policies. We use electronic tools to detect plagiarism among submitted homework solutions and sources from the internet. Read these guidelines before beginning each programming assignment. Any form of plagiarism, collusion, or cheating will result in an "F" in this course and may result in suspension from UCSB for two quarters. When in doubt about any forms of receiving help on your assignments, ask us!

Open Door Policy

I would like the course to be informative and enjoyable. Let me know what you find good and interesting about the course. Let me know sooner if you feel something could be improved. See me (in virtual office hours), message on Slack, or send an e-mail.

Schedule:

Assignments

Every student taking this class needs to fill in the Student Info Questionnaire

HW assignment 1, discussed in class session 2, will be due Tue, Jan 11th, 12:00pm (for the results to be discussed in class session 3. We'll compare notes on your explorations). You can still use late day(s), but hopefully we'll have sufficient experience for a discussion.

HW assignment 2, is a brief product review exercise that will be due Tue, Jan 18th, 12:00pm (we'll have an interactive show and tell in class C5). You can still use late day(s), but hopefully we'll have sufficient submissions for the session.

HW assignment 3, is a paper review exercise (loosely modeled after peer review in the field) that will be due Mon., Jan 24th, 11:59:59pm (we'll have a discussion of your picks in class C7). You can still use late day(s), but hopefully we'll have sufficient submissions for the session.

Project Information: This page lists general considerations and interim deadlines (HW4, HW5) for your group or individual quarter projects.

HW assignment 4, is a two-step project formation and technology exploration exercise, due on Mon., Feb. 7th Wed., Feb. 9th, 11:59:59pm

HW assignment 5, is a two-step MVP demonstration and roadmap definition exercise, due on Wed, Feb. 23, 11:59:59pm