CS185 Homework Assignment 3:
UI Evaluation / Data Entry

Posted:   Wed, April 17th
Due:        Tue, April 23nd  (23:59:59)

Change Log:
April 17th, 5:30pm: Added submission format description

Please start implementation and user study data collection ASAP. In class, we cover user study background information and statistical analysis material, as well as more information for the critiquing part.
 

Objective:

Practice basic comparative user-based interface testing and UI critiquing. We will later analyze the results in class and contrast this approach with a specific quantitative analysis methodology, the GOMS Keystroke-Level Model (KLM) (see handout H14).

Instructions:

Your task is to evaluate the time efficiency (assuming expert users) of two different user interfaces for entering field observation data for an environmental science application (plant monitoring): Date and time of observation, as well as plant type and measurements need to be entered to a web application. Both interfaces will be given to you as compiled React web applications.

The assignment has 2 parts (explained in detail below):

• Part 1) You will be running a simple user study, testing the two data entry interfaces against each other with actual users. You will have to recruit at least 2 volunteers (outside of this class) to do this and also be a participant yourself.
• Part 2) You will critique both user interfaces, taking into account web design guidelines and general UI principles.

Assume that you have a large set of plant observations (date/time of observation; plant location, plant type; and plant measurements) as handwritten entries. Each line should be clearly legible and easily understood by you (assume you wrote it), but you didn't use standard notation, so there is some variance.

Example entries could read:

evergreen broadleaf shrub, April 1st '18, UCSB, N 3.095, C 51.143, LA 0.899
tree, decid., broadl. - 2018/04/01 - SERA - 3.579 Nitro, 49.216 Carbon, 0.88 LeafArea

For this homework, we actually simplify the scenario by giving you the data in a standardized format, e.g.:

Type 1	Type 2	Type 3	Date	Time	Site	Nitrogen	Carbon	LeafArea
shrub	evergreen	broadleaf	2018_04_01	4_06 PM	UCSB	3.095	51.143	0.899
shrub	evergreen	needleleaf	2018_04_01	4_12 PM	UCSB	2.926	52.895	0.527
tree	deciduous	broadleaf	2018_04_01	4_20 PM	SERA	3.597	49.216	0.88

Assume that the ultimate task would be to enter a large set of these entries (thousands), and that you need to find an efficient user interface for doing so.

 

Part 1 (UserStudy):

Here are the two web data entry interfaces:

Interface 1

Interface 2

Please use the Google Chrome browser for this user study. Different browsers do present different interfaces on the same app.

You will have to recruit at least 2 volunteers (outside of this class) to do this user study.

1. Use the following simple shell script commands on a CSIL computer (or your own Unix shell) to generate a list of 10 random different entries from the overall list for your volunteers to enter as test cases:
      sort -Ru ~holl_class/public/EntryRecords.txt | head -10 > TenRandomEntries.txt
      cat ~holl_class/public/TitleRow.txt TenRandomEntries.txt > TenRandomEntriesA.txt
   
   Randomize the same 10 entries again to create a second list:
         sort -Ru -o TenRandomEntries.txt TenRandomEntries.txt
      cat ~holl_class/public/TitleRow.txt TenRandomEntries.txt > TenRandomEntriesB.txt
   
   Then print the resulting A and B txt files or copy them legibly on a piece of paper. Use list A for the first condition for all your 3 subjects, and list B for the second condition (note that, according to the next point, 3. below, you will vary what condition is first and which one second. Make sure all your volunteers receive the lists in exactly the same format, and also make sure to include the TenRandomEntriesA.txt and TenRandomEntriesB.txt files in your homework submission.
   
   We also link here the files EntryRecords.txt and TitleRow.txt
    
1. Recruit at least 2 different people (plus yourself, making for a total of at least 3 volunteers) to compare the two interfaces on this list. This should be a within-subjects study, and you should control for possible learning effects by altering the order of which interface you expose your subjects to first.  
    
2. For each person and system, measure and log the total time (in seconds) it takes to enter all 10 entries. You will have to use a timer/stopwatch. Observe the interactions ("trials") of your subjects with the different systems and if your volunteers make an error or get stuck, please start over or stop the timer to exclude wasted time. We are interested in speed-oriented performance. Don't allow your volunteers to see how long they took until after they filled in the post-study questionnaire.
   
   Vary the order in which your subjects use the two interfaces to mitigate learning effects.  You should make yourself the first test subject, because for your instructions to and observations of your friends you need to familiarize yourself with the capabilities of each interface.
   
   For each user, you have to control for their familiarity with a specific system, so use this pre-study questionnaire to assess your/their "expertise" with environmental science and their familiarity with logging apps before they perform their task.
   
   Please explain each interface to each of your volunteers, before you time them on it. They should use the fastest way possible to enter the data (which may include some experimentation on your part). Give them a chance to use each interface on a few extra random examples before you start timing them on the set of 10.   
    
3. After you timed the performance, make sure to download the entered data log file(s) via the download button. These will need to be part of your submission, comprising a total of 2x30 records [2 interfaces x 10 trials x 3 participants (including you), in minimally 6 and maximally 60 different files].
   
   
4. Ask your volunteers to fill out this post-study questionnaire (and answer it yourself after your own trial).
    
5. Collect all your data into an LibreOffice Calc spreadsheet of the following format: FieldWorkUserStudy.ods (or in Excel format: FieldWorkUserStudy.xls). In the 'Order: Interfaxce 1 first' column, please put a 'Y' if for this row's user you tested interface 1 first, and a 'N' if you tested it second. Please also enter the information from the pre- and post-questionnaires in the corresponding columns. 
    
6. Analyze the data and make statistical conclusions about the user interfaces of the systems. Perform a T-test (using LibreOffice Calc), and state what the finding was in a separate document.  As detailed in class, perform a Student's T-test on the time measurements  in LibreOffice Calc, by selecting an empty spreadsheet cell, going to Menu item 'Insert->Function...' and selecting TTEST. Enter the two data columns (your measurements in columns B and C), and select a two-tailed paired TTEST, as explained in class. Make sure you explain the resulting value in your Results.txt/pdf document corresponding to bullet 4 of Part 1 in the hand-in section below.  Please DO report any findings from the post-study questionnaire answers there as well. You DO NOT have to run another statistical analysis from these, but please report any trends that you may spot in these results. Be aware that the appropriate statistical analysis of Likert Scale questions (such as we used in the questionnaires) is more complex than the analysis of a simple continuous variable.
   

Part 2 (UI Critiquing):

Using your observations from your small user study and your knowledge of Web Design guidelines and general UI design principles, please critique both interfaces we gave you and make improvement suggestions. Please, specifically consider the "8 golden rules" of handout H11, the UI principles from the Universal Principles of Design Book already discussed (H12-H13, H15-H17), as well as modern Material Design Guidelines (H14).

Submitting Your Homework