CS170, Operating Systems

Project 1: User Shell and Lottery Scheduling



Summary
  • SiSH and Lottery Due April 23, 2011, 11:59:59PM
  • To be completed in teams of two
  • Includes two separate components, one inside Minix, one outside.

Now that you've all gotten familiar with Minix, we can start making more significant changes to Minix.  We will start with the Shell and the process scheduler in Minix.  In this assignment, we will implement a user shell that provides most functionalities present in common shells, then we will modify the Minix process scheduler to add Lottery scheduling, a cool scheduling algorithm that's not support by Minix.

Writing a Simple Shell: sish

While the goal is to change Minix, you can do this assignment in either Minix or on Linux.  Building the shell in Minix is going to be more difficult than in Linux, so we will give an extra credit of 5% on your project if your shell works in Minix.  However, my recommendation is to start with Linux to build and debug the core shell functionality, and only worry about the Minix port once you're satisfied with its performance in Linux.

The overall job of the shell program is to take input from the user, parse it, and perform the program or commands specified by the user.  In addition to executing programs the user specifies, the shell also provides support for functionality such as input/output redirection, pipelining output from one program into another, and putting processes into the background.

The basic structure of the shell is essentially a long loop that parses user input.  When running normally, the shell should always present the user with the prompt "sish:>" before each command is executed. In cases where STDIN is not a TTY (input redirection during invocation of your shell), no prompt should be displayed. To help you with basio I/O, you can make use of the fgets() function to read in lines from stdin.  While parsing each line of input, you will need to support the following tokens/functionality:
  • Basic commands:  The user can type in basic commands which are simple alphanumeric tokens that specify a command, e.g. ls, cat.   You should access the PATH environmental variable, and then search in the specified directories for the specified command.  Once found, the command should be run in a new process, and the shell should block while waiting for the command to finish (see below for exceptions). 
  • Command line arguments: Each command can have command line arguments.  You'll need to read in these arguments, and pass them along to the command you're running.  Arguments are any text tokens that follow the command but are separated by white space.
  • Input/output redirection: your shell will need to understand i/o redirection, as directed by special characters <, >, and |:
    • <:  The character '<' must be followed by a token that represents a file name. It means that the command before this character does not take input from the stdin of the shell, but instead reads its input from this file.  '<' must follow the name of a command. Note that white space or white space characters between '<' and the text before and after it is optional.  For example, cat < file and cat<file are equivalent.
    • >: The '>' character does the opposite. cmd > x means that output from the command will be written out to file X instead of being written to stdout.  The token before '>' must be a command, and the token after '>' is a filename.  Again, the same rules regarding white spaces and white space characters apply as above.
    • |: The '|' (i.e., pipe sign) allows multiple commands to be connected. The output of the command before the pipe sign must be connected to the input of the command after the pipe sign. This requires that there is a valid command both before and after the pipe. There can be multiple pipe signs on the command line. For example, your shell has to be able to process an input such as cat f | sort | wc. With this command, the output of the cat command is redirected to the input of sort, which in turn sends its output to the input of the wc program. The same rules as above applies to white space and white space characters.
    • Simplifications / clarifications
      • Only the first command in a line can read in input from a file
      • Only the last command in a line can output to a file
      • A single command can have both input and output redirection, e.g. sort < infile > outfile
      • A single command cannot have two sources of input or two files for output, e.g. cat | sort < file,  cat x > file1 > file2 are not allowed.
  • Background processes:   The ampersand character '&' indicates that the command (or commands) specified should be executed in the background. The shell immediately displays a prompt to wait for the next line, even though commands on the previous line might not have exited yet. The '&' token may only appear as the last token of a line.
  • Exiting the shell: the shell exits when the user input the 'exit' command (without quotes of course), or the ctrl-D or EOF character.
  • White space characters: Any non-alphanumeric characters that are not one of the specialized characters described above is a white-space character.  For example, invisible control characters like ^Q should be treated like blank space.
  • Error handling: your shell should be robust to unexpected input. If there is unexpected input or errors that are generated by programs, the shell should output a message to standard out prefixed with "ERROR:".  No input should be able to kill your shell or make it hang.
Additional Important Notes
  • All shell output must go to standard out, however, it is okay if a command launched by your shell produces output on standard error.
  • All errors (syntax, parsing, command not found) must be prefixed with "ERROR:" (less the quotes) and cannot exceed a single line.
  • The source file for your shell must be named "shell.c" and must compile with this makefile.
  • Your shell should produce this exact output (the error message on line 8 can be different as long as the ERROR: prefix is there) when using this input file via: ./shell < shell_test_input

Hints
  • You should make use of the following system calls: fork(), exec/execvp(), wait/waitpid(), dup2/dup(), pipe() and close()
  • Your input line will not be longer than 1024 characters

Adding a Lottery Scheduler to Minix

This portion of Project 1 requires you to change the process scheduler functionality inside Minix.  Obviously, you will need to modify Minix kernel code to make this happen.  Specifically, you will be adding functionality to support Lottery Scheduling for user processes. Lottery scheduling is a flexible way to schedule CPU time by giving each process a number (n) of tickets.  Each time the scheduler is called, it looks at the total number of tickets outstanding (i.e. held by all processes).  It randomly chooses one of the outstanding tickets, and the process holding that ticket is the process that gets to run.  The intuition is that processes can increase or decrease their share of the CPU by getting or giving away tickets.

Modifying the scheduler for Minix should mostly involve modifying code in kernel/proc.c , specifically thesched() and pick_proc() functions (and perhaps enqueue() and dequeue() ). You may also need to modify kernel/proc.h to add elements to the proc structure and modify queue information (NR_SCHED_QUEUES, TASK_Q, IDLE_Q, etc.) and may need to modify PRIO_MIN  and PRIO_MAX in/usr/include/sys/resource.h. Process priority is set in do_getsetpriority()  inservers/pm/misc.c (don't worry—the code in here is very simple), which calls do_nice()  inkernel/system.c. You might be better off just using the nice() system call , which calls do_nice()directly. You'll probably want to modify what do_nice() does to assign or take away tickets.

The current MINIX scheduler is relatively simple. It maintains 16 queues of "ready" processes, numbered 0-15. Queue 15 is the lowest priority (least likely to run), and contains only the IDLE task. Queue 0 is the highest priority, and contains several kernel tasks that never get a lower priority. Queues 1–14 contain all of the other processes. Processes have a maximum priority (remember, higher priorities are closer to 0), and should never be given a higher priority than their maximum priority.  System processes (queues 0-14) are run using their original algorithm, and queue 15 still contains the idle process. However, queue 16 contains all of the runnable user processes, each of which has some number of tickets. The default number of tickets for a new process is 5. However, processes can add or subtract tickets by calling setpriority(ntickets), which will increase the number of tickets by ntickets (note that a negative argument will take tickets away). A process cannot accumulate more than 100 tickets.

Each time the scheduler is called, it should randomly select a ticket (by number) and run the process holding that ticket. Clearly, the random number must be between 0 and Tickets-1, where Tickets is the sum of all the tickets belonging to processes in the ready queue (processes that are blocked are ineligible to run). You may use the random() call (you may need to use the random number code in/usr/src/lib/other/random.c) to generate random numbers and the srandom() call to initialize the random number generator. A good initialization function to use would be the current date and time.

New processes are created and initialized in kernel/system/do_fork.c. This is probably the best place to initialize any data structures.
Notes
 You need only implement the lottery scheduler portion for user processes. Kernel tasks and server processes can and should continue to use the existing priority scheduler. Priority should be given to kernel tasks and server processes before the user process lottery scheduler.


Submission Process
The submission process for all 170 projects uses the turnin package along with Bryce's special turnin_wrapper. turnin_wrapper is a simple script that in addition to submitting your files via turnin, sends a special email to the cs170 account informing it to verify your submission. A short while following your submission (usually within a minute), you will receive an email (to your @cs.ucsb.edu email) informing you whether or not your submission passed the verification or not.

Your submission must be submitted prior to the deadline, and pass verification in order to be graded (verification time does not count toward the deadline). Ungraded submissions will receive a 0. To find out more about the turnin_wrapper view its source code (less ~cs170/turnin_wrapper.py), and do a man turnin to find more info about the turnin program.

To submit:
  • Ensure your current working directory is a directory containing:
    • shell.c -- the file containing the code for your shell that runs on linix.
    • a file named patch that contains your changes to the Minix source. The patch must apply cleanly to a fresh source code tree and be run on your dev machine via: 
      diff -ruNp minix_src_clean/ proj1/ > patch
    • an optional minix_shell.c which contains your shell ported to minix for extra credit
  • Execute the turnin program: 
    ~cs170/turnin_wrapper.py proj1@cs170 patch shell.c
  • Check your email to ensure the status says verification passed. You will immediately see your tentative grade for the shell portion of the project. The minix portion of the project will need to be demoed.

You can execute turnin as many times as required. The most recent PASSED submission prior to the deadline will be used for grading. You MUST inform Bryce via an email prior to the deadline if you intend on using your two day extension for this project.