Spring 1996, CMPSC-290I (240B) , 4 units, Code 47183
Parallel Computing and Program Parallelization

Instructor: Prof. Tao Yang, Department of Computer Science (tyang@cs.ucsb.edu, office Eng I 2151, phone 4384.) Office hours: Monday 9-11:00.

Location and time: MW 11:00-12:15, Engr. 1132.

HTTP: //www.cs.ucsb.edu/tyang/class/290I/

Topics:

• Introduction to parallel computing. Parallel programming on the Meiko CS-2 and networked workstations.
• Dependence analysis and program transformation.
• Memory hierarchies and data locality. Program and data partitioning.
• Applications (parallel numerical linear algebra such as dense and sparse matrix computations, WWW servers).
• Task scheduling, load balancing. Graph partitioning.
• Parallel programming tools, libraries and languages

Course materials:

1) Selected papers on parallel computing and program parallelization will be available in the Alternative copy shop on campus under name CS290I-Yang (6556 Pardall Road, Isla Vista, Phone 968-1055).

2) On-line references:

James Demmel, Numerical Linear Algebra on Parallel Processors, 1995. An on-line reference book on parallel scientific computing and applications. (http://www.cs.berkeley.edu/demmel/cs267). To be published by SIAM.

Ian Foster, Designing and Building Parallel Programs. 1995, Addison-Wesley Publishing. This book is available on-line through the Internet (http://www.mcs.anl.gov/dbpp).

Expected work: Homework assignments and a group project.

Lecture Plan:

1. Introduction to Parallel Computing. Communication Model. Task Computation Model. SPMD Parallel Programming.
2. Parallel Programming on Meiko CS-2 and networked workstations using PVM.
3. Program dependence analysis. Program and data partitioning.
4. Memory hierarchies and performance impact.
5. High performance libraries. LAPACK and ScaLAPACK.
6. Programming tools. Parallel Khoros. RAPID.
7. 290I Project description.
8. Sparse matrix computations.
9. Static Scheduling. PYRROS. Illinios work.
10. Run-time support. RAPID.
11. Dynamic load balancing and scheduling. SWEB/Cilk.
12. 290I Project midterm presentation.
13. Parallel applications (Simulations, PDE).
14. Graph partitioning.
15. CHAOS/LPARX/Multipole.
16. Parallelizing compilers. Program transformation.
17. Programming languages.
18. More on sparse matrix computations.
19. 290I Project presentation.

References:

• Introduction to Parallel Computing. Communication Model. Task Computation Model. SPMD Parallel Programming. Parallel Programming on Meiko CS-2 and networked workstations using PVM. Program dependence analysis, Program and data partitioning.

  See 290 Lecture notes by Yang.

• Memory hierarchies and performance impact. Demmel Chapter 3.
• High performance libraries. LAPACK and ScaLAPACK. Demmel Chapter 13.
• Programming tools. Parallel Khoros. RAPID. CHAOS/Multipole.

  C. Lee, Y.-F., Wang, and T. Yang, Static Global Scheduling for Optimal Computer Vision and Image Processing Operations on Distributed-Memory Multiprocessors, Proc. of 6th International Conference on Computer Analysis of Images and Patterns, Sept. 1995. pp.920-925.

  Partitioning and Scheduling for Image Processing Operations, C. Lee, T. Yang, and Y.-F., Wang, Proc. of IEEE Symp. on Parallel and Distributed Processing, Texas, Oct. 1995. pp 86-90.

  C. Fu and T. Yang, Run-time Compilation for Parallel Sparse Matrix Computations, To appear in Proc. of the 10th ACM International Conference on Supercomputing, Philadelphia, May, 1996. http://www.cs.ucsb.edu/Research/rapid_sweb/RAPID.html

  R. Das, M. Uysal, J. Saltz, and Y. S. Hwang, Communication Optimizations for Irregular Scientific Computations on Distributed Memory Architectures. Journal of Parallel and Distributed Computing, 22, 462-478 (1994). http://www.cs.umd.edu/projects/hpsl.html.

  Runtime Support for Portable Distributed Data Structures, C. Wen, S. Chakrabarti, E. Deprit, A. Krishnamurthy and K. Yelick. Tech Report, UC Berkeley.
  http://http.cs.berkeley.edu/projects/parallel/castle/multipol/

  Irregular Coarse-Grain Data Parallelism Under LPARX, Scott R. Kohn and Scott B. Baden, February 1995. Submitted for publication to J. Scientific Programming.
  http://www-cse.ucsd.edu/groups/hpcl/scg/lparx.html.

• Static Scheduling. PYRROS. Illinios work.

  T. Yang and A. Gerasoulis, PYRROS: Static scheduling and code generation for message passing multiprocessors. Proc. of 6th ACM International Conference on Supercomputing, Washington D.C., July 1992, pp. 428-437. http://www.cs.ucsb.edu/tyang/papers/PYRROS.html.

  T. Yang, C. Fu, A. Gerasoulis and V. Sarkar, Mapping iterative task graphs on distributed-memory machines. Proc. of 24th Inter. Conference on Parallel Processing, Wisconsin, Aug. 1995. Vol II. 151-158. http://www.cs.ucsb.edu/Research/rapid_sweb/RAPID.html.

  S. Ramaswamy, S. Sapatnekar, P. Banerjee, A Convex Programming Approach for Exploiting Data and Functional Parallelism on Distributed Memory Multicomputers. Proc. of 23th Inter. Conf. on Parallel Processing, 1994. http://www.crhc.uiuc.edu/Paradigm/.

• Dynamic load balancing and Scheduling. SWEB/Cilk.

  D. Andresen, T. Yang, V. Holmedahl, O. Ibarra, SWEB: Towards a Scalable World Wide Web Server on Multicomputers To appear in Proc. of IPPS '96 - 10th International Parallel Processing Symposium, IEEE. Hawaii, April, 1996. http://www.cs.ucsb.edu/Research/rapid_sweb/SWEB.html

  Cilk: An Efficient Multithreaded Runtime System, by Robert D. Blumofe, Christopher F. Joerg, Bradley C. Kuszmaul, Charles E. Leiserson, Keith H. Randall, and Yuli Zhou. in Proceedings of the Fifth ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, July 19-21, 1995, Santa Barbara, California, pp. 207-216 (PPOPP '95). http://theory.lcs.mit.edu/cilk/.

  M. Willebeek-LeMair and A. Reeves, Strategies for Dynamic Load Balancing on Highly Parallel Computers. IEEE Trans. on Parallel and Distributed Systems, Vol 4, No 9, 1993.

• Parallel applications (Simulation, PDE, sparse matrix computations).

  Demmel, Chapters 16 and 17.

  Liu, Computational models and task scheduling for parallel sparse Cholesky factorization. Parallel Computing, 1986. 327-342.

  M. Heath and E. Ng and B. Peyton Parallel Algorithms for Sparse Linear Systems SIAM Review, 1991, Vol 33. No. 3. 420--460.

• Graph partitioning.

  Demmel, Chapters 18 and 19.

• Parallelizing compilers. Program transformation.

  Bacon, Graham, Sharp, Compiler Tranformations for high-performance computing, ACM Computing Survey, 1994, Vol 26 No 4.

  Lecture slides (Yang).

• Programming languages.

  Ian Foster, Designing and Building Parallel Programs. Chapter 7.

  Demmel, Chapters 7 and 8.

CS290I/240B Lecture Notes

Available at the Alternative copy shop under name CS290I-Yang (6556 Pardall Road, Isla Vista, Phone 968-1055).

• Lecture notes on parallelizing scientific programs (Yang).
• Parallel programming on the Meiko CS-2 machine and networked workstations (Yang).
• Lecture notes by James Demmel.
  • Introduction to Parallel Computing
  • Designing fast linear algebra kernels in the presence of memory hierarchies
  • Overview of parallel architectures and programming models
  • Shared Memory Programming with Multithreading.
  • LAPACK and ScaLAPACK.
  • Sources of Parallelism and Locality in Simulation.
  • Graph Partitioning.

• Selected papers:
  1. A. Gerasoulis and T. Yang, Efficient Algorithms and a Software Tool for Scheduling Parallel Computation, a book chapter in Scheduling Theory and Applications, P. Chretienne, E. Coffman, J.K. Lenstra and Z. Liu (Eds.). J. Wiley Publisher, 1995. pp. 111-143.
  2. C. Fu and T. Yang, Efficient Run-time Support for Irregular Task Computations with Mixed Granularities, To appear in Proc. of IPPS '96 - 10th International Parallel Processing Symposium, IEEE. Hawaii, April, 1996.

  3. J. Liu, Computational models and task scheduling for parallel sparse Cholesky factorization. Parallel Computing, 1986. 327-342.
  4. S. Ramaswamy, S. Sapatnekar, P. Banerjee, A Convex Programming Approach for Exploiting Data and Functional Parallelism on Distributed Memory Multicomputers. Proc. of 23th Inter. Conf. on Parallel Processing, 1994.
  5. M. Willebeek-LeMair and A. Reeves, Strategies for Dynamic Load Balancing on Highly Parallel Computers. IEEE Trans. on Parallel and Distributed Systems, Vol 4, No 9, 1993.
  6. D. Andresen, T. Yang, V. Holmedahl, O. Ibarra, SWEB: Towards a Scalable World Wide Web Server on Multicomputers, To appear in Proc. of IPPS '96 - 10th International Parallel Processing Symposium, IEEE. Hawaii, April, 1996.
  7. Cilk: An Efficient Multithreaded Runtime System, by R. D. Blumofe, C. F. Joerg, B. C. Kuszmaul, C. E. Leiserson, K. H. Randall, and Y. Zhou. in Proc. of the Fifth ACM Sym. on Principles and Practice of Parallel Programming, 1995, Santa Barbara, pp. 207-216. (PPOPP '95).
  8. Runtime Support for Portable Distributed Data Structures, C. Wen, S. Chakrabarti, E. Deprit, A. Krishnamurthy and K. Yelick. Tech Report, UC Berkeley.

  9. High Performance Fortran, Chapter 7 from I. Foster, Designing and Building Parallel Programs..
  10. Lecture slides on parallelizing compilers (Yang).

• About this document ...