Scalability Issues for High Performance Digital Libraries on theWorld Wide Web

We investigate scalability issues involved in developing high performancedigital library systems. Our observations and solutions are based on ourexperience with the Alexandria Digital Library (ADL) testbed under developmentat UCSB. The current ADL system provides on-line browsing and processing ofdigitized maps and other geo-spatially mapped data via the World Wide Web(WWW). A primary activity of the ADL system involves computation and disk I/Ofor accessing compressed multi-resolution images with hierarchical datastructures, as well as other duties such as supporting database queries andon-the-fly HTML page generation.Providing multi-resolution image browsing services can reduce network trafficbut impose some additional cost at the server. We discuss the necessity ofhaving a multi-processor DL server to match potentially huge demands insimultaneous access requests from the Internet. We have developed adistributed scheduling system for processing DL requests, which activelymonitors the usages of CPU, I/O channels and the interconnection network toeffectively distribute work across processing units to exploit task and I/Oparallelism. We present an experimental study on the performance of our schemein addressing the scalability issues arising in ADL wavelet processing and fileretrieval. Our results indicate that the system delivers good performance onthese types of tasks.