Emerging parallel and distributed systems promise to economically scale aggregate computation and communication performance across a wide range of resources. Exploiting the value of today's mass market processors, these systems have demonstrated the  potential to deliver high-performance computing to a broader domain of educators and researchers. They enable relatively low-cost technologies to solve problems that, historically, could be resolved only on multimillion-dollar machines. The early successes of commodity and workstation clusters, large-scale parallel systems, and the nascent computational grid have encouraged application scientists to pursue computational models previously considered intractable.

Unfortunately for many of these next-generation applications, constraints imposed by I/O limit the level of achievable performance. The evolution of the commodity storage market, the surprising complexity of application access patterns, and the paucity of knowledge about effective I/O system design have combined to create a performance barrier that rivals or exceeds that for computation. Increasing disparities between disk capacities and transfer rates have exacerbated the performance bottleneck. With few exceptions, the commercial data storage market rewards high storage capacity, small form factors, and low power consumption rather than high bandwidth.

I/O parallelism can offset these technology trends. Involving tens to thousands of cooperating storage devices, I/O parallelism is key to enabling the level of performance promised by new scalable, parallel, distributed environments. However, when large numbers of disks and disk arrays are coupled with a multi-level storage management system and a wide variety of possible parallel file access patterns, the range of potential data management strategies is immense. Identifying optimal, or even acceptable, operating points becomes problematic.

While recent I/O characterization studies have demonstrated that access pattern variations have deep performance implications for I/O libraries and file systems, most file system and storage hierarchy designers have little empirical data on parallel I/O access patterns. Generally, the design of today's parallel file systems is based on extant file access patterns, themselves reflecting how application developers exploited the idiosyncrasies of previous generation file systems. Furthermore, parallel file system and storage hierarchy designers are often forced to extrapolate from the access patterns generated by workstations or personal computers. Neither of these environments reflects the application usage patterns, diversity of configurations, or economic tradeoffs salient in emerging, high-performance environments.

Despite the clear need, particularly of computational scientists and educators who have much to gain from increased access to affordable high-performance systems, I/O and file system research on clusters and scalable, parallel systems remains in its infancy. Indeed, most PC and workstation clusters have no parallel file system; instead they rely on NFS or AFS for distributed file access. Likewise, integration of I/O and network quality of service measures for distributed computational grids remains a hope rather than a reality. The Pablo facility aims to address this situation by disseminating tools and experimental data for the quantitative study of parallel I/O optimization. Building on the Pablo I/O characterization toolkit, the facility provides applications developers, file system and storage hierarchy designers, educators, and other research communities with resources embodying insight gained through the research and development of Pablo tools.

This research is supported in part by the  the National Science Foundation.

Comments and Assistance

Comments, questions, and problem reports regarding the system should be sent to software@renci.org