Caltech, SLAC, and LANL Set New Network Performance Marks
PHOENIX, Ariz.--Teams of physicists, computer scientists, and network engineers from Caltech, SLAC, LANL, CERN, Manchester, and Amsterdam joined forces at the Supercomputing 2003 (SC2003) Bandwidth Challenge and captured the Sustained Bandwidth Award for their demonstration of "Distributed particle physics analysis using ultra-high speed TCP on the Grid," with a record bandwidth mark of 23.2 gigabits per second (or 23.2 billion bits per second).
The demonstration served to preview future Grid systems on a global scale, where communities of hundreds to thousands of scientists around the world would be able to access, process, and analyze terabyte-sized data samples, drawn from data stores thousands of times larger. A new generation of Grid systems is being developed in the United States and Europe to meet these challenges, and to support the next generation of high-energy physics experiments that are now under construction at the CERN laboratory in Geneva.
The currently operating high-energy physics experiments at SLAC (Palo Alto, California), Fermilab (Batavia, Illinois), and BNL (Upton, New York) are facing qualitatively similar challenges.
During the Bandwidth Challenge, the teams used all three of the 10 gigabit/sec wide-area network links provided by Level 3 Communications and Nortel, connecting the SC2003 site to Los Angeles, and from there to the Abilene backbone of Internet2, the TeraGrid, and to Palo Alto using a link provided by CENIC and National LambdaRail. The bandwidth mark achieved was more than 500,000 times faster than an Internet user with a typical modem connection (43 kilobits per second). The amount of TCP data transferred during the 48-minute-long demonstration was over 6.6 terabytes (or 6.6 trillion bytes). Typical single-stream host-to-host TCP data rates achieved were 3.5 to 5 gigabits per second, approaching the single-stream bandwidth records set last month by Caltech and CERN.
The data, generated from servers at the Caltech Center for Advanced Computing Research (CACR), SLAC, and LANL booths on the SC2003 showroom floor at Phoenix, a cluster at the StarLight facility in Chicago as well as the TeraGrid node at Caltech, was sent to sites in four countries (USA, Switzerland, Netherlands, and Japan) on three continents. Participating sites in the winning effort were the Caltech/DataTAG and Amsterdam/SURFnet PoPs at Chicago (hosted by StarLight), the Caltech PoP at Los Angeles (hosted by CENIC), the SLAC PoP at Palo Alto, the CERN and the DataTAG backbone in Geneva, the University of Amsterdam and SURFnet in Amsterdam, the AMPATH PoP at Florida International University in Miami, and the KEK Laboratory in Tokyo. Support was provided by DOE, NSF, PPARC, Cisco Systems, Level 3, Nortel, Hewlett-Packard, Intel, and Foundry Networks.
The team showed the ability to use efficiently both dedicated and shared IP backbones. Peak traffic on the Los Angeles-Phoenix circuit, dedicated to this experiment, reached almost 10 gigabits per second utilizing more than 99 percent of the capacity. On the shared Abilene and TeraGrid circuits the experiment was able to share fairly over 85 percent of the available bandwidth. Snapshots of the maximum link utilizations during the demonstration showed 8.7 gigabits per second on the Abilene link and 9.6 gigabits per second on the TeraGrid link.
This performance would never have been achieved without the use of new TCP implementations because the widely deployed TCP RENO protocol performs poorly at gigabit-per-second speed. The primary TCP algorithm used was new FAST TCP stack developed at the Caltech Netlab. Additional streams were generated using HS-TCP, implemented at Manchester, and scalable TCP.
Harvey Newman, professor of physics at Caltech, said: "This was a milestone in our development of wide-area networks and of global data-intensive systems for science. Within the past year we have learned how to use shared networks up to the 10 gigabit-per-second range effectively. In the next round we will combine these developments with the dynamic building of optical paths across countries and oceans. This paves the way for more flexible, efficient sharing of data by scientists in many countries, and could be a key factor enabling the next round of physics discoveries at the high-energy frontier. There are also profound implications for integrating information sharing and on-demand audiovisual collaboration in our daily lives, with a scale and quality previously unimaginable."
Les Cottrell, assistant director of SLAC's computer services, said: "This demonstrates that commonly available standard commercial hardware and software, from vendors like Cisco, can effectively and fairly use and fill up today's high-speed Internet backbones, and sustain TCP flows of many gigabits per second on both dedicated and shared intracountry and transcontinental networks. As 10 gigabit-per-second Ethernet equipment follows the price reduction curve experienced by earlier lower-speed standards, this will enable the next generation of high-speed networking and will catalyze new data-intensive applications in fields such as high-energy physics, astronomy, global weather, bioinformatics, seismology, medicine, disaster recovery, and media distribution."
Wu-chun (Wu) Feng, team leader of research and development in Advanced Network Technology in the Advanced Computing Laboratory at LANL, noted: "The SC2003 Bandwidth Challenge provided an ideal venue to demonstrate how a multi-institutional and multi-vendor team can quickly come together to achieve a feat that would otherwise be unimaginable today. Through the collaborative efforts of Caltech, SLAC, LANL, CERN, Manchester, and Amsterdam, we have once again pushed the envelope of high-performance networking. Moore's law move over!"
"Cisco was very pleased to help support the SC2003 show infrastructure, SCINET," said Bob Aiken, director of engineering for academic research and technology initiatives at Cisco. "In addition, we also had the opportunity to work directly with the high-energy physics (HEP) research community at SLAC and Caltech in the United States, SURFnet in the Netherlands, CERN in Geneva, and KEK in Japan, to once again establish a new record for advanced network infrastructure performance.
"In addition to supporting network research on the scaling of TCP, Cisco also provided a wide variety of solutions, including Cisco Systems ONS 15540, Cisco ONS 15808, Cisco Catalyst 6500 Series, Cisco 7600 Series, and Cisco 12400 Series at the HEP sites in order for them to attain their goal. The Cisco next-generation 10 GE line cards deployed at SC2003 were part of the interconnect between the HEP sites of Caltech, SLAC, CERN, KEK/Japan, SURFnet, StarLight, and the CENIC network."
"Level 3 was pleased to support the SC2003 conference again this year," said Paul Fernes, director of business development for Level 3. "We've provided network services for this event for the past three years because we view the conference as a leading indicator of the next generation of scientific applications that distinguished researchers from all over the world are working diligently to unleash. Level 3 will continue to serve the advanced networking needs of the research and academic community, as we believe that we have a technologically superior broadband infrastructure that can help enable new scientific applications that are poised to significantly contribute to societies around the globe."
Cees de Laat, associate professor at the University of Amsterdam and organizer of the Global Lambda Integrated Facility (GLIF) Forum, added: "This world-scale experiment combined leading researchers, advanced optical networks, and network research sites to achieve this outstanding result. We were able to glimpse a yet-to-be explored network paradigm, where both shared and dedicated paths are exploited to map the data flows of big science onto a hybrid network infrastructure in the most cost-effective way. We need to develop a new knowledge base to use wavelength-based networks and Grids effectively, and projects such as UltraLight, TransLight, NetherLight, and UKLight, in which the team members are involved, have a central role to play in reaching this goal."
###
About Caltech: With an outstanding faculty, including four Nobel laureates, and such off-campus facilities as the Jet Propulsion Laboratory, Palomar Observatory, and the W. M. Keck Observatory, the California Institute of Technology is one of the world's major research centers. The Institute also conducts instruction in science and engineering for a student body of approximately 900 undergraduates and 1,000 graduate students who maintain a high level of scholarship and intellectual achievement. Caltech's 124-acre campus is situated in Pasadena, California, a city of 135,000 at the foot of the San Gabriel Mountains, approximately 30 miles inland from the Pacific Ocean and 10 miles northeast of the Los Angeles Civic Center. Caltech is an independent, privately supported university, and is not affiliated with either the University of California system or the California State Polytechnic universities. http://www.caltech.edu
About SLAC: The Stanford Linear Accelerator Center (SLAC) is one of the world 's leading research laboratories. Its mission is to design, construct, and operate state-of-the-art electron accelerators and related experimental facilities for use in high-energy physics and synchrotron radiation research. In the course of doing so, it has established the largest known database in the world, which grows at 1 terabyte per day. That, and its central role in the world of high-energy physics collaboration, places SLAC at the forefront of the international drive to optimize the worldwide, high-speed transfer of bulk data. http://www.slac.stanford.edu/
About LANL: Los Alamos National Laboratory is operated by the University of California for the National Nuclear Security Administration of the U.S. Department of Energy and works in partnership with NNSA's Sandia and Lawrence Livermore National Laboratories to support NNSA in its mission. Los Alamos enhances global security by ensuring the safety and reliability of the U.S. nuclear weapons stockpile, developing technical solutions to reduce the threat of weapons of mass destruction, and solving problems related to energy, environment, infrastructure, health, and national security concerns. http://www.lanl.gov/
About Netlab: Netlab is the Networking Laboratory at Caltech led by Professor Steven Low, where FAST TCP has been developed. The group does research in the control and optimization of protocols and networks, and designs, analyzes, implements, and experiments with new algorithms and systems. http://netlab.caltech.edu/FAST/
About CERN: CERN, the European Organization for Nuclear Research, has its headquarters in Geneva. At present, its member states are Austria, Belgium, Bulgaria, the Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Italy, the Netherlands, Norway, Poland, Portugal, Slovakia, Spain, Sweden, Switzerland, and the United Kingdom. Israel, Japan, the Russian Federation, the United States of America, Turkey, the European Commission, and UNESCO have observer status. For more information, see http://www.cern.ch.
About DataTAG: The European DataTAG is a project co-funded by the European Union, the U.S. Department of Energy through Caltech, and the National Science Foundation. It is led by CERN together with four other partners. The project brings together the following European leading research agencies: Italy's Istituto Nazionale di Fisica Nucleare (INFN), France's Institut National de Recherche en Informatique et en Automatique (INRIA), the U.K.'s Particle Physics and Astronomy Research Council (PPARC), and the Netherlands' University of Amsterdam (UvA). The DataTAG project is very closely associated with the European Union DataGrid project, the largest Grid project in Europe also led by CERN. For more information, see http://www.datatag.org.
About StarLight: StarLight is an advanced optical infrastructure and proving ground for network services optimized for high-performance applications. Operational since summer 2001, StarLight is a 1 GE and 10 GE switch/router facility for high-performance access to participating networks and also offers true optical switching for wavelengths. StarLight is being developed by the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago (UIC), the International Center for Advanced Internet Research (iCAIR) at Northwestern University, and the Mathematics and Computer Science Division at Argonne National Laboratory, in partnership with Canada's CANARIE and the Netherlands' SURFnet. STAR TAP and StarLight are made possible by major funding from the U.S. National Science Foundation to UIC. StarLight is a service mark of the Board of Trustees of the University of Illinois. See www.startap.net/starlight.
About the University of Manchester: The University of Manchester, located in the United Kingdom, was first granted a Royal Charter in April 1880 as the Victoria University and became the first of the U.K.'s great civic universities. As a full-range university it now has more than 70 departments involved in teaching and research, with more than 2,000 academic staff. There are more than 18,000 full-time students, including 2,500 international students, from over 120 countries studying for undergraduate and postgraduate level degrees. The University of Manchester has a proud tradition of innovation and excellence which continues today. Some of the key scientific developments of the century have taken place here. In Manchester, Rutherford conducted the research which led to the splitting of the atom and the world's first stored-program electronic digital computer, built by Freddie Williams and Tom Kilburn, successfully executed its first program in June 1948. The departments of Physics, Computational Science, Computer Science and the Network Group together with the E-Science North West Centre research facility are very active in developing a wide range of e-science projects and Grid technologies. See www.man.ac.uk.
About National LambdaRail: National LambdaRail (NLR) is a major initiative of U.S. research universities and private sector technology companies to provide a national scale infrastructure for research and experimentation in networking technologies and applications. NLR puts the control, the power, and the promise of experimental network infrastructure in the hands of the nation's scientists and researchers. Visit http://www.nationallambdarail.org for more information.
About CENIC: CENIC is a not-for-profit corporation serving California Institute of Technology, California State University, Stanford University, University of California, University of Southern California, California Community Colleges, and the statewide K-12 school system. CENIC's mission is to facilitate and coordinate the development, deployment, and operation of a set of robust multi-tiered advanced network services for this research and education community. http://www.cenic.org
About University of Amsterdam: The Advanced Internet Research group of the University of Amsterdam's Faculty of Science researches new architectures and protocols for the Internet. It actively participates in worldwide standardization organizations Internet Engineering Task Force and the Global Grid Forum. The group conducts experiments with extremely high-speed network infrastructures. The Institute carries out groundbreaking research in the fields of security, authorization, authentication and accounting for grid environments. The Institute is developing a virtual laboratory based on grid technology for e-science applications. For more information see http://www.science.uva.nl/research/air>www.science.uva.nl/research/air.
