Already in 2013, CERN participated in the testing of the first prototypes of helium drives, exceeding expectations in terms of power consumption and storage density. Over the past two decades, Western Digital has supplied CERN with several petabytes of very powerful and reliable drives for data storage. […]
CERN (the European Organization for Nuclear Research) is an intergovernmental organization maintained by 23 Member States and headquartered in Geneva, with facilities on both sides of the Franco-Swiss border. The mission and purpose of CERN is international cooperation in the field of high-energy particle physics.
CERN’s IT data center operates separate compute and storage pools with approximately 11,000 servers and 470,000 compute cores connected to approximately 350,000 terabytes of raw disk space on more than 100,000 hard disk drives and approximately 400,000 terabytes of tape-based storage capacity used.
In order to further increase the HDD storage density per rack and to optimize the cost/TB ratio as well as infrastructure costs, CERN will use the Ultrastar Data102 from Western Digital from 2021, which can accommodate 102 drives and offers over 2 PB of storage space in a 4U format.
The flagship of CERN is the Large Hadron Collider (LHC), which is located 100 m underground in a tunnel with a circumference of 27 km. It accelerates proton beams and allows heavier ions, including lead, to collide. Everything that happens in these collisions is recorded. This results in a data volume of about 1 petabyte. Extremely fast computer farms connected directly to the LHC collect and filter this data to send it to CERN’s IT data center via multiple Ethernet connections, where it is stored and processed.
With an increase in collision data by two orders of magnitude compared to the previous run, run 3 required even faster and larger storage solutions. To store the data coming from the collider, CERN needed high-capacity storage systems capable of reading and writing data at a rate of 12.5 GB/s in any direction.
Theoretically, the goal of 12.5 GB/s is achievable for nearline drives with a data throughput of 265 MB/s with 48 drives.
However, if many drives are combined in very dense storage enclosures, the temperature and the vibrations generated by the drives themselves can significantly reduce the overall performance.
The solution: Ultrastar Data60 storage platform
To meet these requirements, CERN IT extensively tested the Data60 JBODs from Western Digital. Each JBOD is equipped with 60x14TB Ultrastar SAS HDDs and connected to a front-end server via 4X12GBIT/s SAS links. The main distinguishing feature of the Data60 JBOD is that each drive can be operated at maximum performance thanks to ArticFlow and patented IsoVibe technology. These solve the dual challenge of vibration and effective cooling that can occur with tight storage enclosures and affect performance and reliability.
As for thermal management, the reliability of a drive can decrease with increasing temperature, and performance is throttled for protection. The ArcticFlow Thermal Zone Cooling Technology introduces cool air into the center of the case, allowing the drives to operate at lower and more constant temperatures than conventional systems. This results in lower fan speeds, less vibration, lower power consumption, quieter operation and ultimately higher reliability.
Adjacent drives operating in a dense array can cause vibration in the adjacent drives, resulting in performance degradation. The precise cuts of the ISOVIBE Vibration Isolation technology in the baseboard provide a suspension for the drives in the housing and isolate them from the transmitted vibrations. The result is consistent performance, even when all drives are under heavy load.
Western Digital’s Data60 in conjunction with the Linux kernel Device Mapper Multipathing has helped CERN achieve the desired throughput of 12.5 GB/s and successfully upgrade part of the LHC storage facility with Western Digital’s Data60 in 2020.