1st Exa-scale system and Europe efforts in HPC

Enric Gibert

By Enric Gibert

On May 30th 2022, the Top500 High-Performance Computing (HPC) list revealed the first exa-scale HPC system ever constructed: the Frontier system at Oak Ridge National Laboratory (ORNL) in Tennessee, United States.

Why is TOP500?

The TOP500 is a non-profit group of experts and Key-Opinion leaders that compiles and processes information on HPC systems and makes all this information available to the HPC community. The group is formed by Erich Strohmaier, Jack Dongarra (I remember reading his papers when I was an undergrad in the 90s), Horst Simon and Martin Meuer.

Among other useful information, twice a year the group compiles the list of the 500 most powerful HPC systems based on their achieved performance (achieved versus peak performance is discussed later in this article). Information compilation is a collaborative effort between these experts and the different institutions involved. Hence, although not all the information can be fully verified, the prestige of the authors and their continuous interactions with the community and the different institutions makes the TOP 500 the most trusted source of HPC information.

One can sign up in their webpage, download data and receive periodic information and newsletters.

What is an exa-scale system and how does the Frontier system look like?

An exa-scale supercomputer is a supercomputer that has the power to compute more than 1 exa-Flop operations per second, that is, more than 1018 floating point operations per second (EFLOPS or EFlop/s). An EFLOP is equivalent to 1,000,000 Tera-Flops (TFLOPS or TFlop/s, used in the rest of the text, being 1 TFLOP equivalent to 1012 floating point operations per second).

The numbers are astonishing.

Frontier is a system that contains 8.73 million Central Processing Units (CPU) cores and 8.14 million Graphics Processing Units (GPU) cores. Its peak performance is 1,685,651 TFLOPS, which means that it can reach 1,685,651,000,000,000,000 floating point operations per second, theoretically. One gets lost with such many zeros and these humongous magnitudes.

Compare this to the number 1 system in the first TOP500 list that appeared back in June 1993: it was a CM-5 supercomputer built by Thinking Machines Corporation with 1,024 cores and a measured performance (not peak) of 59.7 GFLOPS (59,700,000,000 floating point operations per second). This is 18.5M less performance – yes, eighteen million – than the measured performance of Frontier, which is 1,102,000 TFLOPS.

The estimated cost to build Frontier is $600M and the system consumes 21MWatts of power and it occupies a room of 680 square meters.

Aurora will probably be the 2nd exa-scale machine to be constructed and it is expected to be built towards the end of 2022 at the Argonne National Laboratory, Illinois, United States. It will use using Intel processors for CPUs and GPUs as well.

Stay tuned for the TOP500 list update in Q4 2022 and Q2 2023. It will be interesting to see the evolution of the exa-scale HPC race.

Current top 10 HPC systems and locations

The following table shows the top 10 HPC systems from the TOP500 list and it is a summary of the information available in the TOP500 website. Although these systems are more than just CPUs and GPUs, the table below focuses specially on these factors.

PositionNameLocationBuilt in or last update inAchieved performance Rmax (TFLOPS)Peak performance Rpeak (TFLOPS)ManufacturerMain CPU providerNumber of CPU coresMain GPU providerNumber of GPU coresOperating System
1FrontierUnited States20211,102,0001,685,651HPEAMD (x86 ISA)8,730,112AMD8,138,240HPE Cray OS (Linux)
2Supercomputer FugakuJapan2020442,010537,212FujitsuFujistsu (ARM ISA)7,630,848None-Red Hat Enterprise Linux
3LUMIFinland2022151,900214,352HPEAMD (x86 ISA)1,110,144AMD1,034,880HPE Cray OS (Linux)
4SummitUnited States2018148,600200,795IBMIBM (Power ISA)2,414,592NVIDIA2,211,840RHEL 7.4 (Linux)
5SierraUnited States201894,640125,712IBM / NVIDIA / MellanoxIBM (Power ISA)1,572,480NVIDIA1,382,400Red Hat Enterprise Linux
6Sunway TaihuLightChina201693,015125,436NRCPC(ShenWei ISA)10,649,600None-Sunway RaiseOS 2.0.5 (Linux)
7PerlmutterUnited States202170,87093,750HPEAMD (x86 ISA)761,856NVIDIA663,552HPE Cray OS (Linux)
8SeleneUnited States202063,46079,215NVIDIAAMD (x86 ISA)555,520NVIDIA483,840Ubuntu 20.04.1 LTS (Linux)
9Tianhe-2AChina201861,445100,679NUDTIntel (x86 ISA)4,981,760Matrix-20004,554,752Kylin Linux
10AdastraFrance202246,10061,608HPEAMD (x86 ISA)319,072AMD297,440HPE Cray OS (Linux)

Some interesting information and facts from this table.

Achieved performance and peak performance are based on the Rmax and Rpeak scores respectively. Rmax refers to the real measured performance of the system when running the LINPACK benchmark, which consists of a linear system of equations of order ‘n’ that are solved using LU decomposition with partial row pivoting. Peak performance, on the other hand, is the theoretical maximum performance and is it mainly computed by adding the individual peak performances of all individual computing nodes (CPUs, GPUs and other accelerators). Note that observed performance is always lower than peak performance.

There is variety in the Instruction Set Architecture (ISA), which is the repertoire and format of low-level instructions executed by the CPU: x86, Power, ARM… There is also variety in the GPU architecture (GPUs from NVIDIA, AMD and probably Intel once the Aurora system is in place) and manufacturers (HPE, Fujitsu, IBM, …). There is no variety with respect of the underlying operating system: all of them depend on one or another flavor of Linux.

Although most systems are built for public or academic research, there is a notable exception in the TOP 10. Selene, in position number 8, is build by NVIDIA and it is a system entirely owned by a private company. There are other exceptions from Microsoft, Samsung and Saudi Aramco, but these privately owned systems fall in positions between 10 and 20 in the list (one can download the raw list with all the parameters from the TOP500 website).

Another contrasting system is Sunway TaihuLight in position 6 from China. This is a system that uses its own ISA (ShenWei) and there are few internal details about it. The system does not use any other type of processing unit other than the CPUs (no GPUs or other accelerators) and the running frequency of its processors is 1.4 GHz (the rest of the systems run at frequencies between 2GHz and 3 GHz). China is building its own microarchitecture to remove any dependency on western countries and western technology whatsoever. Sunway TaihuLight was the response of the Chinese government to the prohibition by the US to upgrade the Tianhe-2A system (now in position number 9) using Intel’s technology back in 2015.

What is the position of Europe in the TOP500?

Downloading the TOP500 list in Excel format is very interesting to dig a bit more on the available data.

The following tables and figures show the distribution of HPC systems and capacity geographically (North America includes the United States and Canada) and divided in segments. The TOP500 list categorizes the systems in different segments: research, academia, government, industry, vendor and others. I have assumed that the first 3 (research, academia and government) are public initiatives and I have split the data into public and all segments.

AreaNumber of systems - publicTotal peak performance (TFLOPS) - publicPeak performance / system (TFLOPS) - publicNumber of systems - ALLTotal peak performance (TFLOPS) - ALLPeak performance / system (TFLOPS) - ALL
Europe85972,345.9711,439.361181,294,550.9410,970.77
North America812,785,615.9734,390.321423,230,788.2022,752.03
Asia631,228,454.6219,499.282282,251,574.029,875.32
Rest632,924.545,487.421271,528.505,960.71

Europe accumulates 36% of all the public HPC systems in the TOP500 list, but this percentage drops to 24% when all the segments are considered. On the other hand, it is interesting to see that Asia has more private HPC systems in the list (165, which is the result of 228 – 63) than public ones (63). From a country perspective, China owns 173 HPC systems from the TOP500 list, followed by the United States with 128.

Although Europe has a good presence in terms of number of HPC systems, we can see that their size is smaller than in other geographies by looking at their capacity (in this case, peak performance). As said Europe has 36% of all public HPC systems and 24% of all HPC systems, but they accumulate 19.4% and 19% of the worldwide peak performance. This can also be seen by the average peak performance per system. The peak performance of public and all HPC systems in Europe is 11,439 and 10,970 TFLOPS respectively. Note that there is not much difference between public and all systems in Europe, while there are bigger differences between these two groups in America (34,390 TFLOPS per public system on average vs. 22,752 TFLOPS for all segments) and Asia (19,499 vs. 9,875 TFLOPS).

Europe created the European High Performance Computing Joint Undertaking (EuroHPC JU), a pan-European effort to compete in the HPC race with the United States, Japan and China. Among other programs and activities, EuroHPC JU has built a roadmap to deploy new HPC systems. The following table summarizes them.

NameLocationStatusPeak performance (TFLOPS)Comments
LUMIFinlandOperating since 2022550,000Position 3 in TOP 500 June'22
VegaSloveniaOperating since 202110,050Position 131 (CPU partition) and 172 (GPU partition) in TOP 500 June'22
MeluXinaLuxembourgOperating since 202118,290Position 306 (CPU partition) and 48 (GPU partition) in TOP 500 June'22
DiscovererBulgariaOperating since 20215,940Position 113 in TOP 500 June'22
KarolinaCzech RepublicOperating since 202112,910Position 202 (CPU partition) and 79 (GPU partition) in TOP 500 June'22
LeonardoItalyOperational in Q3 2022 (expected)323,400
DeucalionPortugalUnder construction10,000
Mare Nostrum 5SpainUnder construction314,000It will substitute Mare Nostrum 4 and prior systems
TBDGermanyUnder definitionTBD
TBDGreeceUnder definitionTBD
TBDHungaryUnder definitionTBD
TBDIrelandUnder definitionTBD
TBDPolandUnder definitionTBD

Some of them are already available, while others are under construction or under definition. All of them are pre exa-scale systems since none has the capacity to execute 1 EFLOPS (1,000,000 TFLOPS) yet. It seems, though, that “Jupiter”, the system to be installed in Germany, will be Europe’s first exa-scale system.

The peak performance of each system differs between the one reported at the EuroHPC JU website and the one reported in the TOP500 list, although I would not bother much about it: the difference is probably due to the fact that the TOP500 uses the Rpeak metric, whereas it is not clear what the EuroHPC JU website is using.

The only caveat of reporting different metrics / data is that one cannot do an apples-to-apples comparison, so we do not know where these systems would stand in the TOP500 list if they were operational today. We only know the position of the systems that are available today, LUMI occupying the third place in the June 2022 list.

Mare Nostrum 5 in Barcelona is the system that deeper touches my heart. I worked many years at the facility that Intel Labs had near the Barcelona Supercomputing Center and I had and I still have many interactions with researchers in that organization.

Although there are not many details about the exact configuration of Mare Nostrum 5, it seems it will have several parts, each with a specific CPU / GPU configuration targeting specific application domains. Mare Nostrum will probably be a hybrid system including CPUs from Intel, AMD and NVIDIA (in this case, using the ARM instruction set architecture), and GPUs from NVIDIA, AMD and Intel. There were debates about adding some sort of “European” hardware, but the European Processor Initiative (EPI) based on RISC-V is not ready yet. The deployment of Mare Nostrum 5 has been delayed several times (it is common in all this sort of infrastructures) and it does not have, to the best of my knowledge, a public kick-off date.

Summary

The TOP500 HPC list is the best source to see the evolution of the HPC race and how each continent and country is positioned in this area. The United States is ahead in terms of accumulated HPC performance (2.7 times more than China, in 2nd position), the first exa-scale computer named Frontier and the second one in the horizon (expected in Q4 2022). In terms of number of HPC systems, China is ahead with 173 systems in the TOP500, followed by 128 from the United States.

Europe is trying to compete through the EuroHPC JU initiative, among other smaller efforts, and it expects to have its first exa-scale system set up in Germany (without any specific public due date yet). The average size of HPC systems in Europe is smaller than other geographies, which may be explained by a lesser unified market with more pressure from individual governments. The North American market (clearly dominated by the United States) and the Asian market (dominated by China and Japan) are more unified in this regard.

From a business perspective, it is a stimulating time since there is a fierce competition among different CPU vendors, GPU vendors and manufacturers. And this is just the tip of the iceberg, since an HPC system is not only compute power, but cooling methods and systems, memory, permanent storage, connectivity and networking, … And software! The best HPC system not equipped with a proper operating system, programming languages and compilers, libraries and tools is useless. But let’s leave the discussion here for today.

Let’s stay tuned with future TOP500 releases.

References

Contact

CONTACT INFORMATION

HEADQUARTERS

Torre R, 4a planta, Despatx A05, Parc Científic de Barcelona (PCB). C/ Baldiri Reixac 4-8 08028 Barcelona