ADVANCED HIGH-PERFORMANCE RECONFIGURABLE COMPUTERS WITH IMMERSION COOLING
Abstract
The paper deals with prospects for the design of high-performance reconfigurable computing devices
based on modern Xilinx UltraScale+ FPGAs. The aim of the research is the computational density
up to 128 LSI FPGAs in one 3U 19’ product. Besides, it is necessary to provide the required powersupply and cooling of the system’s computing elements during execution of computationally expensive
tasks. To provide the product’s required parameters in the given design package, we made the topology
of our printed circuit boards and the design technology of their parts more complex. To cool the components
of our computer system, we use the immersion technology. The distinction of the designed computer
systems is wide capabilities of data exchange inside the block and among the blocks. It is crucial for
tightly-coupled tasks, where the number of data transfers among functional devices is much higher than
the number of such devices. As the main links between FPGAs, we use differential lines with multigigabit
transceivers (MGT). The optical channels based system of data exchange among the blocks is
provides the throughput over 2 Tbit/sec. We designed and produced a prototype of a computational
module based on UltraScale+ FPGAs, and a prototype of a reconfigurable computational block. The
computational block contains a general-purpose processor and all needed input-output interfaces. It is a
stand-alone device. We used the new-generation computational module for implementation of several
algorithms of various scientific and technical problems, and proved its wide applicability. We designed
a modified immersion cooling subsystem, which dissipates total heat up to 20 kW. To achieve such level
of heat dissipation, we implemented technical solutions concerning all components of the cooling system:
the cooling agent, heat sinks, the pump, the heat-exchange unit. It is possible to unite several blocks
into computer complexes with the performance up to tens of petaflops. Such complexes require a suitable
engineering infrastructure.
References
[Reconfigurable computer systems: textbook], study guide end. by I.A. Kalyaeva. 2016. Rostov-
on-Don: Izd-vo YuFU, 2016, 472 p. ISBN 978-5-9275-1980-7.
2. Abramov Sergey, Amel'kin Sergey, Klyuev Leonid, CHichkovskiy Aleksandr. Zhidkostnoe
okhlazhdenie vychislitel'nykh kompleksov [Liquid cooling of computer complexes],
Radioelektronnye tekhnologii [Radio-electronic technologies], 2017, No. 5, pp. 79-82.
3. Klyuev Leonid, Khrebtovskiy Ivan. Opyt postroeniya HPC klasterov IMMERS s ispol'zovaniem
pogruzhnogo zhidkostnogo okhlazhdeniya [Design experience of IMMERS HPC clusters with
immerse liquid cooling]. Available at: http://www2.sscc.ru/Seminars/paper/2015/2015-05-
14_IMMERS.pdf (accessed 10 October 2019).
4. Levin I.I., Dordopulo A.I., Doronchenko Y.I., Raskladkin M.K., Fedorov A.M., Kalyaev Z.V.
Immersion liquid cooling FPGA-based reconfigurable computer system, 14th IFAC International
Conference оn Programmable Devices аnd Embedded Systems (PDES 2016),
Brno/Lednice, Czech Republic, 2016, Vol. No. 3, Part No. 1, pp. 175-180. Available at:
https://www.sciencedirect.com/science/article/pii/S2405896316327082.
5. Levin I.I., Doronchenko Y.I., Kovalenko A.G., Dordopulo А.I. Realization of Problem of Liquid
Filtering in Porous Medium on Reconfigurable Computer System, 5th International Conference on
Informatics, Electronics & Vision (ICIEV), 13~14 May, 2016, Dhaka, Bangladesh, pp. 1117-1121.
Doi: 20.1109/ICIEV/2016.7760173. Available at: https://ieeexplore.ieee.org/document/7760173.
6. Levin I.I., Dordopulo A.I., Fedorov A.M., Doronchenko Yu.I. Razvitie tekhnologii postroeniya
rekonfiguriruemykh vychislitel'nykh sistem s zhidkostnym okhlazhdeniem [Development of
the design technology of reconfigurable computer systems with liquid cooling],
Superkomp'yuternye tekhnologii (SKT-2018): Mater. 5-y Vserossiyskoy nauchnotekhnicheskoy
konferentsii [Supercomputer technologies (SCT-2018): Proceedings of the 5th
All-Russian scientific and technical conference]: in 2 vol. Vol. 1. Rostov-on-Don; Taganrog:
Izd-vo YuFU, 2018, pp. 184-187. ISBN 978-5-9275-2834-9.
7. Levin Ilya, Dordopulo Alexey, Fedorov Alexander, Doronchenko Yuriy. High-Performance
Reconfigurable Computer Systems with Immersion Cooling, Parallel Computational Technologies
(PCT 2018). 12th International Conference (Rostov-on-Don, Russia, April 2–6, 2018),
pp. 62-76. Available at: https://link.springer.com/content/pdf/10.1007%2F978-3-319-99673-
8_5.pdf (accessed 10 October 2019).
8. Levin I.I., Fedorov A.M., Udovenko D.L. Patent No. 2643173 for an invention
No. 2016152777. Russian Federation. Immersionnaya sistema okhlazhdeniya dlya elektronnykh
ustroystv [Immerse cooling system for electronic devices]. Applicant of invention and patent holder
is OOO “SRC of supercomputers and neurocomputers”; registered on 31.01.2018.
9. Levin I.I., Fedorov A.M., Udovenko D.L. Patent No. 2683425 for an invention No.
2018104165. Russian Federation. Sistema teploobmena dlya zhidkostnogo okhlazhdeniya
elektronnykh ustroystv (varianty) [Heat-transfer system for liquid cooling of electronic devices
(variants)]. Applicant of invention and patent holder is OOO “SRC of supercomputers and
neurocomputers”; registered on 28.03.2019.
10. Levin Ilya, Dordopulo Alexey, Fedorov Alexander, Doronchenko Yuriy. Design Technology
for Reconfigurable Computer Systems with Immersion Cooling, In: Voevodin V., Sobolev S.
(eds), Supercomputing. RuSCDays 2018. Communications in Computer and Information Science,
Vol. 965. Springer, Cham, 2018, pp. 554-564. Available at: https://link. springer.
com/chapter/10.1007/978-3-030-05807-4_47.
11. Levin I.I., Dordopulo A.I., Doronchenko Yu.I., Fedorov A.M. Postroenie perspektivnykh
rekonfiguriruemykh vychislitel'nykh sistem s zhidkostnym okhlazhdeniem [Design of advanced
reconfigurable computer systems with liquid cooling], Shestoy Natsional'nyy
Superkomp'yuternyy Forum, (NSKF-2017), IPS im. A.K. Aylamazyana RAN, g. Pereslavl'-
Zalesskiy, Rossiya, 28 noyabrya–01 dekabrya 2017 g. [6th National Supercomputer Forum,
(NSCF-2017), Program Systems Institute of the RAS, Pereslavl-Zalesskiy, Russia, November
28 – December 01, 2017]. Available at: http://2017.nscf.ru/in-fo/Schedule.pdf.
12. Levin I.I., Dordopulo A.I., Fedorov A.M. Rekonfiguriruemye vychislitel'nye sistemy
sverkhvysokoy proizvoditel'nosti s zhidkostnoy sistemoy okhlazhdeniya [Super-high performance
reconfigurable computer systems with liquid cooling], Mater. Vserossiyskoy nauchnoprakticheskoy
konferentsii s mezhdunarodnym uchastiem. Tekhnicheskie nauki «Aspekty
razvitiya nauki, obrazovaniya i modernizatsii promyshlennosti». Taganrog, 20–21 aprelya
2017 g., DGTU [Proceedings of All-Russian scientific and practical conference with international
participance. Technical sciences “Development aspects of science, education and modernization
of industry”. Taganrog, April 20–21, 2017., DSTU]. Rostov-on-Don: DGTU, 2017,
pp. 67-73. ISBN 978-5-7890-1329-8.
13. Levin I.I., Dordopulo A.I., Doronchenko Yu.I., Fedorov A.M. Perspektivnye rekonfiguriruemye
vychislitel'nye sistemy s zhidkostnym okhlazhdeniem [Advanced reconfigurable computer systems
with liquid cooling], Mater. 10-y Vse-rossiyskoy mul'tikonferentsii po problemam
upravleniya (MKPU-2017), 11-16 sentyabrya 2017 g. [Proceedings of the 10th All-Russian
multi-conference on problems of control (MCPC-2017), September 11-16, 201]. Rostov-on-
Don: Izd-vo YuFU, 2017, Vol. 3, pp. 148-150. ISBN 978-5-9275-2463-1.
14. Levin I., Dordopulo A., Fedorov A., Doronchenko Y. Design of advanced reconfigurable computer
systems with liquid cooling, In: Voevodin V., Sobolev S. (eds) Supercomputing. RuSCDays
2017. Communications in Computer and Information Science, Vol. 793. Springer, Cham, 2017,
pp. 442-455. Available at: https://link.springer. com/chapter/10.1007/978-3-319-71255-0_36.
15. Levin I.I., Dordopulo A.I., Fedorov A.M., Gulenok A.A. Reconfigurable computer based on
Virtex UltraScale+ FPGAs with immersion cooling system, XI mezhdunarodnaya nauchnaya
konferentsiya "Parallel'nye vychislitel'nye tekhnologii (PaVT) 2017 (Parallel computational
technologies (PCT’2017) agora.guru.ru/pavt)", Kazan', Respublika Tatarstan, Rossiya, 3-7
aprelya 2017 g. [XI international scientific conference “Parallel computational technologies
(PCT’2017) agora.guru.ru/pavt)”, Kazan, Republic of Tatarstan, Russia, April 3-7, 2017].
Chelyabinsk: Izd. tsentr YuUrGU, 2017, pp. 27-41. ISBN 978-5-696-04880-2. Available at:
https://link.springer.com/chapter/10.1007/978-3-319-67035-5_3 (accessed 10 October 2019).
16. Levin I.I., Fedorov A.M., Doronchenko Yu.I., Raskladkin M.K. Perspektivnye vysokoproizvoditel'nye
rekonfiguriruemye vychisliteli s immersionnym okhlazhdeniem [Advanced
high-performance reconfigurable computers with immersion cooling], Vserossiyskaya
nauchnaya konferentsiya "Superkomp'yuternye tekhnologii (SKT) 2020": Tr. molodykh
uchenykh. 05-10 oktyabrya 2020 g.: Krym, Alushta, Rossiya [All-Russian scientific conference
“Supercomputer technologies (SCT) 2020”. Proceedings of young scientists. October 05-10,
2020: Crimea, Alushta, Russia]. Rostov-on-Don; Taganrog: Izd-vo YuFU, 2020, pp. 29-34.
17. Levin Il'ya, Fedorov Aleksandr, Doronchenko Yuriy, Dordopulo Aleksey, Raskladkin Maksim.
Vysokoproizvoditel'nyy rekonfiguriruemyy vychislitel'nyy blok na osnove PLIS XILINX
ULTRASCALE+ [High-performance reconfigurable computational block based on XILINX
ULTRASCALE+ FPGAs], 14th International Conference «Parallel Computational Technologies
», PCT 2020, 31 marta-2 aprelya 2020 g., Perm' [14th International Conference “Parallel
Computational Technologies”, PCT 2020, March 31 – April 2, 2020, Perm], pp. 174-184.
Available at: http://omega.sp.susu.ru/pavt2020/short/079.pdf.
18. Kalyaev I.A., Dordopulo A.I., Levin I.I., Fedorov A.M. Razvitie otechestvennykh
mnogokristal'nykh rekonfiguriruemykh vychislitel'nykh sistem: ot vozdushnogo k
zhidkostnomu okhlazhdeniyu [Development of domestic multichip reconfigurable computer
systems: from air to liquid cooling], Tr. SPIIRAN [Proceedings of SPIIRAS]. Saint Petersburg:
Izd-vo SPIIRAN FGBUN Sankt-Peterburgskiy institut informatiki i avtomatizatsii RAN, 2017,
No. 1 (50), pp. 5-31. Doi: 10.15622/sp.50.1. Available at: http://dx.doi.org/10.15622/sp.50
(accessed 10 October 2019).
19. Levin I.I., Fedorov A.M., Doronchenko Yu.I., Raskladkin M.K. Vysokoproizvoditel'nyy
rekonfiguriruemyy vychislitel' na osnove PLIS XILINX ULTRASCALE+ dlya resheniya
sil'nosvyazannykh zadach [High-performance reconfigurable computer based on XILINX
ULTRASCALE+ FPGAs for tightly-coupled tasks], 12-ya Mul'tikonferentsiya po problemam
upravleniya (MKPU-2019), 23-28 sentyabrya 2019 g., s. Divnomorskoe, Rossiya [12th
Multiconference on problems of control (MCPC-2019), September 23-28, 2019,
Divnomoskoye, Russia]. Rostov-on-Don; Taganrog: Izd-vo YuFU, 2019, Vol. 3, pp. 113-117.
ISBN 978-5-9275-3188-2.
20. Levin I.I., Dordopulo A.I., Fedorov A.M., Gulenok A.A. Reconfigurable computer based on
Virtex UltraScale+FPGAs with immersion cooling system, In: Sokolinsky L., Zymbler M.
(eds), Parallel Computational Technologies. PCT 2017. Communications in Computer and Information
Science, Vol 753, pp. 27-41. Springer, Cham, 2017. Available
at:https://link.springer.com/chapter/10.1007/978-3-319-67035-5_3.