Приволжский научно-образовательный центр суперкомпьютерных технологий
Research
  • Research overview

    • Numerical simulation of laser plasma on supercomputers

    Brief overview

    lazer

    Simulation of plasma dynamics and laser interaction with various targets is a high-demand area of computational physics. Among important applications are compact sources for hadron therapy for cancer treatment, bio-imaging, devices for research of intra-molecular and intra-atomic processes. Due to highly nonlinear effects and geometric complexity plasma simulation is often based on Particle-in-Cell (PIC) method. The distinguishing feature of this method is handling two distinct sets of data: ensemble of charged particles described by position, momentum, charge, and mass, and grid values of the electromagnetic field. Lack of straightforward ways of localizing memory access pattern causes challenges of efficient implementation for both traditional and heterogeneous cluster systems based on accelerators. Applications of the Particle-in-Cell method require supercomputers: some tasks require simulation of 1010 particles and 109 grid cells.

    Since 2010 our team develops electromagnetic PIC code PICADOR for heterogeneous cluster systems with CPUs, GPUs, and Intel Xeon Phi. The code is among the first implementations optimized for Xeon Phi.

    The project is developed in Lobachevsky State University of Nizhni Novgorod (UNN) in collaboration with researchers from the Institute of Applied Physics of the Russian Academy of Sciences (IAP RAS), Chalmers University of Technology, Gothenburg University and Umea University (Sweden).

    Simulations are currently performed on the following supercomputers: Lobachevsky (UNN), Lomonosov (MSU), MVS-10P (JSC RAS), Abisko (HPC2N, Sweden), Triolith (NSC, Sweden).

    About the project (the talk at the seminar of the Center for Supercomputing Technologies of ICMMG SB RAS).

    Team

    Project leaders

    • Arkady Gonoskov, PhD, assistant professor, Gothenburg University (Sweden)
    • Iosif Meyerov, PhD, vice-head of the Software department, Lobachevsky University (UNN, Russia)
    • Evgeny Efimenko, PhD, researcher, Institute of Applied Physics of RAS (IAP RAS, Russia)

    Code developers

    • Sergei Bastrakov, PhD, researcher, HZDR (Dresden, Germany) (lead developer 2010-2018)
    • Valentin Volokitin, PhD student, UNN (lead developer since 2019)
    • Igor Surmin, UNN
    • Elena Panova, student, UNN
    • Alexander Panov, student, UNN
    • Kirill Tarakanov, student, UNN
    • Yury Rodimkov, student, UNN
    • Anastasiia Arisova, student, UNN

    Collaborators, extension developers, users

    • Alexey Bashinov, researcher, IAP RAS
    • Artem Korzhimanov, PhD, researcher, IAP RAS
    • Alexander Muraviev, researcher, IAP RAS
    • Felix Mackenroth, PhD, researcher, Max Planck Institute for the Physics of Complex Systems (Germany)
    • Joel Magnusson, PhD student, Chalmers University of Technology (Sweden)
    • Erik Wallin, PhD student, Umea University (Sweden)
    • Tom Blackburn, PhD, researcher, Chalmers University of Technology (Sweden)

    Former members

    • Roman Donchenko, MS student, UNN
    • Alexander Malyshev, MS student, UNN
    • Michail Shyriaev, MS student, UNN
    • Michail Savichev, MS student, UNN
    • Michail Mlodik, student, UNN
    • Anatoly Rozanov, student, UNN
    • Anton Larin, student, UNN

    Features of PICADOR

    PICADOR is a tool for three-dimensional plasma simulation for traditional and heterogeneous supercomputers. It is based on the Particle-in-Cell method and has the following features:

    1. Widely used numerical schemes for EM-PIC: FDTD and NDF field solvers, spectral field solvers, Boris and Vay particle pushers, CIC and TSC particle form factors, Villasenor-Buneman and Esirkepov current deposition, moving frame.
    2. Module development kit for extensions. Main extensions:
    3. Capability to utilize CPUs (OpenMP, MPI), GPUs (CUDA), Intel Xeon Phi (OpenMP, MPI), including heterogeneous configurations (I.A. Surmin et al. CPC, 202, 2016).
    4. Dynamic load balancing on distributed (Surmin I. et al. LNCS, 9251, 2015) and shared (Meyerov I. et al. LNCS, 12043, 2019) memory, including optimization for QED simulations.
    5. A wide set of resampling techniques (Muraviev A. et al. CPC 262, 107826, 2021).
    6. Strong scaling efficiency:
      • Distributed memory: 92% on 256 CPUs, 90% on 64 Xeon Phi coprocessors;
      • Shared memory: 99% on 16 CPU cores, 78% on 60 Xeon Phi cores.
    7. Performance on a benchmark problem with CIC form factor, Villasenor-Buneman current deposition, double precision:
      • On Intel Xeon E5-2697 v3: 7.9 ns/particle update, 42.5 GFLOPS
      • On Intel Xeon Phi 7250 (KNL): 3.4 ns/particle update, 100 GFLOPS

    Hi-Chi project

    The project High-Intensity Collisions and Interactions (Hi-Chi) is an open-source collection of Python-controlled tools for performing simulations and data analysis in the research area of strong-field particle and plasma physics. The tools are being developed in C++ and provide high performance using either local or supercomputer resources. The project is intended to offer an environment for testing, benchmarking and aggregative use of individual components, ranging from basic routines to supercomputer codes.

    At the end of 2020, the oneAPI Center of Excellence was established at UNN under Intel support. One of the main direction of the Center is porting hi-Chi to the Data Parallel C ++ (DPC ++) programming language to ensure its operation on heterogeneous architectures.

    Since 2019, the group has been working on the use of machine learning methods and artificial neural networks for reconstructing experimental conditions (Gonoskov A. et al. Sci. rep., 9, 7043, 2019, Rodimkov Y. et al. Entropy, 23(1), 21, 2021).

    Ongoing research

    1. Performance optimization for CPU, GPU, Xeon Phi, and heterogeneous configurations.
    2. Dynamic factorization of particle ensemble for QED simulations.
    3. The use of artificial neural networks for the analysis of experimental data.
    4. Low-precision and mixed-precision computations in laser-plasma simulations.

    Grants and achievements

    1. Grant of the Russian Ministry of Science and Higher Education “Reliable and logically transparent artificial intelligence: technology, verification and application in socially significant and infectious diseases” (2020; one of the groups involved in research collaboration).
    2. Best talk award at the conference for young scientists (PCT-2019, E. Panova).
    3. Best paper award at Russian Supercomputing Days (2015).
    4. Grants of the Russian Foundation for Basic Research
      • 18-47-520001 (2018-2020);
      • 15-37-21015 (2015-2016);
      • 14-07-31211 (2014-2015).
    5. Software registration certificates
      • 2015611475 (2015);
      • 2013613052 (2013).
    6. Winner of all three stages of all-Russia contest “Efficient application of GPUs for large-scale problems” (MSU + T-Platforms, 2011).
    7. Grant of the Federal targeted program “Scientific and educational staff of innovative Russia” No. 02.740.11.0839, part of collective project (2010-2011).

    Press

    1. Intel Academic Program for oneAPI. Intel (2020). 
    2. Chirgwin, R. Hot iron: Knights Landing hits 100 gigaflops in plasma physics benchmark. The Register (2016)
    3. Barney L. Particle-in-cell Plasma Simulation Using Supercomputers Enhances Computational Physics. Scientific Computing (2016).

    Selected papers and talks

    Main descriptions of the code

    1. Surmin I.A., Bastrakov S.I., Efimenko E.S., Gonoskov A.A., Korzhimanov A.V., Meyerov I.B. Particle-in-Cell laser-plasma simulation on Xeon Phi coprocessors. Computer Physics Communications, 202, 204-210 (2016)
    2. Gonoskov A., Efimenko E., Ilderton A., Marklund M., Meyerov I., Muraviev A., Sergeev A., Surmin I., Wallin E. Extended particle-in-cell schemes for physics in ultrastrong laser fields: Review and developments. Physical review E, 92 (2015)
    3. Bastrakov S., Gonoskov A., Donchenko R., Efimenko E., Malyshev A., Meyerov I., Surmin I. Particle-in-Cell Plasma Simulation on Heterogeneous Cluster Systems. Journal of Computational Science. 3 (6), 474-479 (2012)

    Parallelism, optimization

    1. Volokitin, V., Bashinov, A., Efimenko, E., Gonoskov, A., Meyerov, I. (2021). High Performance Implementation of Boris Particle Pusher on DPC++. A First Look at oneAPI. arXiv:2104.04579.
    2. Muraviev, A., Bashinov, A., Efimenko, E., Volokitin, V., Meyerov, I., Gonoskov, A. (2021). Strategies for particle resampling in PIC simulations. Computer Physics Communications, 262, 107826.
    3. Volokitin, V., Bastrakov, S., Bashinov, A., Efimenko, E., Muraviev, A., Gonoskov, A., Meyerov, I. (2020). Optimized routines for event generators in QED-PIC codes. In Journal of Physics: Conference Series (Vol. 1640, No. 1, p. 012015). IOP Publishing.
    4. Panova, E.; Volokitin, V.; Efimenko, E.; Ferri, J.; Blackburn, T.; Marklund, M.; Muschet, A.; De Andres Gonzalez, A.; Fischer, P.; Veisz, L.; Meyerov, I.; Gonoskov, A. Optimized Computation of Tight Focusing of Short Pulses Using Mapping to Periodic Space. Appl. Sci. 2021, 11, 956.
    5. Meyerov, I., Panov, A., Bastrakov, S., Bashinov, A., Efimenko, E., Panova, E., Surmin, I., Volokitin, V., Gonoskov, A. (2019). Exploiting Parallelism on Shared Memory in the QED Particle-in-Cell Code PICADOR with Greedy Load Balancing. In International Conference on Parallel Processing and Applied Mathematics (pp. 335-347). Springer, Cham.
    6. Larin A. et al. (2018) Load Balancing for Particle-in-Cell Plasma Simulation on Multicore Systems. In: Wyrzykowski R., Dongarra J., Deelman E., Karczewski K. (eds) Parallel Processing and Applied Mathematics. PPAM 2017. Lecture Notes in Computer Science, vol 10777. Springer, Cham.
    7. Surmin I., Bastrakov S., Matveev Z., Efimenko E., Gonoskov A., Meyerov I. (2016) Co-design of a Particle-in-Cell Plasma Simulation Code for Intel Xeon Phi: A First Look at Knights Landing. In: Carretero J. et al. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2016. Lecture Notes in Computer Science, vol 10049. Springer, Cham.
    8. Bastrakov S., Meyerov I., Surmin I., Bashinov A., Efimenko E., Gonoskov A., Korzhimanov A., Larin A., Muraviev A., Rozanov A. Performance and Scalability Evaluation of Particle-in-Cell Code PICADOR on CPUs and Intel Xeon Phi Coprocessor. Poster at International Supercomputing Conference (2016)
    9. Meyerov I., Surmin I., Efimenko E., Gonoskov A., Malyshev A., Shiryaev M. Particle-in-Cell Plasma Simulation on CPUs, GPUs and Xeon Phi Coprocessors. Poster at International Supercomputing Conference (2014)
    10. Efimenko E., Bastrakov S., Gonoskov A., Meyerov I., Shiryaev M., Surmin I. Particle-in-Cell Plasma Simulation on The Intel Xeon Phi in PICADOR // Invited presentation at the International Conference on Numerical Simulation of Plasmas (23rd ICNSP, Beijing, 2013, September 14-16).

    Applications of PICADOR

    1. Efimenko, E. S., Bashinov, A. V., Gonoskov, A. A., Bastrakov, S. I., Muraviev, A. A., Meyerov, I. B., Kim, A.V., Sergeev, A. M. (2019). Laser-driven plasma pinching in e− e+ cascade. Physical Review E, 99(3), 031201.
    2. Efimenko E., Bashinov A., Bastrakov S., Gonoskov A., Muraviev A., Meyerov I., Kim A., Sergeev A. (2018). Extreme plasma states in laser-governed vacuum breakdown. Scientific reports, 8(1), 2329.
    3. Gonoskov A., Bashinov A., Bastrakov S., Efimenko E., Ilderton A., Kim A., Marklund M., Meyerov I., Muraviev A., Sergeev A. Ultrabright GeV Photon Source via Controlled Electromagnetic Cascades in Laser-Dipole Waves. Phys. Rev. X 7, 041003 (2017)
    4. Mackenroth F., Gonoskov A., Marklund M. Chirped-Standing-Wave Acceleration of Ions with Intense Lasers. Physical Review Letters. 117 (10), 104801 (2016)
    5. Mackenroth F., Gonoskov A., Marklund M. Theoretical benchmarking of laser-accelerated ion fluxes by 2D-PIC simulations (to appear).
    6. Muraviev A.A., Bastrakov S.I., Bashinov A.V., Gonoskov A.A., Efimenko E.S., Kim A.V., Meyerov I.B., Sergeev A.M. Generation of current sheets and giant quasistatic magnetic fields at the ionization of vacuum in extremely strong light fields. JETP Letters. 102 (3), 148-153 (2015)

    Machine learning techniques

    1. Rodimkov Y., Efimenko E., Volokitin V., Panova E., Polovinkin A., Meyerov I., Gonoskov A. ML-Based Analysis of Particle Distributions in High-Intensity Laser Experiments: Role of Binning Strategy. Entropy, 23(1), 21 (2021).
    2. Gonoskov A., Wallin E., Polovinkin A., Meyerov I. Employing machine learning for theory validation and identification of experimental conditions in laser-plasma physics. Scientific reports, 9(1), 1-15 (2019). 

    All publications

    2021

    1. Volokitin, V., Bashinov, A., Efimenko, E., Gonoskov, A., Meyerov, I. (2021). High Performance Implementation of Boris Particle Pusher on DPC++. A First Look at oneAPI. arXiv:2104.04579.
    2. Rodimkov Y., Efimenko E., Volokitin V., Panova E., Polovinkin A., Meyerov I., Gonoskov A. ML-Based Analysis of Particle Distributions in High-Intensity Laser Experiments: Role of Binning Strategy. Entropy, 23(1), 21 (2021).
    3. Muraviev, A., Bashinov, A., Efimenko, E., Volokitin, V., Meyerov, I., Gonoskov, A. (2021). Strategies for particle resampling in PIC simulations. Computer Physics Communications, 262, 107826.
    4. Panova, E.; Volokitin, V.; Efimenko, E.; Ferri, J.; Blackburn, T.; Marklund, M.; Muschet, A.; De Andres Gonzalez, A.; Fischer, P.; Veisz, L.; Meyerov, I.; Gonoskov, A. Optimized Computation of Tight Focusing of Short Pulses Using Mapping to Periodic Space. Appl. Sci. 2021, 11, 956.

    2020

    1. Volokitin, V., Bastrakov, S., Bashinov, A., Efimenko, E., Muraviev, A., Gonoskov, A., Meyerov, I. (2020). Optimized routines for event generators in QED-PIC codes. In Journal of Physics: Conference Series (Vol. 1640, No. 1, p. 012015). IOP Publishing.

    2019

    1. Meyerov, I., Panov, A., Bastrakov, S., Bashinov, A., Efimenko, E., Panova, E., Surmin, I., Volokitin, V., Gonoskov, A. (2019). Exploiting Parallelism on Shared Memory in the QED Particle-in-Cell Code PICADOR with Greedy Load Balancing. In International Conference on Parallel Processing and Applied Mathematics (pp. 335-347). Springer, Cham.
    2. Efimenko, E. S., Bashinov, A. V., Gonoskov, A. A., Bastrakov, S. I., Muraviev, A. A., Meyerov, I. B., Kim, A.V., Sergeev, A. M. (2019). Laser-driven plasma pinching in e− e+ cascade. Physical Review E, 99(3), 031201.

    2018

    1. Gonoskov A., Wallin E., Polovinkin A., Meyerov I. Employing machine learning for theory validation and identification of experimental conditions in laser-plasma physics. Scientific reports, 9(1), 1-15 (2019). 
    2. Efimenko E., Bashinov A., Bastrakov S., Gonoskov A., Muraviev A., Meyerov I., Kim A., Sergeev A. (2018). Extreme plasma states in laser-governed vacuum breakdown. Scientific reports, 8(1), 2329.
    3. Larin A. et al. (2018) Load Balancing for Particle-in-Cell Plasma Simulation on Multicore Systems. In: Wyrzykowski R., Dongarra J., Deelman E., Karczewski K. (eds) Parallel Processing and Applied Mathematics. PPAM 2017. Lecture Notes in Computer Science, vol 10777. Springer, Cham.

    2017

    1. Gonoskov A., Bashinov A., Bastrakov S., Efimenko E., Ilderton A., Kim A., Marklund M., Meyerov I., Muraviev A., Sergeev A. Ultrabright GeV Photon Source via Controlled Electromagnetic Cascades in Laser-Dipole Waves. Phys. Rev. X 7, 041003 (2017)
    2. Bastrakov S., Surmin I., Efimenko E., Gonoskov A., Meyerov I. Performance Aspects of Collocated and Staggered Grids for Particle-in-Cell Plasma Simulation. In: Malyshkin V. (eds) Parallel Computing Technologies. PaCT 2017. Lecture Notes in Computer Science, vol 10421. Springer, Cham (2017)

    2016

    1. Meyerov I., Bastrakov S., Surmin I., Bashinov A., Efimenko E., Korzhimanov A., Muraviev A., Gonoskov A. Hybrid CPU + Xeon Phi implementation of the Particle-in-Cell method for plasma simulation. Supercomputing frontiers and innovations, 3 (3), 5-10  (2016)
    2. Surmin I., Bastrakov S., Matveev Z., Efimenko E., Gonoskov A., Meyerov I. (2016) Co-design of a Particle-in-Cell Plasma Simulation Code for Intel Xeon Phi: A First Look at Knights Landing. In: Carretero J. et al. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2016. Lecture Notes in Computer Science, vol 10049. Springer, Cham.
    3. Surmin I.A., Bastrakov S.I., Efimenko E.S., Gonoskov A.A., Korzhimanov A.V., Meyerov I.B. Particle-in-Cell laser-plasma simulation on Xeon Phi coprocessors. Computer Physics Communications, 202, 204-210 (2016)

    2015

    1. Gonoskov A., Efimenko E., Ilderton A., Marklund M., Meyerov I., Muraviev A., Sergeev A., Surmin I., Wallin E. Extended particle-in-cell schemes for physics in ultrastrong laser fields: Review and developments. Physical review E, 92 (2015)
    2. Meyerov I.B., Bastrakov S.I., Surmin I.A., Gonoskov A.A., Efimenko E.S., Bashinov A.V., Korzhimanov A.V., Larin A.V., Muraviev A.A., Rozanov A.I., Savichev M.R. Three-dimensional particle-in-cell plasma simulation on Intel Xeon Phi: performance optimization and case study (in Russian). Numerical methods and programming, 16, 486-500 (2015)
    3. Muraviev A.A., Bastrakov S.I., Bashinov A.V., Gonoskov A.A., Efimenko E.S., Kim A.V., Meyerov I.B., Sergeev A.M. Generation of current sheets and giant quasistatic magnetic fields at the ionization of vacuum in extremely strong light fields. JETP Letters. 102 (3), 148-153 (2015)
    4. Surmin I., Bashinov A., Efimenko E, Gonoskov A, Meyerov I. Dynamic load balancing based on rectilinear partitioning in Particle-in-Cell plasma simulation. Lecture Notes in Computer Science, 9251, 107-119 (2015)

    2014

    1. Surmin I.A., Bastrakov S.I., Gonoskov A.A., Efimenko E.S., Meyerov I.B. Particle-in-Cell Plasma Simulation Using Intel Xeon Phi Coprocessors (in Russian). Numerical methods and programming, 15, 530-536 (2014)
    2. Meyerov I., Surmin I., Efimenko E., Gonoskov A., Malyshev A., Shiryaev M. Particle-in-Cell Plasma Simulation on CPUs, GPUs and Xeon Phi Coprocessors. Lecture Notes in Computer Science. 8488, 513-514 (2014)

    2013

    1. Meyerov I., Gergel V., Gonoskov A., Gorshkov A., Efimenko E., Ivanchenko M., Kirillin M., Malova A., Osipov G., Petrov V., Surmin I., Vildemanov A. High performance computing in biomedical applications. Procedia Computer Science. 18, 10-19 (2013)
    2. Bastrakov S.I., Meyerov I.B., Surmin I.A., Gonoskov A.A., Efimenko E.S., Malyshev A.S., Shiryaev M.A. Dynamic load balancing in the PICADOR plasma simulation code (in Russian). Numerical methods and programming, 14, 67-74 (2013)
    3. Bastrakov S., Efimenko E., Gonoskov A., Meyerov I., Surmin I. GPU-based Particle-in-Cell Plasma Simulation in PICADOR: Optimization Techniques. Proceedings of The 34th Progress In Electromagnetics Research Symposium (2013)

    2012

    1. Bastrakov S., Gonoskov A., Donchenko R., Efimenko E., Malyshev A., Meyerov I., Surmin I. Particle-in-Cell Plasma Simulation on Heterogeneous Cluster Systems. Journal of Computational Science. 3 (6), 474-479 (2012)