Numerical simulation of laser plasma on supercomputers

Brief overview

 

lazer

Simulation of plasma dynamics and laser interaction with various targets is a high-demand area of computational physics. Among important applications are compact sources for hadron therapy for cancer treatment, bio-imaging, devices for research of intra-molecular and intra-atomic processes. Due to highly nonlinear effects and geometric complexity plasma simulation is often based on Particle-in-Cell (PIC) method. The distinguishing feature of this method is handling two distinct sets of data: ensemble of charged particles described by position, momentum, charge, and mass, and grid values of the electromagnetic field. Lack of straightforward ways of localizing memory access pattern causes challenges of efficient implementation for both traditional and heterogeneous cluster systems based on accelerators. Applications of the Particle-in-Cell method require supercomputers: some tasks require simulation of 10particles and 10grid cells.

Since 2010 our team develops electromagnetic PIC code PICADOR for heterogeneous cluster systems with CPUs, GPUs and Intel Xeon Phi. The code is among the first implementations optimized for Xeon Phi.

The project is developed in Lobachevsky State University of Nizhni Novgorod (UNN) in collaboration with researchers from the Institute of Applied Physics of the Russian Academy of Sciences (IAP RAS), Chalmers University of Technology and Umea University (Sweden).

Simulations are currently performed on the following supercomputers: Lobachevsky (UNN), Lomonosov (MSU), MVS-100K & MVS-10P (JSC RAS), Abisko (HPC2N, Sweden), Triolith (NSC, Sweden).

 

Team

Project leaders

  • Arkady Gonoskov, PhD, assistant professor, Chalmers University of Technology (Sweden), IAP RAS, UNN
  • Iosif Meyerov, PhD, vice-head of the Software department, UNN
  • Evgeny Efimenko, researcher, IAP RAS

Code developers

  • Sergei Bastrakov, PhD, researcher, HZDR (Dresden, Germany), UNN
  • Valentin Volokitin, PhD student, UNN
  • Igor Surmin, UNN
  • Anton Larin, UNN
  • Elena Panova, student, UNN
  • Alexander Panov, student, UNN
  • Kirill Tarakanov, student, UNN
  • Yury Rodimkov, student, UNN

Collaborators, extension developers, users

  • Alexey Bashinov, researcher, IAP RAS
  • Artem Korzhimanov, PhD, researcher, IAP RAS
  • Alexander Muraviev, researcher, IAP RAS
  • Felix Mackenroth, PhD, researcher, Max Planck Institute for the Physics of Complex Systems (Germany)
  • Joel Magnusson, PhD student, Chalmers University of Technology (Sweden)
  • Erik Wallin, PhD student, Umea University (Sweden)
  • Tom Blackburn, PhD, researcher, Chalmers University of Technology (Sweden)

Former members

  • Roman Donchenko, MS student, UNN
  • Alexander Malyshev, MS student, UNN
  • Michail Shyriaev, MS student, UNN
  • Michail Savichev, MS student, UNN
  • Michail Mlodik, student, UNN
  • Anatoly Rozanov, student, UNN

Features of PICADOR

PICADOR is a tool for three-dimensional plasma simulation for traditional and heterogeneous supercomputers. It is based on the Particle-in-Cell method and has the following features:

  1. Widely used numerical schemes for EM-PIC: FDTD and NDF field solvers, Boris particle pusher, CIC and TSC particle form factors, Villasenor-Buneman and Esirkepov current deposition, moving frame.
  2. Module development kit for extensions. Main extensions:
  3. Capability to utilize CPUs (OpenMP, MPI), GPUs (CUDA), Intel Xeon Phi (OpenMP, MPI), including heterogeneous configurations.
  4. Dynamic domain decomposition with load balancing.
  5. Strong scaling efficiency:
    • Distributed memory: 92% on 256 CPUs, 90% on 64 Xeon Phi coprocessors;
    • Shared memory: 99% on 16 CPU cores, 78% on 60 Xeon Phi cores.
  6. Performance on a benchmark problem with CIC form factor, Villasenor-Buneman current deposition, double precision:
    • On Intel Xeon E5-2697 v3: 7.9 ns/particle update, 42.5 GFLOPS
    • On Intel Xeon Phi 7250 (KNL): 3.4 ns/particle update, 100 GFLOPS

Current work

  1. Performance optimization for CPU, GPU, Xeon Phi, and heterogeneous configurations.
  2. Dynamic factorization of particle ensemble for simulations with QED.
  3. The use of artificial neural networks for the analysis of experimental data.

Grants and achievements

  1. Best paper award at Russian Supercomputing Days (2015).
  2. Grants of the Russian Foundation for Basic Research
    • 18-47-520001 (2018-2020);
    • 15-37-21015 (2015-2016);
    • 14-07-31211 (2014-2015).
  3. Software registration certificates
    • 2015611475 (2015);
    • 2013613052 (2013).
  4. Winner of all three stages of all-Russia contest “Efficient application of GPUs for large-scale problems” (MSU + T-Platforms, 2011).
  5. Grant of the Federal targeted program “Scientific and educational staff of innovative Russia” No. 02.740.11.0839, part of collective project (2010-2011).

Press

  1. Chirgwin, R. Hot iron: Knights Landing hits 100 gigaflops in plasma physics benchmark. The Register (2016)
  2. Barney L. Particle-in-cell Plasma Simulation Using Supercomputers Enhances Computational Physics. Scientific Computing (2016).

Selected papers and talks

Main descriptions of the code

  1. Surmin I.A., Bastrakov S.I., Efimenko E.S., Gonoskov A.A., Korzhimanov A.V., Meyerov I.B. Particle-in-Cell laser-plasma simulation on Xeon Phi coprocessors. Computer Physics Communications, 202, 204-210 (2016)
  2. Gonoskov A., Efimenko E., Ilderton A., Marklund M., Meyerov I., Muraviev A., Sergeev A., Surmin I., Wallin E. Extended particle-in-cell schemes for physics in ultrastrong laser fields: Review and developments. Physical review E, 92 (2015)
  3. Bastrakov S., Gonoskov A., Donchenko R., Efimenko E., Malyshev A., Meyerov I., Surmin I. Particle-in-Cell Plasma Simulation on Heterogeneous Cluster Systems. Journal of Computational Science. 3 (6), 474-479 (2012)

Parallelism, optimization

  1. Larin A. et al. (2018) Load Balancing for Particle-in-Cell Plasma Simulation on Multicore Systems. In: Wyrzykowski R., Dongarra J., Deelman E., Karczewski K. (eds) Parallel Processing and Applied Mathematics. PPAM 2017. Lecture Notes in Computer Science, vol 10777. Springer, Cham.
  2. Surmin I., Bastrakov S., Matveev Z., Efimenko E., Gonoskov A., Meyerov I. (2016) Co-design of a Particle-in-Cell Plasma Simulation Code for Intel Xeon Phi: A First Look at Knights Landing. In: Carretero J. et al. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2016. Lecture Notes in Computer Science, vol 10049. Springer, Cham.
  3. Bastrakov S., Meyerov I., Surmin I., Bashinov A., Efimenko E., Gonoskov A., Korzhimanov A., Larin A., Muraviev A., Rozanov A. Performance and Scalability Evaluation of Particle-in-Cell Code PICADOR on CPUs and Intel Xeon Phi Coprocessor. Poster at International Supercomputing Conference (2016)
  4. Meyerov I.B., Bastrakov S.I., Surmin I.A., Gonoskov A.A., Efimenko E.S., Bashinov A.V., Korzhimanov A.V., Larin A.V., Muraviev A.A., Rozanov A.I., Savichev M.R. Three-dimensional particle-in-cell plasma simulation on Intel Xeon Phi: performance optimization and case study (in Russian). Numerical methods and programming, 16, 486-500 (2015)
  5. Meyerov I., Surmin I., Efimenko E., Gonoskov A., Malyshev A., Shiryaev M. Particle-in-Cell Plasma Simulation on CPUs, GPUs and Xeon Phi Coprocessors. Poster at International Supercomputing Conference (2014)
  6. Efimenko E., Bastrakov S., Gonoskov A., Meyerov I., Shiryaev M., Surmin I. Particle-in-Cell Plasma Simulation on The Intel Xeon Phi in PICADOR // Invited presentation at the International Conference on Numerical Simulation of Plasmas (23rd ICNSP, Beijing, 2013, September 14-16). 

Applications of PICADOR

  1. Efimenko, E. S., Bashinov, A. V., Gonoskov, A. A., Bastrakov, S. I., Muraviev, A. A., Meyerov, I. B., Kim, A.V., Sergeev, A. M. (2019). Laser-driven plasma pinching in e-e+ cascade (submitted).
  2. Efimenko E., Bashinov A., Bastrakov S., Gonoskov A., Muraviev A., Meyerov I., Kim A., Sergeev A. (2018). Extreme plasma states in laser-governed vacuum breakdownScientific reports8(1), 2329.
  3. Gonoskov A., Bashinov A., Bastrakov S., Efimenko E., Ilderton A., Kim A., Marklund M., Meyerov I., Muraviev A., Sergeev A. Ultrabright GeV Photon Source via Controlled Electromagnetic Cascades in Laser-Dipole Waves. Phys. Rev. X 7, 041003 (2017)
  4. Mackenroth F., Gonoskov A., Marklund M. Chirped-Standing-Wave Acceleration of Ions with Intense Lasers. Physical Review Letters. 117 (10), 104801 (2016)
  5. Mackenroth F., Gonoskov A., Marklund M. Theoretical benchmarking of laser-accelerated ion fluxes by 2D-PIC simulations (to appear).
  6. Muraviev A.A., Bastrakov S.I., Bashinov A.V., Gonoskov A.A., Efimenko E.S., Kim A.V., Meyerov I.B., Sergeev A.M. Generation of current sheets and giant quasistatic magnetic fields at the ionization of vacuum in extremely strong light fields. JETP Letters. 102 (3), 148-153 (2015)

All publications

2019

  1. Efimenko, E. S., Bashinov, A. V., Gonoskov, A. A., Bastrakov, S. I., Muraviev, A. A., Meyerov, I. B., Kim, A.V., Sergeev, A. M. (2019). Laser-driven plasma pinching in e-e+ cascade (submitted).
  2. Panova E., Efimenko E., Bastrakov S., Volokitin V., Gonoskov A., Meyerov I. Parallel implementation of the pseudo-spectral Maxwell equations solver (In Russian, accepted).

2018

  1. Gonoskov, A., Wallin, E., Polovinkin, A., & Meyerov, I. (2018). Employing machine learning for theory validation and identification of experimental conditions in laser-plasma physicsarXiv preprint arXiv:1812.06304.
  2. Efimenko E., Bashinov A., Bastrakov S., Gonoskov A., Muraviev A., Meyerov I., Kim A., Sergeev A. (2018). Extreme plasma states in laser-governed vacuum breakdownScientific reports8(1), 2329.
  3. Larin A. et al. (2018) Load Balancing for Particle-in-Cell Plasma Simulation on Multicore Systems. In: Wyrzykowski R., Dongarra J., Deelman E., Karczewski K. (eds) Parallel Processing and Applied Mathematics. PPAM 2017. Lecture Notes in Computer Science, vol 10777. Springer, Cham.
  4. Volokitin V.D., Bastrakov S.I., Bashinov A.V., Gonoskov A.A., Efimenko E.S., Surmin I.A., Meyerov I.B. (2018) Approximation of synchrotron functions for speeding up numerical simulation of plasma with account for effects of quantum electrodynamics.  Russian Supercomputing Days, Moscow, 24-25 September 2018. P. 989-990.

2017

  1. Gonoskov A., Bashinov A., Bastrakov S., Efimenko E., Ilderton A., Kim A., Marklund M., Meyerov I., Muraviev A., Sergeev A. Ultrabright GeV Photon Source via Controlled Electromagnetic Cascades in Laser-Dipole Waves. Phys. Rev. X 7, 041003 (2017)
  2. Bastrakov S., Surmin I., Efimenko E., Gonoskov A., Meyerov I. Performance Aspects of Collocated and Staggered Grids for Particle-in-Cell Plasma Simulation. In: Malyshkin V. (eds) Parallel Computing Technologies. PaCT 2017. Lecture Notes in Computer Science, vol 10421. Springer, Cham (2017)

2016

  1. Meyerov I., Bastrakov S., Surmin I., Bashinov A., Efimenko E., Korzhimanov A., Muraviev A., Gonoskov A. Hybrid CPU + Xeon Phi implementation of the Particle-in-Cell method for plasma simulation. Supercomputing frontiers and innovations, 3 (3), 5-10  (2016)
  2. Surmin I., Bastrakov S., Matveev Z., Efimenko E., Gonoskov A., Meyerov I. (2016) Co-design of a Particle-in-Cell Plasma Simulation Code for Intel Xeon Phi: A First Look at Knights Landing. In: Carretero J. et al. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2016. Lecture Notes in Computer Science, vol 10049. Springer, Cham.
  3. Surmin I.A., Bastrakov S.I., Efimenko E.S., Gonoskov A.A., Korzhimanov A.V., Meyerov I.B. Particle-in-Cell laser-plasma simulation on Xeon Phi coprocessors. Computer Physics Communications, 202, 204-210 (2016)

2015

  1. Gonoskov A., Efimenko E., Ilderton A., Marklund M., Meyerov I., Muraviev A., Sergeev A., Surmin I., Wallin E. Extended particle-in-cell schemes for physics in ultrastrong laser fields: Review and developments. Physical review E, 92 (2015)
  2. Meyerov I.B., Bastrakov S.I., Surmin I.A., Gonoskov A.A., Efimenko E.S., Bashinov A.V., Korzhimanov A.V., Larin A.V., Muraviev A.A., Rozanov A.I., Savichev M.R. Three-dimensional particle-in-cell plasma simulation on Intel Xeon Phi: performance optimization and case study (in Russian). Numerical methods and programming, 16, 486-500 (2015)
  3. Muraviev A.A., Bastrakov S.I., Bashinov A.V., Gonoskov A.A., Efimenko E.S., Kim A.V., Meyerov I.B., Sergeev A.M. Generation of current sheets and giant quasistatic magnetic fields at the ionization of vacuum in extremely strong light fields. JETP Letters. 102 (3), 148-153 (2015)
  4. Surmin I., Bashinov A., Efimenko E, Gonoskov A, Meyerov I. Dynamic load balancing based on rectilinear partitioning in Particle-in-Cell plasma simulation. Lecture Notes in Computer Science, 9251, 107-119 (2015)

2014

  1. Surmin I.A., Bastrakov S.I., Gonoskov A.A., Efimenko E.S., Meyerov I.B. Particle-in-Cell Plasma Simulation Using Intel Xeon Phi Coprocessors (in Russian). Numerical methods and programming, 15, 530-536 (2014)
  2. Meyerov I., Surmin I., Efimenko E., Gonoskov A., Malyshev A., Shiryaev M. Particle-in-Cell Plasma Simulation on CPUs, GPUs and Xeon Phi Coprocessors. Lecture Notes in Computer Science. 8488, 513-514 (2014)

2013

  1. Meyerov I., Gergel V., Gonoskov A., Gorshkov A., Efimenko E., Ivanchenko M., Kirillin M., Malova A., Osipov G., Petrov V., Surmin I., Vildemanov A. High performance computing in biomedical applications. Procedia Computer Science. 18, 10-19 (2013)
  2. Bastrakov S.I., Meyerov I.B., Surmin I.A., Gonoskov A.A., Efimenko E.S., Malyshev A.S., Shiryaev M.A. Dynamic load balancing in the PICADOR plasma simulation code (in Russian). Numerical methods and programming, 14, 67-74 (2013)
  3. Bastrakov S., Efimenko E., Gonoskov A., Meyerov I., Surmin I. GPU-based Particle-in-Cell Plasma Simulation in PICADOR: Optimization Techniques. Proceedings of The 34th Progress In Electromagnetics Research Symposium (2013)

2012

  1. Bastrakov S., Gonoskov A., Donchenko R., Efimenko E., Malyshev A., Meyerov I., Surmin I. Particle-in-Cell Plasma Simulation on Heterogeneous Cluster Systems. Journal of Computational Science. 3 (6), 474-479 (2012)