Pynamic

Pynamic is a benchmark tool for measuring the performance of dynamic linking in Python-based, MPI-parallel applications. It generates a configurable number of Python C extension modules and shared libraries, then exercises dynamic loading and inter-module function calls in a parallel execution environment.

Pynamic was developed at Lawrence Livermore National Laboratory (LLNL) and is released under a BSD license (UCRL-CODE-228991).

---

Background

Large-scale HPC applications increasingly embed Python interpreters and depend on many dynamically linked shared objects. Pynamic provides a controlled, reproducible way to stress-test the dynamic linker and measure import times across a parameterized module graph. It is suitable for:

• Benchmarking dynamic library loading on parallel file systems
• Testing debugger and tool performance with Python C extensions
• Evaluating linker behavior under varying module counts and call depths

---

Requirements

Dependency	Minimum Version
CMake	3.10
Python	3.9
MPI	Any standard implementation (OpenMPI, MPICH, etc.)
C compiler	C99-compatible

Optional:

• Ninja (alternative CMake generator, often faster than Unix Makefiles)
• mpi4py (required at runtime for the Python driver)

---

Building

Pynamic uses a two-step process: first, Python source files are generated by config_pynamic.py; second, CMake compiles those sources into shared objects and an executable.

Step 1: Generate sources and build

python3 config_pynamic.py <num_modules> <avg_functions_per_module> [OPTIONS]

This script generates C source files in gen_src/, then invokes CMake to build all targets. Build artifacts are placed in build/.

Example:

python3 config_pynamic.py 9 4 -j 4

Generates 9 modules with an average of 4 functions each, building with 4 parallel jobs.

Step 2: Run the benchmark

mpirun -n <ranks> ./build/pynamic-mpi4py

---

Configuration Options

positional arguments:
  num_modules           Total number of shared object modules to generate
  avg_functions         Average number of functions per module

optional arguments:
  -b, --big-exe         Generate additional large executable variants
  -d DEPTH              Maximum call stack depth (default: 10)
  -e, --external        Enable cross-module function calls
  -i DIR                Add DIR to Python include search path
  -j N                  Build with N parallel jobs
  -n N                  Append N extra characters to all function names
  -p, --print           Insert print statements into generated functions
  -s SEED               Seed the random number generator for reproducibility
  -u [N] [AVG]          Generate N utility modules with AVG functions each
  -G GENERATOR          CMake generator (default: "Unix Makefiles")
  --with-cc CMD         Path to C compiler
  --with-python CMD     Path to Python interpreter
  -c OPTIONS            Pass additional options to CMake configure

---

Build Outputs

Artifact	Location	Description
libmodule*.so	build/	Python C extension modules
libpynamic.a	build/	Static archive of all modules
pynamic-mpi4py	build/	MPI executable with embedded Python
pynamic_driver_mpi4py.py	build/	Python driver script

---

Generated Source Structure

gen_src/
  pynamic.h                    # Module initializer declarations
  libmodule0.c ... libmoduleN.c  # Generated Python extension modules
  pynamic_driver_mpi4py.py     # Populated Python driver

Template files in templates/ are not modified; they are used by the generator to produce the final sources.

---

Running Tests

Unit tests for the generator script are located in tests/ and use pytest.

pytest tests/ -v

---

Project Structure

pynamic/
  config_pynamic.py            # Source generator and build driver
  CMakeLists.txt               # CMake build definition
  LICENSE.md                   # BSD license
  requirements.txt             # Python package requirements
  templates/
    libmodulebegin.c           # Entry-point breakpoint module template
    libmodulefinal.c           # Exit-point breakpoint module template
    mpi4py_main.c              # C main: MPI init and Python embedding
    pynamic_driver_mpi4py.py   # Python driver template
  tests/
    test_config.py             # Unit tests for config_pynamic.py
  gen_src/                     # Generated at runtime (not committed)
  build/                       # CMake build output (not committed)

---

Contributors

Original authors (LLNL): Gregory Lee, Dong Ahn, John Gyllenhaal, Bronis de Supinski

Samuel Herts (Oak Ridge National Laboratory) substantially updated the codebase to support modern Python 3 environments. Contributions include:

• Replaced the legacy autoconf/Makefile build system with a CMake-based build system
• Updated the Python C extension API usage and embedding code to be Python 3 compatible, including sys.argv initialization and module initialization conventions
• Refactored config_pynamic.py into a class-based design for clarity and maintainability
• Reorganized the project layout: separated template files into templates/, directed generated sources into gen_src/, and consolidated the CMake definition at the project root
• Added a pytest-based unit test suite for the source generator
• Added parallel build support and Ninja generator option
• Various bug fixes to includes, CMake target definitions, and generated driver file handling

---

License

Pynamic is released under a BSD 3-Clause license. Copyright (c) 2007, The Regents of the University of California. Produced at the Lawrence Livermore National Laboratory. UCRL-CODE-228991.

See LICENSE.md for the full text.

Authors: Gregory Lee, Dong Ahn, John Gyllenhaal, Bronis de Supinski Contact: Greg Lee (lee218@llnl.gov)

This work was produced at the University of California, Lawrence Livermore National Laboratory under Contract No. W-7405-ENG-48 with the U.S. Department of Energy.