Skip to content

Conversation

@mmason-nvidia
Copy link
Contributor

The CUDA Toolkit 13.1 release provides beta support for debugging Numba CUDA programs using CUDA GDB and VSCode.

This PR includes the documentation for setup and debugging, and an example VSCode workspace that can be used as both a tutorial and as a starting point for other development projects.

Closes #664

…DB and VSCode

The CUDA Toolkit 13.1 release provides beta support for debugging Numba CUDA programs using CUDA GDB and VSCode.

This PR includes the documentation for setup and debugging, and an example VSCode workspace that can be used as both a tutorial and as a starting point for other development projects.
@copy-pr-bot
Copy link

copy-pr-bot bot commented Dec 12, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Dec 12, 2025

Greptile Summary

This PR adds comprehensive debugging documentation for Numba CUDA programs using CUDA GDB and Visual Studio Code, supporting the beta debugging features released in CUDA Toolkit 13.1.

Key additions:

  • Complete debugging guide (docs/source/user/debugging.rst) covering VSCode setup, configuration, command-line debugging, and known limitations
  • Example VSCode workspace (examples/debugging/) with properly configured launch.json and settings.json files
  • Educational example (hello.py) demonstrating breakpoints, variable inspection, shared memory, and polymorphic variables
  • Four documentation screenshots showing the debugging workflow
  • Appropriate pre-commit hook exclusions for debugging example files

Minor issues found:

  • Two identical grammar typos: "The can also be set" should be "They can also be set" in both the documentation and example code

The documentation is well-structured, thorough, and provides clear instructions for both GUI and command-line debugging workflows. The example workspace serves as an excellent starting point for users.

Confidence Score: 4/5

  • This PR is safe to merge with minimal risk after fixing minor typos
  • The PR adds comprehensive documentation and example files without modifying any runtime code. Two minor grammar typos were found that should be corrected. All configuration changes are appropriate, and the example code demonstrates debugging features correctly.
  • Files docs/source/user/debugging.rst and examples/debugging/hello.py need minor typo corrections

Important Files Changed

Filename Overview
.pre-commit-config.yaml Appropriately excludes debugging example files from pre-commit hooks
docs/source/user/debugging.rst Comprehensive debugging documentation with minor typos found
examples/debugging/.vscode/launch.json Well-documented VSCode launch configuration for CUDA debugging
examples/debugging/hello.py Educational example demonstrating debugging features with minor typo

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Comments (2)

  1. docs/source/user/debugging.rst, line 135 (link)

    style: The PYTHONPATH in this documentation snippet (${workspaceFolder}/../../misc) differs from the actual launch.json file which uses ${workspaceFolder}. Consider aligning these for consistency.

  2. docs/source/user/debugging.rst, line 127 (link)

    logic: The referenced file numba-cuda/misc/gdb_print_extension.py does not exist in this repository. The misc/ directory at the repo root is missing. Is this file intended to be added in a separate PR, or should the documentation be updated to reflect its actual location?

4 files reviewed, 2 comments

Edit Code Review Agent Settings | Greptile

@rparolin rparolin assigned gmarkall and rparolin and unassigned gmarkall Dec 12, 2025
@rparolin rparolin added this to the next milestone Dec 12, 2025
]
}

Debugging Host Python Code
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we call out that we can debug via the CLI, a howto that walks the user through how to debug hello.py via cuda-gdb could be a potentially good addition.

Copy link
Contributor

@rparolin rparolin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, the documentation looks good. I made a minor comment about adding a walk through for cli debugging via cuda-gdb.


CUDA GDB supports extensions to the debugger written in Python. The Numba CUDA cuda-gdb pretty printer is such an extension. This is used to provide a more readable representation of Numba CUDA arrays in the debugger.

The Numba CUDA pretty printer extension is located in the ``numba-cuda/misc/gdb_print_extension.py`` file. This extension is loaded automatically by the ``launch.json`` file at the beginning of the debugging session. The `launch.json` example below assumes that you've opened the `debugging.code-workspace` workspace file in the `numba-cuda/examples/debugging` directory (due to the use of `..`). If the path to the ``gdb_print_extension.py`` file is not correct, simply edit the path to the ``misc/`` directory below.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gdb_print_extension.py appears to be in the numba repo -- not in numba-cuda.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❯ find . -type f | rg gdb_print
./.pixi/envs/cu-13-0-py313/lib/python3.13/site-packages/numba/misc/gdb_print_extension.py
./.pixi/envs/cu-13-0-py313/lib/python3.13/site-packages/numba/misc/__pycache__/gdb_print_extension.cpython-313.pyc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gdb_print_extension.py appears to be in the numba repo -- not in numba-cuda.

Yes, but to make it work properly with multiple address spaces it needed to be enhanced to use some CUDA specific APIs which are not available in GDB itself. I would suggest including this CUDA aware version of the gdb_print_extension.py file here instead of relying on the Numba repository. PR and merge request for that coming soon (if you concur).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that gdb_print_extension.py needs adding to Numba-CUDA. If it comes in a separate PR, then this PR needs to be merged only after the separate PR, otherwise we'll be publishing instructions that won't work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened #692 to add an unmodified copy of the GDB pretty printer from the Numba repository. CUDA specific enhancements will be handled with follow-on PRs / merge requests.

Copy link
Contributor

@rparolin rparolin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing filepaths in the documentation should be corrected.

.. image:: ../_static/starting-debugging.png
:alt: Starting debugging.

Because ``breakOnLaunch`` is set to ``true`` in the ``launch.json`` configuration file, the program will automatically stop on the first source line of any launched kernel, which can be useful for getting started debugging. If this is not desired, change ``breakOnLaunch`` to be ``false``.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe something like:

Setting breakOnLaunch to true in the launch.json configuration file causes the program to pause automatically on the very first source line of any kernel that is launched, which is a helpful starting point for the debugging process.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, incorporated in the next revision.


The following is a description of the functionality of each of the run control buttons and their corresponding CUDA GDB CLI commands.

* ``Continue`` will continue the program running until the next breakpoint or the program terminates. Equivalent to the ``continue`` command in CUDA GDB. When pressed, this icon will change to a pause icon. Pressing the pause icon will pause the program and return control to the user for further debugging.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"change to the pause icon"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incorporated.


CUDA GDB supports extensions to the debugger written in Python. The Numba CUDA cuda-gdb pretty printer is such an extension. This is used to provide a more readable representation of Numba CUDA arrays in the debugger.

The Numba CUDA pretty printer extension is located in the ``numba-cuda/misc/gdb_print_extension.py`` file. This extension is loaded automatically by the ``launch.json`` file at the beginning of the debugging session. The `launch.json` example below assumes that you've opened the `debugging.code-workspace` workspace file in the `numba-cuda/examples/debugging` directory (due to the use of `..`). If the path to the ``gdb_print_extension.py`` file is not correct, simply edit the path to the ``misc/`` directory below.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The launch.json example below assumes that"
For some reason in compiled version launch.json is not highlighted.

Image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found it. There were single backquotes instead of doubles. Addressed in the next version.


.. code-block:: bash

cd numba-cuda/examples/debugging

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mmason-nvidia Why didn't I find the examples directory?

test@cudatools-daniel:/daniel$ git clone https://github.com/NVIDIA/numba-cuda.git
Cloning into 'numba-cuda'...
remote: Enumerating objects: 7877, done.
remote: Counting objects: 100% (470/470), done.
remote: Compressing objects: 100% (248/248), done.
remote: Total 7877 (delta 371), reused 226 (delta 218), pack-reused 7407 (from 3)
Receiving objects: 100% (7877/7877), 4.89 MiB | 10.07 MiB/s, done.
Resolving deltas: 100% (5308/5308), done.
test@cudatools-daniel:
/daniel$ cd numba-cuda/
test@cudatools-daniel:~/daniel/numba-cuda$ ls
ci conda CONTRIBUTING.md greptile.json LICENSE.numba pixi.lock pyproject.toml scripts site-packages toolshed
CLA.md configs docs LICENSE numba_cuda pixi.toml README.md setup.py testing

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This merge request introduces the new examples/ directory - it won't show up in your clone until after it's been merged.

@mmason-nvidia
Copy link
Contributor Author

Missing filepaths in the documentation should be corrected.

Addressed in the next revision.

Comment on lines 27 to 29
To begin you’ll need an installation of CUDA Toolkit 13.1 and corresponding CUDA Driver for Linux. The installer for Linux can be downloaded from https://developer.nvidia.com/cuda-downloads. Follow the instructions provided on the download page to install it in ``/usr/local/cuda-13.1`` and create a symlink to it called ``/usr/local/cuda`` (which is handled by the install script).

You will also need a working Numba CUDA development environment. The documentation for Numba CUDA including the installation instructions can be found at: https://nvidia.github.io/numba-cuda. It is highly recommended that you use Anaconda, venv, or another Python virtual environment. The examples in this documentation are based on an Anaconda environment. In particular, make sure the required packages are installed in the virtual environment you wish to debug in.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are multiple ways a CUDA toolkit and driver could have been installed, so it seems better to me to just leave it at "you’ll need an installation of CUDA Toolkit 13.1 and corresponding CUDA Driver for Linux."

Is there any reason that a user specifically needs to use the CUDA installer from the download page and have the symlink to the 13.1 toolkit? Or would another valid method of installation work (e.g. driver installed from the driver downloads, CUDA toolkit from conda-forge)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's not really a user requirement to use that specific installation method, it was more a matter of offering a pointer to install directions. I've revised it to still include the download link, but now also mention that other CTK installation methods can be used instead.

Comment on lines 31 to 37
The following commands can be used to create a Numba CUDA development environment using Anaconda. This example creates a virtual environment called ``numba-cuda-debug`` using Python 3.12.

.. code-block:: bash

conda create --name numba-cuda-debug python=3.12
conda activate numba-cuda-debug
pip install cuda-python numba numba-cuda
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have installation instructions in the documentation: https://nvidia.github.io/numba-cuda/user/installation.html

This seems to be partially duplicating them, so I think it's probably be more concise and straightforward to omit this too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the "pip install" part, but left the Anaconda environment creation. Please let me know if you'd prefer that to be removed as well.

]
}

The CUDA GDB pretty printer uses the Python ``numpy`` package to inspect and print numpy arrays. The ``launch.json`` file sets up the environment necessary for debugging. However, the pretty printer runs inside of cuda-gdb and not as part of the Python program being debugged. This means that the ``numpy`` package must be installed in both:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A blank link is needed for the unordered list to render correctly:

Suggested change
The CUDA GDB pretty printer uses the Python ``numpy`` package to inspect and print numpy arrays. The ``launch.json`` file sets up the environment necessary for debugging. However, the pretty printer runs inside of cuda-gdb and not as part of the Python program being debugged. This means that the ``numpy`` package must be installed in both:
The CUDA GDB pretty printer uses the Python ``numpy`` package to inspect and print numpy arrays. The ``launch.json`` file sets up the environment necessary for debugging. However, the pretty printer runs inside of cuda-gdb and not as part of the Python program being debugged. This means that the ``numpy`` package must be installed in both:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. Fixed.

Copy link
Contributor

@gmarkall gmarkall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many thanks for the documentation!

I think as a tutorial of new functionality it's a good addition. In time I think we will want to evolve the documentation so that it presents it as less of a new feature / release note / tutorial all rolled into one, but keeping it in the form it is in now and getting it merged as soon as we can resolve the issues is good.

I left a couple of extra notes on the diff - mainly I think we should avoid telling people how to install things here, and a small issue with the rendering.

@gmarkall gmarkall added the 4 - Waiting on author Waiting for author to respond to review label Dec 16, 2025
@gmarkall
Copy link
Contributor

/ok to test 50bf976

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Comments (2)

  1. docs/source/user/debugging.rst, line 127-138 (link)

    logic: PYTHONPATH in documentation shows ../../misc but actual launch.json uses ${workspaceFolder} (line 50). These need to match.

  2. docs/source/user/debugging.rst, line 127 (link)

    logic: The file numba-cuda/misc/gdb_print_extension.py doesn't exist in the repository. Verify this file was included or update documentation to reference the correct path. Should gdb_print_extension.py be added in this PR, or does it already exist in a different location?

4 files reviewed, 2 comments

Edit Code Review Agent Settings | Greptile

Includes a new section on CUDA GDB command-line debugging.
Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Comments (4)

  1. docs/source/user/debugging.rst, line 329 (link)

    syntax: Missing space between "optionally" and "uses"

  2. docs/source/user/debugging.rst, line 58-59 (link)

    logic: The referenced gdb_print_extension.py file does not exist in the repository. The documentation mentions it should be in numba-cuda/misc/gdb_print_extension.py, but there is no misc/ directory at the repository root. The only misc/ directory is at numba_cuda/numba/cuda/misc/ and doesn't contain gdb_print_extension.py. This file needs to be added or the documentation and launch.json configuration need to be updated to either remove references to it or point to the correct location.

  3. examples/debugging/.vscode/launch.json, line 48 (link)

    logic: The path ${workspaceFolder}/../../misc does not exist in the repository. The gdb_print_extension.py file referenced in line 35 cannot be found at this location. Either the misc/gdb_print_extension.py file needs to be created at the repository root, or this path needs to be corrected to point to an existing location.

  4. docs/source/_static/shared-array.png, line 1 (link)

    style: This image file is included in the PR but is not referenced anywhere in the documentation. Consider removing it if unused, or add a reference to it in debugging.rst if it's meant to illustrate a debugging concept.

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

4 files reviewed, 4 comments

Edit Code Review Agent Settings | Greptile

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Comments (4)

  1. docs/source/user/debugging.rst, line 281 (link)

    logic: The gdb_print_extension.py file referenced here doesn't exist in the repository. The documentation mentions it should be in numba-cuda/misc/gdb_print_extension.py, but the misc/ directory doesn't exist. This file needs to be added to the repository for the pretty printer functionality to work.

  2. docs/source/user/debugging.rst, line 316 (link)

    logic: References misc/gdb_print_extension.py file which doesn't exist in the repository. The launch.json configuration will fail to load the pretty printer extension.

  3. examples/debugging/.vscode/launch.json, line 48 (link)

    logic: PYTHONPATH references ${workspaceFolder}/../../misc but the misc/ directory doesn't exist in the repository. The pretty printer extension file needs to be added at this location.

  4. docs/source/user/debugging.rst, line 329 (link)

    syntax: typo: optionallyuses should be two words

4 files reviewed, 4 comments

Edit Code Review Agent Settings | Greptile

@mmason-nvidia
Copy link
Contributor Author

Regarding comments 1, 2, and 3, the gdb_print_extension.py file is part of a separate merge request, which will be merged before this one.

Additional Comments (4)

  1. docs/source/user/debugging.rst, line 281 (link)
    logic: The gdb_print_extension.py file referenced here doesn't exist in the repository. The documentation mentions it should be in numba-cuda/misc/gdb_print_extension.py, but the misc/ directory doesn't exist. This file needs to be added to the repository for the pretty printer functionality to work.
  2. docs/source/user/debugging.rst, line 316 (link)
    logic: References misc/gdb_print_extension.py file which doesn't exist in the repository. The launch.json configuration will fail to load the pretty printer extension.
  3. examples/debugging/.vscode/launch.json, line 48 (link)
    logic: PYTHONPATH references ${workspaceFolder}/../../misc but the misc/ directory doesn't exist in the repository. The pretty printer extension file needs to be added at this location.
  4. docs/source/user/debugging.rst, line 329 (link)
    syntax: typo: optionallyuses should be two words

4 files reviewed, 4 comments

Edit Code Review Agent Settings | Greptile

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Comments (3)

  1. docs/source/user/debugging.rst, line 58-59 (link)

    logic: The misc/gdb_print_extension.py file referenced here doesn't exist in the repository. Verify this file exists or update the documentation.

  2. docs/source/user/debugging.rst, line 330-336 (link)

    logic: This section references numba-cuda/misc/gdb_print_extension.py but this file doesn't exist in the repository. The misc/ directory itself is also missing. Is the gdb_print_extension.py file supposed to be added in this PR, or does it exist elsewhere?

  3. examples/debugging/.vscode/launch.json, line 47-48 (link)

    logic: The path ${workspaceFolder}/../../misc points to a non-existent directory. The misc/gdb_print_extension.py file doesn't exist in the repository. Should this file be included in the PR or is it provided separately by CUDA Toolkit?

4 files reviewed, 3 comments

Edit Code Review Agent Settings | Greptile

@mmason-nvidia
Copy link
Contributor Author

The initial version of the gdb_print_extension.py file has been submitted, which should resolve most if not all the remaining issues.

/ok to test 2833d91

@gmarkall gmarkall added 4 - Waiting on reviewer Waiting for reviewer to respond to author and removed 4 - Waiting on author Waiting for author to respond to review labels Jan 21, 2026
@gmarkall
Copy link
Contributor

/ok to test e2b0f91

- Exclude debugging example from the debug_prints check
- Add SPDX header to debugging example. I think the env line isn't
  needed so I just replaced it.
- Fix ruff-identified issues
- Exclude the committed VSCode workspace file from the SPDX check.
@gmarkall
Copy link
Contributor

/ok to test e36bd4b

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 files reviewed, 2 comments

Edit Code Review Agent Settings | Greptile

CUDA thread hit application kernel entry function breakpoint, hello<<<(1,1,1),(8,1,1)>>> (input=..., output=...) at hello.py:18
18 gid = cuda.grid(1)
(cuda-gdb) list
13 # Editor window, and are shown as red dots next to the line numbers. The can also be set
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

syntax: Grammar error: "The can also be set" should be "They can also be set"

Suggested change
13 # Editor window, and are shown as red dots next to the line numbers. The can also be set
13 # Editor window, and are shown as red dots next to the line numbers. They can also be set

# breakpoints.
#
# Breakpoints can also be set by clicking to the left of the source line numbers in the
# Editor window, and are shown as red dots next to the line numbers. The can also be set
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

syntax: Same grammar error as in the documentation: "The can also be set" should be "They can also be set"

Suggested change
# Editor window, and are shown as red dots next to the line numbers. The can also be set
# Breakpoints can also be set by clicking to the left of the source line numbers in the

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

4 - Waiting on reviewer Waiting for reviewer to respond to author

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEA] Add documentation for Numba CUDA debugging

5 participants