Add safe Python-extension module C API headers by jakelishman · Pull Request #15762 · Qiskit/qiskit

jakelishman · 2026-03-05T13:38:23Z

With the vtables now compiled into the _accelerate object and stored in suitable PyCapsules, the last step to exposing the complete ability to compile Python extension modules is providing header-file support for actually using the result. This support is modelled on NumPy.

We generate alternate versions of the function declarations as part of the pyext build script, which are loaded (instead of the standard function prototypes) when the QISKIT_PYTHON_EXTENSION macro is set prior to the inclusion of qiskit.h. These declarations are all pre-processor macros that resolve to compile-time constant offset lookups into the vtables stored in the PyCapsules, except we cache the internal pointer of each PyCapsule into a compilation-unit-local static. If we didn't have this cache, all function calls would have Python-API overhead and require an attached Python thread state (holding the GIL).

The cache population is done by a new header-only function qk_import defined in (the non-stub version of) funcs_py.h, which then must be called before any C API function. This will almost invariably be done inside the PyInit_* module-initialisation function of the extension.

The cache mechanism introduced in this commit is local to a single translation unit. It is possible to extend this to allow sharing it between different translation units, but since this necessarily requires exposing a non-static symbol out of a library, we will have to take care to do it with a mechanism that allows the user to override the names used.

Depends on:

Close #15572

qiskit-bot · 2026-03-05T13:38:28Z

One or more of the following people are relevant to this code:

@abbycross
@beckykd
@Qiskit/terra-core

coveralls · 2026-03-07T00:13:59Z

Pull Request Test Coverage Report for Build 22786547376

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

461 of 557 (82.76%) changed or added relevant lines in 12 files are covered.
14 unchanged lines in 4 files lost coverage.
Overall coverage increased (+0.09%) to 87.77%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
crates/bindgen-c/src/main.rs	0	2	0.0%
crates/pyext/src/capi.rs	33	36	91.67%
crates/pyext/build.rs	101	106	95.28%
crates/cext/src/transpiler/target.rs	0	9	0.0%
crates/cext/src/dag.rs	0	14	0.0%
crates/cext-vtable/src/impl_.rs	35	66	53.03%
crates/cext/src/py.rs	0	32	0.0%

Files with Coverage Reduction	New Missed Lines	%
crates/circuit/src/parameter/parameter_expression.rs	1	86.99%
crates/qasm2/src/expr.rs	1	93.82%
crates/qasm2/src/lex.rs	2	92.29%
crates/circuit/src/parameter/symbol_expr.rs	10	73.89%

Totals
Change from base Build 22767384343:	0.09%
Covered Lines:	101352
Relevant Lines:	115474

💛 - Coveralls

Cryoris · 2026-03-12T09:44:09Z

crates/pyext/include/qiskit/funcs_py.h

+// We rely on `qiskit.h` to `#include <Python.h>` on our behalf, since it needs to be included
+// before all other includes. (Though really, a user should also have included it themselves,
+// surely, if they're delcaring a Python extension module.)


I assume Python.h has an include guard, so why not just include it here anyways?

Because it has to be included before all other includes - if we put it here, it must be redundant because this isn't a top-level import.

crates/pyext/include/qiskit/funcs_py.h

crates/pyext/build.rs

Cryoris · 2026-03-12T10:43:16Z

crates/pyext/include/qiskit/funcs_py.h

+    if (!_Qk_API_QI)
+        return -1;
+
+    // TODO: any validity checks on the version of the Qiskit API?


Could you explain what kind of checks you're thinking of? Since the vtables and functions are all autogenerated they should be consistent, right? Does this refer to coherence checks in case the generation goes wrong?

Oh, I forgot this was still here. The check I have in mind is a runtime version check; the compiled extension knows what header it's compiled with (it's the version info in there) but at runtime, it's got no way of knowing what version the library is. We're supposed to keep our ABI/API safe and forwards compatible, but just for forwards thinking, I'm thinking we should maybe have a version handshake during the import, so we can insert backwards-compatibility shims if we need to, or more basically just warn if there are known incompatibilities.

I might want to squeeze in a qk_version() function into the release just so we can put it at slot number 0 and have it available in the future, but there's no need to do anything with it in the header file in 2.4.0rc1, I think.

#15831 is the qk_api_version function, though I haven't made the header files for 2.4 check it here. I still would like to get it in to give us the option in the future, though.

jakelishman · 2026-03-18T18:36:15Z

I've just rebased the PR here to make the diff clean, but I need to come back later this evening and write the commit message and the release note.

With the vtables now compiled into the `_accelerate` object and stored in suitable `PyCapsule`s, the last step to exposing the complete ability to compile Python extension modules is providing header-file support for actually using the result. This support is modelled on NumPy. We generate alternate versions of the function declarations as part of the `pyext` build script, which are loaded (instead of the standard function prototypes) when the `QISKIT_PYTHON_EXTENSION` macro is set prior to the inclusion of `qiskit.h`. These declarations are all pre-processor macros that resolve to compile-time constant offset lookups into the vtables stored in the `PyCapsule`s, except we cache the internal pointer of each `PyCapsule` into a compilation-unit-local `static`. If we didn't have this cache, _all_ function calls would have Python-API overhead and require an attached Python thread state (holding the GIL). The cache population is done by a new header-only function `qk_import` defined in (the non-stub version of) `funcs_py.h`, which then must be called _before_ any C API function. This will almost invariably be done inside the `PyInit_*` module-initialisation function of the extension. The cache mechanism introduced in this commit is local to a single translation unit. It is possible to extend this to allow sharing it between different translation units, but since this necessarily requires exposing a non-`static` symbol out of a library, we will have to take care to do it with a mechanism that allows the user to override the names used.

jakelishman · 2026-03-18T23:55:20Z

Ok, the release note and commit message are now all written, so this should be good to review.

mrossinek

I found mostly typos. Other than that, I think this looks good 👍

crates/pyext/include/qiskit/funcs_py.h

mrossinek · 2026-03-19T08:14:33Z

crates/pyext/build.rs

+            }
+            ir::Type::Path(p) => acc.push_str(p.export_name()),
+            ir::Type::Primitive(ty) => acc.push_str(ty.to_repr_c(config)),
+            ir::Type::Array(..) => todo!("array types not yet handled"),


How likely are these to come up in the future? Do you want a (or more) tracking issue(s) for this as well as the other not-handled cases below?

Really pretty unlikely I think - minimum-size arrays as function arguments in C are super uncommon. I put in the panic so that if we ever do try it, it's marked as needing extra implementation.

Makes sense 👍

crates/pyext/Cargo.toml

releasenotes/notes/c-extensions-07460be1c3bb0074.yaml

docs/cdoc/config.rst

Co-authored-by: Max Rossmannek <21973473+mrossinek@users.noreply.github.com> Co-authored-by: Jake Lishman <jake@binhbar.com>

raynelfss

I again couldn't find anything concerning here. I assume this isn't being used yet as we haven't designed any Python extensions yet but the logic seems to make sense.

From what I could read, based on usage of an extension definition, this module will first import qiskit and use the vtable to generate the functions header file.

That said, I only had one question about a check

raynelfss · 2026-03-19T14:57:56Z

crates/pyext/build.rs

+                ty,
+                array_length,
+            } = arg;
+            assert!(array_length.is_none(), "array arguments not handled");


Is this supposed to be array_length.is_some() or am I reading this wrong?

The assertion is that the argument isn't an array - if it is, there's a problem because we explicitly don't handle that case. I didn't want the potential bug to pass silently.

jakelishman · 2026-03-19T15:55:49Z

I assume this isn't being used yet as we haven't designed any Python extensions yet but the logic seems to make sense.

It's not used within the Qiskit/qiskit repository itself yet, but I will follow up adding integration tests involving it. There's just more repo admin work to do about that because I'll most likely need/want a more powerful/less wildly over-opinionated command runner than tox.

From what I could read, based on usage of an extension definition, this module will first import qiskit and use the vtable to generate the functions header file.

Well, the header file is generated in the build script of pyext, so before "Qiskit" as a Python-space package actually exists yet. The qk_import function is inlined into downstream extension modules, and that ensures (at import time) that Python-space qiskit is imported, then caches the vtable pointers from the PyCapsule objects stored inside the qiskit._accelerate.capi module object.

raynelfss

Thank you for quickly addressing my comments and explaining how things work here. I don't have much more to add, so feel free to merge this after @alexanderivrii takes a look

jakelishman added this to the 2.4.0 milestone Mar 5, 2026

jakelishman requested a review from a team as a code owner March 5, 2026 13:38

github-project-automation bot added this to Qiskit 2.4 Mar 5, 2026

jakelishman added on hold Can not fix yet Changelog: Added Add an "Added" entry in the GitHub Release changelog. C API Related to the C API labels Mar 5, 2026

github-project-automation bot moved this to Ready in Qiskit 2.4 Mar 5, 2026

jakelishman mentioned this pull request Mar 5, 2026

Provide a safe way for Python extension modules to use the C API #15572

Closed

jakelishman force-pushed the c/pycapsule branch 2 times, most recently from 2a1471a to 62752e0 Compare March 6, 2026 23:39

alexanderivrii assigned alexanderivrii and raynelfss Mar 10, 2026

Cryoris reviewed Mar 12, 2026

View reviewed changes

crates/pyext/include/qiskit/funcs_py.h Show resolved Hide resolved

Cryoris reviewed Mar 12, 2026

View reviewed changes

crates/pyext/build.rs Outdated Show resolved Hide resolved

Cryoris reviewed Mar 12, 2026

View reviewed changes

jakelishman mentioned this pull request Mar 17, 2026

Add C API vtable to pyext #15761

Merged

3 tasks

jakelishman force-pushed the c/pycapsule branch from 62752e0 to d6c1640 Compare March 18, 2026 18:33

jakelishman removed the on hold Can not fix yet label Mar 18, 2026

jakelishman mentioned this pull request Mar 18, 2026

Add runtime-version checking qk_api_version #15831

Merged

jakelishman force-pushed the c/pycapsule branch from d6c1640 to 6ecaa68 Compare March 18, 2026 23:51

jakelishman added 2 commits March 19, 2026 00:33

Remove dbg! call from build script

9bae95c

Remove unneeded pyo3-build-config dependence

0d54073

mrossinek reviewed Mar 19, 2026

View reviewed changes

Fix typos

22229d6

Co-authored-by: Max Rossmannek <21973473+mrossinek@users.noreply.github.com> Co-authored-by: Jake Lishman <jake@binhbar.com>

mrossinek approved these changes Mar 19, 2026

View reviewed changes

raynelfss reviewed Mar 19, 2026

View reviewed changes

raynelfss approved these changes Mar 19, 2026

View reviewed changes

raynelfss added this pull request to the merge queue Mar 19, 2026

Merged via the queue into Qiskit:main with commit 55f4eb1 Mar 19, 2026
25 checks passed

github-project-automation bot moved this from Ready to Done in Qiskit 2.4 Mar 19, 2026

jakelishman deleted the c/pycapsule branch March 19, 2026 17:36

Conversation

jakelishman commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qiskit-bot commented Mar 5, 2026

Uh oh!

coveralls commented Mar 7, 2026

Pull Request Test Coverage Report for Build 22786547376

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jakelishman commented Mar 18, 2026

Uh oh!

jakelishman commented Mar 18, 2026

Uh oh!

mrossinek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raynelfss left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jakelishman commented Mar 19, 2026

Uh oh!

raynelfss left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jakelishman commented Mar 5, 2026 •

edited

Loading