Add linting of vtable slots compared to exports#15778
Add linting of vtable slots compared to exports#15778jakelishman wants to merge 7 commits intoQiskit:mainfrom
Conversation
|
One or more of the following people are relevant to this code:
|
4c429bd to
c1f9165
Compare
jan-an
left a comment
There was a problem hiding this comment.
Hi Jake, thank you so much for this PR.
Looks good to me, I just have some very minor questions / comments.
.github/workflows/lint_docs.yml
Outdated
| - name: Run lint | ||
| run: | | ||
| make cformat | ||
| tools/check_slots.bash |
There was a problem hiding this comment.
Given that this lint check may not be relevant for every PR but only when there are changes to files such as in crates/cext or crates/cext-vtable, would it make sense to want to decouple this to have that kind of control, say such as via a dedicated lint check for C API related check that is skipped for example, on a python-changes-only sort of PR but runs when there are relevant changes?
There was a problem hiding this comment.
A related comment that perhaps answers my above question. I noticed that further into the PR that we also have more changes where we have separated standalone C API exports (qiskit.h) from Python-extension-specific exports (qiskit/funcs_py.h), meaning that this lint check would need to cover a larger set of potentially modifiable files. So running it on every PR is likely the simpler and safer choice?
There was a problem hiding this comment.
Your point is totally valid - ideally we'd only do differential linting - but it's pretty hard to get working reliably in practice. You really don't want false negatives in your lint/test suite, and it's very easy to accidentally introduce false negatives when we do incremental checking. It's easier to err on the safe side and just always do it - the build should not be expensive.
jan-an
left a comment
There was a problem hiding this comment.
Just a few more additional minor comments and suggestions correcting typos. Thank you!
| .add_child(250, &circuit_library::FUNCTIONS); | ||
| pub static FUNCTIONS_QI: ExportedFunctions = | ||
| ExportedFunctions::empty().add_child(0, &sparse_observable::FUNCTIONS); | ||
| pub use transpiler::FUNCTIONS as FUNCTIONS_TRANSPILE; |
There was a problem hiding this comment.
Is there a reason FUNCTIONS_TRANSPILE is re-exported as an alias whereas the others are directly declared statically?
There was a problem hiding this comment.
This is actually introduced in a parent commit of this one (you've reviewed a complete changeset, not just the single change of the last commit), and I talked about it a bit more here: #15761 (comment)
mrossinek
left a comment
There was a problem hiding this comment.
Overall this looks great. Just a few comments from my side 👍
| ("circuit", &qiskit_cext_vtable::FUNCTIONS_CIRCUIT), | ||
| ("transpile", &qiskit_cext_vtable::FUNCTIONS_TRANSPILE), | ||
| ("qi", &qiskit_cext_vtable::FUNCTIONS_QI), |
There was a problem hiding this comment.
Just noting that, this hard-coded list of VTables to check against needs to be kept in sync with the root-level VTables of the crate that is being checked. While probably not ideal, this is the simplest solution for now. But this should be documented over in the cext-vtable crate accordingly.
There was a problem hiding this comment.
It does, yeah. The trouble is that there are several associations of name: vtable used in various places (C header files, vtable creation, this linter, etc) and I didn't want the vtables to actually have to own them all. If we come up with something better, we can swap to it.
There was a problem hiding this comment.
This is definitely beyond scope right now (i.e. for 2.4) but I think it would be great if we could add some tests to make sure that these lints actually catch the errors that they are supposed to catch.
One edge case that I have in mind immediately is for example duplicate exported functions but across different root-level vtables..
There was a problem hiding this comment.
Fwiw this should work - I tested it locally and used it to catch errors - but yeah, it would be better to have tests.
| cargo run -p qiskit-bindgen-c -- lint-slots -c crates/cext | ||
| ``` | ||
|
|
||
| This checks various coherence properties between the declared `extern "C"` functions in |
There was a problem hiding this comment.
What is the point of allowing --cext-path be specified from the CLI while hard-coding qiskit-cext-vtable internally?
I'm not suggesting we relax the hard-coding of the latter, just questioning the flexibility on the former instead of hard-coding that, too.
There was a problem hiding this comment.
because qiskit_cext_vtable is a compiled-in dependency, so cargo arranges for it to be compiled into the binary, but cbindgen needs to be pointed at the source tree on disk to parse qiskit-cext
| all functions have a slot and there are no duplicates; in other words, that each exported function | ||
| is referenced exactly onces. | ||
|
|
||
| Note that this command does not test for ABI compatibility between different Qiskit versions. |
There was a problem hiding this comment.
Is this something you plan to be able to lint in the future?
There was a problem hiding this comment.
Yes - I have half of it on a local laptop, and it's just waiting for me to finish off writing a simple diff algorithm into it and pushing it up
crates/bindgen-c/README.md
Outdated
|
|
||
| - `no-export`: assert that the function will not be in the slots list (such as for functions | ||
| deprecated before the introduction of the vtables). | ||
| - `allow-duplicates`: allow functions to be in more than one slot (if, for example, we want to |
There was a problem hiding this comment.
Do you have a motivation for allowing this?
There was a problem hiding this comment.
Not really a useful motivation: the reason is that I have a lint that forbids you from doing it, so I also added the escape valve to bypass the lint. I don't have a use in mind for bypassing that lint.
There was a problem hiding this comment.
Fair enough I suppose. Although given that you don't actually support it below, not having this bypass for the time being might be the easier fix rather than supporting it within the 2.4 time window..
While Qiskitgh-15778 is not yet merged, we don't have CI enforcement that the vtable slots are always completely up-to-date. Several PRs adding new C API functions merged between the merge-base of Qiskitgh-15761[^1] and `main` at its time of merge (and subsequently). This PR uses the machinery in Qiskitgh-15778 to bring the slots completely up-to-date. [^1]: 888c877: Add C API vtable to `pyext`
While gh-15778 is not yet merged, we don't have CI enforcement that the vtable slots are always completely up-to-date. Several PRs adding new C API functions merged between the merge-base of gh-15761[^1] and `main` at its time of merge (and subsequently). This PR uses the machinery in gh-15778 to bring the slots completely up-to-date. [^1]: 888c877: Add C API vtable to `pyext`
A risk of having the definitions of C API functions (`cext`) separate to the crate defining the static vtable structure (`cext-vtable`) is that these can more easily get out of sync. This extends the capabilities of `qiskit-bindgen-c` to add linting of those two crates against each other, providing a convenience wrapper (`tools/check_slots.bash`) to run a suitable set of checks, and enforcing this in CI.
c1f9165 to
c8033da
Compare
mrossinek
left a comment
There was a problem hiding this comment.
Nice, I like the way you went about allowing duplicates (even though we still don't have a use case for it, per se) 👍
| // SAFETY: per documentation, we are attached to a Python interpreter, `object` points to a | ||
| // valid Python object and `address` points to enough space to write a pointer. |
There was a problem hiding this comment.
Although valid changes, these seem unrelated to this PR
There was a problem hiding this comment.
They're from Janani's review above
| your compiler's search path using :func:`qiskit.capi.get_include`. | ||
| build: | ||
| - Packages dependening on the Qiskit C API to build can now specify Qiskit as a Python-space build | ||
| - Packages depending on the Qiskit C API to build can now specify Qiskit as a Python-space build |
There was a problem hiding this comment.
Same as previous comment: relevant but unrelated
jan-an
left a comment
There was a problem hiding this comment.
Thanks for the changes Jake, LGTM 👍🏻
There seem to be a couple of failures in the CI jobs, I've re-run them just to be sure there is something needing correction.. if it fails again, maybe some minor change maybe needed to account for that |
A risk of having the definitions of C API functions (
cext) separate to the crate defining the static vtable structure (cext-vtable) is that these can more easily get out of sync. This extends the capabilities ofqiskit-bindgen-cto add linting of those two crates against each other, providing a convenience wrapper (tools/check_slots.sh) to run a suitable set of checks, and enforcing this in CI.Depends on:
pyext#15761I will also write a follow-on linter to this that checks for ABI stability between different Qiskit versions (some of the code structure of this patch already indicates where I'm going here - I cut this patch out of a WIP commit).
This isn't a public code feature for 2.4, just an internal QA tool for developers.