Leakage-aware GST #410

rileyjmurray · 2024-03-19T19:15:31Z

TODOs

Rewrite this PR description into an acceptable summary. (Marking as ready for review now!)
Justify or revert the change in bc7cb8b. (Justified.)
Write explanations of report metrics in pygsti/report/reportables.py.
Move various helper functions from my notebooks (i.e., the ones I make static functions of an Infrastructure class) into appropriate places within pyGSTi.
Make a leakage-friendly built-in basis.
An example notebook.

Leakage analysis that's important, but out of scope for this PR.

systems with > 1 qubit.
systems with > 1 leakage level.
CPTP kite models

rileyjmurray · 2025-03-25T14:36:05Z

EDIT: we now have an open issue about handling this sort of thing #633.

Text from an email Piper sent (not showing plots since they contained real data):

I was looking at the model violation tab, trying to work out why the smaller L models weren’t fitting the data as well as might be expected as quantified by Nsigma. I think this maybe be a function of how we calculate Nsigma = (2delta logl -k)/sqrt(2k). The number of non-gauge model parameters N_p is reported as 351 in the leakage-enhanced GST report (I don’t have any particular sense of whether this is correct). The number of data points N_s is 193 for L=1. This means that k (the degrees of freedom) is negative if you just calculate it as N_s-N_p. In cases like this, pyGSTi chooses k=1. k remains small until L is fairly large, which means two delta logl must be quite small for Nsigma to be small. This explains why we don’t see any red boxes in the per sequence detail for the L=1 iteration (and Nsigma is large enough that I would expect to see some or at least more dark gray squares than can be seen).

If you directly compare two delta logl between the two fits, you can see that the two delta logl is in fact smaller in the leakage-enhanced report than in the standard report.

I think the solution here is just to make sure that the first iteration of leakage-aware GST uses L=2 instead of L=1. (Or, more generally, we use the smallest L where N_s > N_p.)

rileyjmurray

Leaving file-level comments so @coreyostrove and I can figure out what to split out from this PR.

…inearOperator (not just plain numpy ndarrays)

…frobenius norm computation. Bugfix in test_optools.py.

…lots of in-line comments explaining the logic.

…poses

…anch that omitted jac as a keyword argument to scipy.minimize, even if jac was None.

…nt fidelity from the calculation of that metric itself. This makes it easier to reuse the setup code in other metrics. While making this change, also make the setup more efficient by using TensorProduct bases from the beginning.

…imizers. Right now only entanglement fidelity and jtracedist have implementations that can consider leakage. (So there are no leakage-aware metrics for SPAM.)

…dded annotations to all gauage optimization objectives used in the non-LS optimizer, indicating if that particular objective has support for leakage-aware metrics.

… while. The tests only consider when no leakage is modeled. New tests wil be needed for when theres a leakage dimension

coreyostrove · 2025-08-07T01:34:24Z

Text from an email Piper sent (not showing plots since they contained real data):

I was looking at the model violation tab, trying to work out why the smaller L models weren’t fitting the data as well as might be expected as quantified by Nsigma. I think this maybe be a function of how we calculate Nsigma = (2delta logl -k)/sqrt(2k). The number of non-gauge model parameters N_p is reported as 351 in the leakage-enhanced GST report (I don’t have any particular sense of whether this is correct). The number of data points N_s is 193 for L=1. This means that k (the degrees of freedom) is negative if you just calculate it as N_s-N_p. In cases like this, pyGSTi chooses k=1. k remains small until L is fairly large, which means two delta logl must be quite small for Nsigma to be small. This explains why we don’t see any red boxes in the per sequence detail for the L=1 iteration (and Nsigma is large enough that I would expect to see some or at least more dark gray squares than can be seen).
If you directly compare two delta logl between the two fits, you can see that the two delta logl is in fact smaller in the leakage-enhanced report than in the standard report.

I think the solution here is just to make sure that the first iteration of leakage-aware GST uses L=2 instead of L=1. (Or, more generally, we use the smallest L where N_s > N_p.)

This isn’t a uniquely leakage-related shortcoming, I’ve encountered this in other settings too (with really aggressive FPR schemes when not forcing inclusion of LGST circuits, for example) and I suggest we open up a separate github issue for this and give some additional thought to how we should be approaching the statistics in this case.

coreyostrove

Just finished my first pass through. This is a ton of effort and these contributions are sincerely appreciated. Thank you as always for the excellent work, @rileyjmurray!

Of the comments I've left, the only one I am deeply concerned about is the potential bug that I've pointed out in the cost function for gauge optimization as it pertains to instruments. I have good reason to suspect (I mean, the git history is pretty clear and this present on develop so it isn't really a suspicion) that in the event there is a bug here it is very likely you had nothing to do with introducing it, but since you're already knee deep in this code I'd ask that you take a look and if necessary clean this up on our behalf.

Rest of my comments are either minor clarifying questions or otherwise nice-to-have but non-blocking requests. I've also added my concurrence with a couple comments/suggestions from @nkoskelo.

pygsti/algorithms/gaugeopt.py

coreyostrove · 2025-08-07T01:59:13Z

pygsti/algorithms/gaugeopt.py

+        assert gates_metric != "frobeniustt"
+        assert spam_metric  != "frobeniustt"
+        # assert spam_metric == gates_metric
+        # ^ Erik and Corey said these are rarely used. I've removed support for
+        # them in this codepath (non-LS optimizer) in order to make it easier to
+        # read my updated code for leakage-aware metrics. It wouldn't be hard to
+        # add support back, but I just want to keep things simple. -- Riley


Please add an error message for the AssertionError indicating that these metrics are not supported when doing leakage-aware gauge optimization (i.e. n_leak>0).

coreyostrove · 2025-08-07T02:14:01Z

pygsti/algorithms/gaugeopt.py

+                Leakage-aware metric supported, per implementation in mdl.frobeniusdist.
+                Refer to how mdl.frobeniusdist handles the case when transform_mx_arg
+                is a tuple in order to understand how the leakage-aware metric is defined.  
+                """


I think there might be a (likely preexisting) bug here related to instruments. For the fidelity and tracedistance metrics I’m not seeing the instrument elements being accessed (except in passing in the spam metric computation where it is just used to filter the item weights). For frobenius this is harder to see, but I took a look at the implementation of the ‘frobeniusdist’ method for ExplicitOpModels here and I’m not seeing instruments there either.

coreyostrove · 2025-08-07T02:32:58Z

pygsti/circuits/circuit.py

+            elif self._static and x._static and self._hash != x._hash:
+                return False
            else:
-                if self._static and x._static:
-                    return self._hash == x._hash
-                else:
-                    return self.tup == x.tup
+                return self.tup == x.tup


These look functionally equivalent to me. I’ll take your word for it, but for my own edification what about the original version was incorrect?

coreyostrove · 2025-08-07T02:39:09Z

pygsti/data/dataset.py

+        opLabels = set()
+        empty_label = _Label(())
+        for circuit in self:
+            for layer in circuit.layertup:


This is a good suggestion and I think it would be an independently useful primitive for circuits to return their unique gate labels, probably with some flags to control matching behavior (e.g. just label names versus full labels with sslbls).

pygsti/models/explicitcalc.py

coreyostrove · 2025-08-08T03:20:30Z

pygsti/protocols/confidenceregionfactory.py

+        sym_err_rel = sym_err_abs / _la.norm(hessian)
+        assert(sym_err_rel < TOL)
+        hessian += hessian.T
+        hessian /= 2


This is just using the following factoid about quadratic forms, right? https://en.wikipedia.org/wiki/Quadratic_form#General_case

coreyostrove · 2025-08-08T03:31:57Z

pygsti/report/reportables.py

It looks like most/all of these reportable functions hard code in that n_leak is one. I don't think there is anything to flag a user specifying something different in construct_standard_report, though. That is fine for now, but maybe add an assertion for this to avoid unexpected behavior? (i.e. a user being confused by why this was quietly changed under the hood).

Good point. I'll update the construct_standard_report docstring and raise an exception if the user passes in n_leak other than 0 or 1.

pygsti/tools/leakage.py

coreyostrove · 2025-08-08T03:45:26Z

pygsti/tools/sdptools.py

Similar comment to the one I left on the leakage model regarding docstrings and API-level.

I'm going to say that everything here is in the private API for now.

… suppression we do in sdptools.py)

… a result. Type annotations, docs, and streamlined implementation of frobeniusdist_squared in linearop.py, effect.py, and state.py.

pygsti/tools/leakage.py

…ix bugs introduced in the last couple of commits.

rileyjmurray · 2025-08-19T07:06:00Z

@coreyostrove I think I've addressed your comments. Just pushed what should be the last round of changes.

coreyostrove · 2025-08-19T14:36:57Z

@coreyostrove I think I've addressed your comments. Just pushed what should be the last round of changes.

I think the only major outstanding concern here is the bug I alluded to with gauge optimization for instruments. Since this is a pre-existing bug I am alright with tabling this for the purposes of this PR, but we should open up a new github issue to track it. If you have time today can you open this issue? If not I can do so myself later this evening.

coreyostrove

Fantastic work, @rileyjmurray! Thank you for the significant effort this work represents and your contributions advancing our leakage modeling capabilities. Merge at will.

One last request as you merge things in, can you flesh out the summary in the PR? That’d be helpful as I prepare release notes (and for posterity).

rileyjmurray force-pushed the leakage branch from f422726 to 2f23347 Compare September 24, 2024 21:07

rileyjmurray changed the title ~~WIP: leakage-aware gauge optimization~~ WIP: leakage-aware gauge optimization -- bad commit history Sep 24, 2024

rileyjmurray changed the base branch from master to develop September 24, 2024 21:15

rileyjmurray changed the title ~~WIP: leakage-aware gauge optimization -- bad commit history~~ WIP: leakage-aware gauge optimization Sep 24, 2024

rileyjmurray mentioned this pull request Sep 28, 2024

Maintenance of the Basis class #493

Merged

rileyjmurray force-pushed the leakage branch from 48eac7d to 0346cca Compare January 17, 2025 16:18

rileyjmurray changed the title ~~WIP: leakage-aware gauge optimization~~ WIP: leakage-aware GST Mar 25, 2025

rileyjmurray force-pushed the leakage branch from 904c2d9 to b3aba0d Compare July 24, 2025 18:54

rileyjmurray commented Jul 24, 2025

View reviewed changes

rileyjmurray mentioned this pull request Jul 25, 2025

à la carte standard GST reports, and exception handling for computing metrics #615

Merged

rileyjmurray force-pushed the leakage branch from 2608cdd to 7b38788 Compare July 25, 2025 18:49

rileyjmurray mentioned this pull request Jul 25, 2025

New gauge group, plus minor changes to error generator classes #616

Merged

rileyjmurray added 17 commits July 29, 2025 11:26

bugfix for my attemped efficiency improvement

f529155

extend basistools to gracefully accomodate objects that derive from L…

d389349

…inearOperator (not just plain numpy ndarrays)

update function in reportables to indicate API limitation. Bugfix in …

d7e880e

…frobenius norm computation. Bugfix in test_optools.py.

update implementation of state fidelity

9489fbd

comments. About to remove.

b402562

extend implementation to support bases other than pauli-product. Add …

43ce7af

…lots of in-line comments explaining the logic.

remove unused and broken function that I only added for debugging pur…

a4eca61

…poses

comment clarification

98cae3a

clean up implementation of ExplicitOpModelCalc.residuals

f47070b

slightly simplify minimize(...) function. There was an unnecessary br…

45e6b17

…anch that omitted jac as a keyword argument to scipy.minimize, even if jac was None.

add implementation of leaky jtracedist

4899bb8

enable gauge optimization with leakage-aware metrics using non-LS opt…

714f0e1

…imizers. Right now only entanglement fidelity and jtracedist have implementations that can consider leakage. (So there are no leakage-aware metrics for SPAM.)

support for leakage-aware Frobenius distance with non-LS optimizer. A…

c5e2fe1

…dded annotations to all gauage optimization objectives used in the non-LS optimizer, indicating if that particular objective has support for leakage-aware metrics.

left out a function needed for code in the last commit to work

b2ca23a

tests for the leaky_entanglement_fidelity function that Ive had for a…

6de1eaf

… while. The tests only consider when no leakage is modeled. New tests wil be needed for when theres a leakage dimension

fix syntax mistake

a863ab8

rileyjmurray and others added 8 commits August 6, 2025 14:08

change variable name. Leave comment.

d149bb0

remove unused import

ef19093

import formatting

ac7c99f

clarifying comment

d034343

type annotations

0983337

docs and types for tensorized_with_eye

886b75a

type annotations and docstring fixes (need double-backslash for math)

2c1378e

Merge branch 'develop' into leakage

822148a

rileyjmurray added 2 commits August 7, 2025 07:49

typing and updated comment

49cfc93

fix import and types

9378e89

coreyostrove requested changes Aug 8, 2025

View reviewed changes

rileyjmurray added 8 commits August 8, 2025 12:03

fix deprecation warning. style.

a46b33a

minor bugfix in leaky_entanglement_fidelity. Variable names

5dca93c

type annotations

0433764

suppress warnings in tests (seems pytest overrides the normal warning…

83678ea

… suppression we do in sdptools.py)

add IdentityOperator class to matrixtools.py. Simplify gaugeopt.py as…

2f83ab8

… a result. Type annotations, docs, and streamlined implementation of frobeniusdist_squared in linearop.py, effect.py, and state.py.

add useful error message

1b818c2

caching, documentation, function signatures, variable names

77ff35c

finish docs for metrics

eb84b99

rileyjmurray commented Aug 8, 2025

View reviewed changes

pygsti/tools/leakage.py Outdated Show resolved Hide resolved

rileyjmurray commented Aug 8, 2025

View reviewed changes

pygsti/tools/leakage.py Outdated Show resolved Hide resolved

rileyjmurray added 5 commits August 18, 2025 17:21

docs for leaky_qubit_model_from_pspec

d951dff

doc and type annotation

c668f46

mostly just docs

b8da831

move some functions from leakage.py to explicitmodel.py and gst.py. F…

4554ad8

…ix bugs introduced in the last couple of commits.

resolve typos identified by Nick

43a2d6a

coreyostrove approved these changes Aug 19, 2025

View reviewed changes

Leakage-aware GST #410

Are you sure you want to change the base?

Leakage-aware GST #410

Conversation

rileyjmurray commented Mar 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rileyjmurray commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rileyjmurray left a comment

Choose a reason for hiding this comment

Uh oh!

coreyostrove commented Aug 7, 2025

Uh oh!

coreyostrove left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rileyjmurray commented Aug 19, 2025

Uh oh!

coreyostrove commented Aug 19, 2025

Uh oh!

coreyostrove left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rileyjmurray commented Mar 19, 2024 •

edited

Loading

rileyjmurray commented Mar 25, 2025 •

edited

Loading