Re-run Sage 2.1.0 #69
Conversation
|
/run-optimization-benchmarks submissions/2025-10-20-Sage-2.1.0/input.yaml |
|
A workflow has been dispatched to run the benchmarks for this PR.
|
|
A workflow dispatched to run optimization benchmarks for this PR has just finished.
|
|
I'm seeing some differences between this run and Both RMSD results on the same plot: Distribution of RMSD differences: Both DDE results on the same plot: Distribution of DDE differences: Here's the code I used to generate these plots and statistics, which I ran from this branch |
|
From a quick look the differences here look probably fine, and I think (caveated with the note in the next sentence) we should merge it so Chapin has a more up-to-date comparison for the protein FF benchmarks. The only note I have is that you'd ideally want to be using the unconstrained version. I'll leave some of my working below since this was a quick-and-dirty skim. A more rigorous check would actually compare the geometries between the two runs, instead of (here) comparing the difference from QM. I visually checked the molecules with the highest RMSDs differences, which are long and floppy. While we expect that flexible molecules can sometimes slide into a different minimum with minor differences in optimization steps, affecting the torsions, I'd expect bonds and angles to remain relatively inflexible. Bond ICRMSD differences range up to 0.005 at the worst. The majority are very low in magnitude. The outlier points around 0.2 remain outlier points. If I had more than 5 min I'd be curious which bond/s are contributing to the molecules with the highest differences in bond ICRMSD between runs (highest bond RMSD shown below) and look at geometries. Same goes for angles, differences range up to 1 degree difference. Again if I had more time I'd wonder what's going on with this seemingly uncomplicated molecule. About 1.2k conformers had the exact same RMSD in both runs, leading me to think they probably minimized to the same structure. Checking the ddEs for these conformers (~750 of which didn't have nan energies) showed some differences ranging -0.4 to 0.4 (kcal/mol?). It's likely this comes from the conformer minimum being different in geometry, although hard to guarantee without checking. 452 conformers had ddE differences < 1e-6 though, and 601 < 1e-3, which seems reasonable. 
|
|
That's a lot for 5 minutes! I've done much less in much more time this morning. Here's a subset of environment differences, none of which should be a smoking gun: {'openeye-toolkits': ('2024.1.3', '2025.1.1'),
'openff-amber-ff-ports': ('0.0.4', '2025.09.0'),
'openff-forcefields': ('2024.09.0', '2025.10.0'),
'openff-interchange': ('0.4.0', '0.4.8'),
'openff-interchange-base': ('0.4.0', '0.4.8'),
'openff-qcsubmit': ('0.53.0', '0.57.0'),
'openff-toolkit': ('0.16.5', '0.17.1'),
'openff-toolkit-base': ('0.16.5', '0.17.1'),
'openff-units': ('0.2.2', '0.3.1'),
'openff-utilities': ('0.1.12', '0.1.16'),
'openmm': ('8.1.2', '8.3.1'),
'rdkit': ('2024.03.5', '2025.03.6')}The automation does a I've pulled out a few molecules which get high ICRMSD differences but it's hard to draw conclusions quickly. I will pick this up later. Otherwise I've
|
|
For now I will follow your recommendation to keep this moving along. There's lots we can do to make these analyses easier; the barrier here was higher than I hoped it to be |
Submission Checklist
submissionsdirectory containing the YDS input file and optionally a force field.offxmlfile/run-optimization-benchmarks path/to/submission/input.yamlor/run-torsion-benchmarks path/to/submission/input.yamlJob status: successto be posted