Mismatched model->cluster mapping in `clustrmsd`: only when called multiple times in a single run

[AnnaKravchenko]

Module/Workflow/Library affected

clustrmds, but ONLY when called 2nd or more times

Expected behavior

When mapping models to respective clusters, both clustrmsd.txt/rmsd_matrix.json and cluster.out should carry the same mapping.

Actual behavior

Look at sinlge model in cluster # 5.
In 4_clustrmsd/cluster.out:

Cluster 1 -> 1 2 5 6 7 8 9 11 14 16 18
Cluster 2 -> 13 15 19
Cluster 3 -> 10 17 20
Cluster 4 -> 3 12
Cluster 5 -> 4

But 4_clustrmsd/clustrmsd.txt and rmsd_matrix.json display different model (full file at the bottom):

…
-----------------------------------------------
Cluster 3 (#5, n=1, top1_avg_score = -677.47)
clt_rank	model_name	score
1	mdref_1.pdb	-677.47	*
-----------------------------------------------
...

Steps to reproduce the behavior

Workflow and input pdb are here

Version

657917f

Additional context

In the FIRST occurrence of clustrmsd, this mapping is CORRECT. But in the 2nd and subsequent occurrences of clustrmsd, it is broken.

Raw 4_clustrmsd/clustrmsd.txt. Clusters # 5, # 4 and # 3 are fully mismatched.

### clustrmsd output ###

Clustering parameters
> linkage_type=average
> criterion=maxclust
> n_clusters=5
> min_population=1

-----------------------------------------------

Total # of clusters: 5

-----------------------------------------------
Cluster 1 (#1, n=11, top1_avg_score = -702.24)

clt_rank	model_name	score
1	mdref_18.pdb	-702.24
2	mdref_2.pdb	-692.84
3	mdref_16.pdb	-673.15
4	mdref_13.pdb	-672.02
5	mdref_14.pdb	-669.53
6	mdref_3.pdb	-663.94	*
7	mdref_17.pdb	-661.30
8	mdref_5.pdb	-657.26
9	mdref_7.pdb	-656.27
10	mdref_11.pdb	-653.63
11	mdref_8.pdb	-650.35

-----------------------------------------------
Cluster 2 (#4, n=2, top1_avg_score = -682.43)

clt_rank	model_name	score
1	mdref_9.pdb	-682.43	*
2	mdref_20.pdb	-677.80

-----------------------------------------------
Cluster 3 (#5, n=1, top1_avg_score = -677.47)

clt_rank	model_name	score
1	mdref_1.pdb	-677.47	*

-----------------------------------------------
Cluster 4 (#3, n=3, top1_avg_score = -672.13)

clt_rank	model_name	score
1	mdref_19.pdb	-672.13
2	mdref_4.pdb	-671.03	*
3	mdref_6.pdb	-652.01

-----------------------------------------------
Cluster 5 (#2, n=3, top1_avg_score = -671.46)

clt_rank	model_name	score
1	mdref_10.pdb	-671.46
2	mdref_12.pdb	-661.66	*
3	mdref_15.pdb	-654.04
———————————————————————

[amjjbonvin]

The main question is: Why would you do two rmsdclust steps after each other??? And in the second step you are using maxclust instead of distance, meaning you are asking it to return 5 clusters, which is does. You are using two different clustering methods for an ensemble of models which are all very close to each other using only the backbone for the RMSD matrix. I don’t think there is a real problem here.

[AnnaKravchenko]

Why would you do two rmsdclust steps after each other???

To check several different distance cutoffs - not sure why this is so surprising.

[amjjbonvin]

If you want to test different cutoff, use then the same clustering method. Now you are using different methods (distance and maxclust), on a set of very similar models. Not really surprised about the differences.

[AnnaKravchenko]

Same mismatched mapping occurs when running two “distance” with different “clust_cutoff” - just harder to spot since differemt number of clusters are being returned

[AnnaKravchenko]

This happends because 1st occurence of clustrmsd reorders models based on cluster ranks, so more than 1 clustrmsd simply should not be used in a single run - case closed

[amjjbonvin]

Indeed - an unintended use. We can have more clustrmsd steps, but not sequential and each following a rmsdmatrix step

…

This happends because 1st occurence of clustrmsd reorders models based on cluster ranks, so more than 1 clustrmsd simply should not be used in a single run - case closed

[rvhonorato]

The main question is: Why would you do two rmsdclust steps after each other???

I cannot understand how this would be a valid question or why this is surprising. The whole idea behind this project is to allow users to compose the workflows to whatever they think could be valuable. By asking this in a bug report you are simply dismissing a potential real bug based on what "should or should not be done". Please read the description and see if you can replicate the issue and do not just dismiss it. If it's something relevant enough to be made into an issue its likely something relevant to be addressed in the code, either as a code change or a simple comment/documentation.

The documentation of this module explicitly says:

This module passes the path to the RMSD matrix is to the next step of the workflow through the rmsd_matrix.json file, thus allowing to execute several clustrmsd modules (possibly with different parameters) on the same RMSD matrix.

So running multiple clustrmsd modules in the same workflow was the intended design, looking at the code clustrmsd will change the order of the models;

haddock3/src/haddock/modules/analysis/clustrmsd/__init__.py

Line 139 in ea89942

self.output_models = add_cluster_info(sorted_score_dic, clt_dic)

This behavior of changing the order will introduce a non-documented side-effect which can lead to wrong interpretation of results - from my point of view this module should not have any modification power and should simply produce the output files, or at least this should be properly documented in the module. What do you think @AnnaKravchenko?

[amjjbonvin]

This is simply another case of incompatible steps - something we have an issue open for (not this exact scenario), but currently no check. #1530 In this case, may-be the rmsdclust module should always look for an io file from a previous rmsdmatrix step. And users might try whatever they think is valuable to them indeed, with without warranty from our side it will work, which is why we are providing examples. With the number of modules we have and the flexibility in the number of steps, the possible number of combinations is a big factorial!

[rvhonorato]

you are talking about something else; this issue here is about rmsdclust in which the documentation says it can be run multiple times, however because it changes the order of the modules it cannot.

so here we either change the behavior or update the documentation to reflect the current state.

[AnnaKravchenko]

Yes, exactly! This mismatch shows up on the plot (rmdsmatrix.html) and it is easy to misinterpret results when using several clustrmsd modules with different parameters.

But even more - in case the 1st clustrmsd removes any models (specifically, unclustered models are dropped when min_population > 1), then the 2nd clustrmsd reads a new list of reordered models and original RMSD values from the initial rmdsmatrix. Meaning that running several clustrmsds will always either return IndexError or mapping mismatch - so practically speaking, it is impossible to successfully execute several clustrmsd modules on the same RMSD matrix in the uninterrupted workflow.

Either the docs need to say that "execute several clustrmsd modules (possibly with different parameters) on the same RMSD matrix” means using --restarting, and this is the ONLY way; or each clustrmsd has to look into the last rmsdmatrix for both rmds values and also a list of models.

[rvhonorato]

clear, both clustfcc and rmsdclust have this behavior of changing the data in-place; making this change of recovering the "original" modelset would need to be done in both. this change could be quite disruptive as it may change the output of workflows that already exist and expect the current behavior.

so here I'd suggest we just change the documentation and also add a warning to the module to make it clear the modules have been re-ordered - @AnnaKravchenko do you agree? should I take care of it or can you do it?