Using manually "clustered" faces from memories to improve future clustering

[phirestalker]

I'm not sure if I used the right terms. I installed Memories in Nextcloud (Awesome app by the way, check it out). If I go to people under memories, I can merge people. If I go to a person, and it has the face of someone else, I can remove it or move it to another person. I can create new groupings for a new person and name them.
I would like to know if recognize will use these corrections to better its detection and/or clustering going forward. If not, could that be implemented somehow?

[marcelklehr]

I would like to know if recognize will use these corrections to better its detection and/or clustering going forward.

The degree of improvement that recognize learns from manual assignments is rather slim, sadly. Mainly it might learn that a specific person has a diverse set of faces associated with it and can try to use that somehow in the clustering. But the original face embeddings are not trained with your assignments, so e.g. weak spots of the embedding model like children will always be a problem, no matter how much you try by assigning faces to the right people.

[phirestalker]

So, do you mean it does not add to or improve the actual model, or that it just doesn't do much to improve it even though it adds to the model?
The main thing that would help, in my opinion, is to use the times I remove a face from a group. If it could mark that as a "negative example" in the model, would that improve accuracy more? I would be willing to read if you have the documentation of the system used. I am interested in the technology.

[marcelklehr]

So, there's two modules in this system. The first module detects faces in pictures and generates an embedding vector for the detect face. The second module then runs a clustering algorithm over these embedding vectors.

The first module is fixed. We don't retrain it and it doesn't learn from the face assignments you make in the UI.
The second module implicitly does learn from the assignments. When you assign a new face to an existing cluster of faces (called a Person in the UI) the clustering module will then know that faces similar to this face might belong to this new cluster as well. The same goes for removing a face from a cluster. The module then learns that faces similar to this one are unlikelier to belong to this face.

So, the system learns by virtue of having access to the cluster data that you produce manually in tandem with the system. But if you reset everything and let it run from scratch it will make the same mistakes again, because the data to help it is gone.

[phirestalker]

Great. That answers my questions then. Thanks for the explanation.