[FEATURE] Manticore Automatic Restore and Synchronization [DB]

[fredbcode]

I am testing piler-ee (Docker) and, unless I have misunderstood, I have not found a way to automatically restore the Manticore MySQL database in case of a crash. This is problematic in the context of a large cluster, as the failed node becomes desynchronized from the others.

I have created a script that extracts entries from the Piler database and injects them into the Manticore database, but this is not trivial for operators and is not necessarily consistent with the other clustered components (MinIO, Piler Galera). Thank you.

[jsuto]

Not sure what you mean by "extracts entries from the Piler database and injects them into the Manticore database"
Please elaborate.

[fredbcode]

After digging deeper into the architecture, I may have misunderstood part of the design, so please correct me if I'm wrong.

From what I understand, MariaDB (galera) is the source of truth while Manticore (mysql) stores a derived search index. In a multi-node deployment, MariaDB (Galera) and MinIO remain synchronized, but Manticore appears to be local to each node (at least after a crash)

Following a few failure and recovery tests, I noticed that a node can come back with an incomplete Manticore index while the metadata and archived content remain fully available. To better understand and mitigate this, I wrote a synchronization script that rebuilds or repairs the Manticore RT index directly from the MariaDB metadata table by reinjecting missing entries.

At the moment, my goal is mainly to optimize and automate this process rather than perform full rebuilds every time. On my test environment (~27k messages), a complete reindex takes around 20–30 seconds, which is perfectly acceptable, but I am trying to understand what the recommended approach would be for much larger archives.

Perhaps I am missing something, but is there already a built-in mechanism or recommended procedure to keep Manticore synchronized after a node outage?

More specifically:

• Is rebuilding or repairing the index from MariaDB the expected recovery method?
• Is there a way to identify and reindex only missing documents rather than rebuilding everything?
• Are Manticore indexes expected to be completely independent on each node?
• How is this usually handled in large Piler EE deployments with millions of archived messages?

For now I am experimenting with incremental synchronization based on message IDs and range comparisons, but I would be very interested to know whether there is an existing or preferred solution that I may have overlooked.

I'm using 2.1.3 ee images

[jsuto]

Assuming that you use the default rt index settings, then when piler archives a message, it not only updates the mysql database, stores the files, but updates the manticore table (=piler) as well. So after your recovery tests you find missing entries from manticore (that have been there before), then you have a durability issue. Anyway, the reindex utility can repopulate the manticore index data.

[fredbcode]

I have three identical machines running in a cluster.

What I am trying to describe is the following situation:

If I take one node down (for example due to a failure) and then bring it back online later, everything looks fine from a Piler and MinIO perspective. MariaDB (Galera) is consistent and MinIO also has all the data.

However, the Manticore index on the node that was down appears to have lost or missed data during that downtime window.

I was not able to fully restore or resynchronize that index using the built-in tools (such as the reindex utility), so I probably missed something in the process or in the expected workflow.

My understanding so far is that Manticore is a local RT index on each node, but I am not sure what the correct or recommended procedure is to fully recover it after a node has been offline.

Is there an expected way to rebuild or resync only the missing part of the index from MariaDB after such an event, or is a full reindex always required in that case?

[jsuto]

So you have a single piler node that uses a galera cluster and a manticore cluster, and stores the files on minio. Well, if this is the case, I'd check out the manticore docs how to do clustering and recovery. However, there's a simpler approach as well: have only a single manticore node, make regular backups, snapshots, etc. So your RPO is quite small, and reindex can re-populate what's missing.