piler database broken since Upgrade to Ubuntu 26 LTS

[vmario89]

Since Upgrade to piler 1.4.8 (master branch commit (b6aab54)) and Upgrade from Ubuntu 22 LTS to 26 LTS by 2 steps, piler is not working anymore. At the least the web frontend is rendering, but it shows no data. It gives error.

i invested around a day to find out to make it work again but i have no ideas anymore. I get the overview of mails, but when clicking on any mail, attachments/data is empty. Clicking "show headers", it reveals:

the manticore log:
tail -f /var/piler/manticore/query.log

/* Sat May 16 01:06:29.130 2026 conn 1 (127.0.0.1:41844) real 0.000 wall 0.000 found 0 */ SELECT tag FROM tag1 WHERE uid=1 AND mid=7445;
/* Sat May 16 01:06:29.133 2026 conn 1 (127.0.0.1:41844) real 0.000 wall 0.000 found 0 */ SELECT note FROM note1 WHERE uid=1 AND mid=7445;
/* Sat May 16 01:06:29.656 2026 conn 5 (127.0.0.1:41860) real 0.001 wall 0.002 found 5589 */ SELECT id FROM piler1 ORDER BY sent DESC LIMIT 50;
/* Sat May 16 01:06:29.661 2026 conn 5 (127.0.0.1:41860) real 0.000 wall 0.000 found 0 */ SELECT mid, tag FROM tag1 WHERE uid=1 AND mid IN (7400,7401,7402,7403,7404,7405,7406,7407,7408,7409,7410,7411,7412,7413,7414,7415,7416,7417,7418,7419,7420,7421,7422,7423,7424,7425,7426,7427,7428,7429,7430,7431,7432,7433,7434,7435,7436,7437,7438,7439,7440,7441,7442,7443,7444,7445,7446,7447,7448,7449);
/* Sat May 16 01:06:29.678 2026 conn 5 (127.0.0.1:41860) real 0.000 wall 0.000 found 0 */ SELECT mid, note FROM note1 WHERE uid=1 AND mid IN (7400,7401,7402,7403,7404,7405,7406,7407,7408,7409,7410,7411,7412,7413,7414,7415,7416,7417,7418,7419,7420,7421,7422,7423,7424,7425,7426,7427,7428,7429,7430,7431,7432,7433,7434,7435,7436,7437,7438,7439,7440,7441,7442,7443,7444,7445,7446,7447,7448,7449);
/* Sat May 16 01:06:29.721 2026 conn 1 (127.0.0.1:41844) real 0.001 wall 0.001 found 5589 */ SELECT id FROM piler1 ORDER BY sent DESC LIMIT 50;
/* Sat May 16 01:06:29.725 2026 conn 1 (127.0.0.1:41844) real 0.000 wall 0.000 found 0 */ SELECT mid, tag FROM tag1 WHERE uid=1 AND mid IN (7400,7401,7402,7403,7404,7405,7406,7407,7408,7409,7410,7411,7412,7413,7414,7415,7416,7417,7418,7419,7420,7421,7422,7423,7424,7425,7426,7427,7428,7429,7430,7431,7432,7433,7434,7435,7436,7437,7438,7439,7440,7441,7442,7443,7444,7445,7446,7447,7448,7449);
/* Sat May 16 01:06:29.734 2026 conn 1 (127.0.0.1:41844) real 0.000 wall 0.000 found 0 */ SELECT mid, note FROM note1 WHERE uid=1 AND mid IN (7400,7401,7402,7403,7404,7405,7406,7407,7408,7409,7410,7411,7412,7413,7414,7415,7416,7417,7418,7419,7420,7421,7422,7423,7424,7425,7426,7427,7428,7429,7430,7431,7432,7433,7434,7435,7436,7437,7438,7439,7440,7441,7442,7443,7444,7445,7446,7447,7448,7449);
/* Sat May 16 01:06:32.374 2026 conn 3 (127.0.0.1:41848) real 0.000 wall 0.000 found 0 */ SELECT tag FROM tag1 WHERE uid=1 AND mid=7445;
/* Sat May 16 01:06:32.375 2026 conn 3 (127.0.0.1:41848) real 0.000 wall 0.000 found 0 */ SELECT note FROM note1 WHERE uid=1 AND mid=7445;
/* Sat May 16 01:08:33.172 2026 conn 1 (127.0.0.1:41844) real 0.000 wall 0.000 found 0 */ SELECT tag FROM tag1 WHERE uid=1 AND mid=7449;
/* Sat May 16 01:08:33.178 2026 conn 1 (127.0.0.1:41844) real 0.000 wall 0.000 found 0 */ SELECT note FROM note1 WHERE uid=1 AND mid=7449;

Ubuntu automatically installed Manticore v13. I removed this and try to revert to Manticore 9x versions and 10x versions, but always the same.

rm /etc/apt/sources.list.d/manticore*

sudo systemctl stop pilersearch.service

apt purge manticore* -y

wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore_10.1.0-25060608-c63706038_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-tools_10.1.0-25060608-c63706038_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-server_10.1.0-25060608-c63706038_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-dev_10.1.0-25060608-c63706038_all.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-server-core_10.1.0-25060608-c63706038_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-tzdata_1.0.0-240522-a8aa66e_all.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-common_10.1.0-25060608-c63706038_all.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-icudata-65l.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-buddy_3.29.0%2B25060414-beb1b320_all.deb

dpkg -i manticore*
rm *.deb

sudo systemctl start pilersearch.service

the config:

archive_address=REDACTED@REDACTED
archive_emails_not_having_message_id=0
archive_only_mydomains=0
backlog=20
check_for_client_timeout_interval=20
cipher_list=ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
default_retention_days=3650
enable_chunking=1
enable_cjk=0
#folders deaktivert, weil verbuggt
enable_folders=0
encrypt_messages=1
extra_to_field=X-Envelope-To:
extract_attachments=1
helper_timeout=20
hostid=piler.REDACTED.de
listen_addr=0.0.0.0
listen_port=25
max_connections=64
max_header_size=131072
max_message_size=50000000
max_requests_per_child=10000
max_smtp_memory=500000000
memcached_servers=127.0.0.1
memcached_to_db_interval=900
memcached_ttl=86400
min_message_size=100
min_word_len=1
mmap_dedup_test=0
mysql_connect_timeout=2
mysqlcharset=utf8mb4
mysqldb=piler
mysqlpwd=REDACTED
mysqlsocket=/var/run/mysqld/mysqld.sock
mysqluser=piler
number_of_worker_processes=3
pemfile=/usr/local/etc/piler/piler.pem
pidfile=/run/piler/piler.pid
piler_header_field=X-piler-id:
process_rcpt_to_addresses=0
queuedir=/var/piler/store
recipient_delimiter=+
rtindex=1
security_header=
server_id=0
smtp_access_list=1
smtp_timeout=60
spam_header_line=
#sphxdb=piler1
sphxdb=""
sphxhost=127.0.0.1
sphxport=9306
syslog_recipients=0
tls_enable=1
tls_min_version=TLSv1.3
tweak_sent_time_offset=0
update_counters_to_memcached=0
username=piler
#verifyssl=1
verbosity=5
workdir=/var/piler/tmp

config is readable, pem file exists

pilersearch.service output:

Mai 16 01:06:13 REDACTED systemd[1]: Starting pilersearch.service - Searchd for piler daemon...
Mai 16 01:06:13 REDACTED searchd[61340]: [Sat May 16 01:06:13.658 2026] [61340] using config file '/usr/local/etc/piler/manticore.conf' (4763 chars)...
Mai 16 01:06:13 REDACTED searchd[61343]: Manticore 10.1.0 c63706038@25060608
Mai 16 01:06:13 REDACTED searchd[61343]: Copyright (c) 2001-2016, Andrew Aksyonoff
Mai 16 01:06:13 REDACTED searchd[61343]: Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Mai 16 01:06:13 REDACTED searchd[61343]: Copyright (c) 2017-2025, Manticore Software LTD (https://manticoresearch.com)
Mai 16 01:06:13 REDACTED searchd[61345]: starting daemon version '10.1.0 c63706038@25060608' ...
Mai 16 01:06:13 REDACTED searchd[61345]: listening on 127.0.0.1:9312 for sphinx and http(s)
Mai 16 01:06:13 REDACTED searchd[61345]: listening on 127.0.0.1:9306 for mysql
Mai 16 01:06:13 REDACTED searchd[61345]: listening on 127.0.0.1:9307 for RO mysql
Mai 16 01:06:13 REDACTED searchd[61345]: Manticore 10.1.0 c63706038@25060608
Mai 16 01:06:13 REDACTED searchd[61345]: Copyright (c) 2001-2016, Andrew Aksyonoff
Mai 16 01:06:13 REDACTED searchd[61345]: Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Mai 16 01:06:13 REDACTED searchd[61345]: Copyright (c) 2017-2025, Manticore Software LTD (https://manticoresearch.com)
Mai 16 01:06:13 REDACTED searchd[61345]: precaching table 'piler1'
Mai 16 01:06:13 REDACTED searchd[61345]: precaching table 'tag1'
Mai 16 01:06:13 REDACTED searchd[61345]: precaching table 'note1'
Mai 16 01:06:13 REDACTED searchd[61345]: precaching table 'audit1'
Mai 16 01:06:14 REDACTED searchd[61340]: Manticore 10.1.0 c63706038@25060608
Mai 16 01:06:14 REDACTED searchd[61340]: Copyright (c) 2001-2016, Andrew Aksyonoff
Mai 16 01:06:14 REDACTED searchd[61340]: Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Mai 16 01:06:14 REDACTED searchd[61340]: Copyright (c) 2017-2025, Manticore Software LTD (https://manticoresearch.com)
Mai 16 01:06:14 REDACTED systemd[1]: pilersearch.service: Supervising process 61345 which is not our child. We'll most likely not notice when it exits.
Mai 16 01:06:14 REDACTED systemd[1]: Started pilersearch.service - Searchd for piler daemon.

any ideas how to solve?

[vmario89]

relates to #290 somehow

[jsuto]

Check if the uid and gid are the same for the piler user.

[vmario89]

Hi,

i checked, everything fine:

ps aux | grep [s]earchd
piler      61344  0.0  0.0  49772  2364 ?        S    Mai16   0:00 /usr/bin/searchd --config /etc/piler/manticore.conf
piler      61345  0.0  0.1 444480  4764 ?        Sl   Mai16   1:00 /usr/bin/searchd --config /etc/piler/manticore.conf

stat -c "%u %g" /proc/6134*
1001 1001
1001 1001

ps aux | grep [p]iler-smtp
piler       2658  0.0  0.2  21800  9576 ?        Ss   13:28   0:00 /usr/sbin/piler-smtp -d

stat -c "%u %g" /proc/2658
1001 1001

ps aux | grep [/]usr/local/sbin/piler
piler       1985  0.0  0.0  15808  3292 ?        Ss   13:20   0:00 /usr/local/sbin/piler -d
piler       1987  0.0  0.1  15808  4180 ?        S    13:20   0:00 /usr/local/sbin/piler -d
piler       1989  0.0  0.1  15808  4108 ?        S    13:20   0:00 /usr/local/sbin/piler -d
piler       1990  0.0  0.1  15808  4180 ?        S    13:20   0:00 /usr/local/sbin/piler -d
stat -c "%u %g" /proc/1985
stat -c "%u %g" /proc/1987
stat -c "%u %g" /proc/1989
stat -c "%u %g" /proc/1990

i had to revert piler execultable to some old version after 1.4.6 and before 1.4.7, because i figured out that piler did not properly start. after starting it immediately fails silently.

strace piler gives:
newfstatat(AT_FDCWD, "/var/piler/run/piler.pid", 0x7ffc0c6241a0, 0) = -1 ENOENT (No such file or directory)

• this behavious does not occure for all commits before 8cf96dc
• with commit a5308a8 it has no issue with pid fails and keeps alive
• i guess there is some problem in configure or configure.in file.

tried to go step back with manticore again, also:

rm /etc/apt/sources.list.d/manticore*
sudo systemctl stop pilersearch.service
apt purge manticore* -y
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore_9.3.2-25050208-428075261_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-tools_9.3.2-25050208-428075261_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-server_9.3.2-25050208-428075261_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-dev_9.3.2-25050208-428075261_all.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-server-core_9.3.2-25050208-428075261_amd64.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-tzdata_1.0.0-240522-a8aa66e_all.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-common_9.3.2-25050208-428075261_all.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-icudata-65l.deb
wget https://repo.manticoresearch.com/repository/manticoresearch_jammy/dists/jammy/main/binary-amd64/manticore-buddy_3.29.0%2B25060414-beb1b320_all.deb
dpkg -i manticore*
rm *.deb
sudo systemctl start pilersearch.service

i have no idea if i destroyed my manticore library, as i dont know it behaves to upgrade manticore from v9 to v13 back to v10 back to v9 ...

the errors in frontend, i get rid by doing

chmod 777 piler.conf
chmod 777 piler.key

which is bad idea. still i am not getting any content to display in web ui. i reverted my server to Ubuntu 22 by snapshot as last resort to keep the archive alive

[jsuto]

Based on the screenshot it's not the manticore database. Run su - piler, then check head /etc/piler/piler.conf

Btw. I suggest you to use manticore's repo, and install only the manticore package.

[github-actions]

This issue is stale because it has been open for 14 days with no activity.