Support future libxapian 2.0

[kelson42]

Xapian 2.0 is almost over and bring a few improvements of great value for the libzim and Kiwix.

Even if we will have to keep backward compatibility for a while to give time to GNU/Linux distributions to move on smoothly to latest version, it would be great the we can compile against libxapian 2.0 and distribute binaries based on it.

[kelson42]

@ojwb Thank you again for your last PR. During your effort have you already identified a few things we should/could do?

[ojwb]

The intention is that the Xapian API should be compatible except for features marked as deprecated since 1.4.0 or earlier, so supporting both 1.4.x and 2.0.x should be easy. (A rare exception is that we also removed LMWeight as the formulae were incorrectly implemented so it's not actually useful to use in 1.4.x; 2.0.x replaces it with 4 separate language-model weighting scheme classes, but it looks like you only use BM25Weight so not relevant here.)

The two changes you've now merged seem to be enough to get libzim working with Xapian 2.0.x. There are some test failures, but they seem to all be due to missing .zim files which I'm guessing are large testdata that isn't in the git repo.

There are a number of new stemmers so if you have any knowledge of those encoded that should be updated (a quick grep didn't find any though).

The glass database format is compatible across 1.4.x and 2.0.x, but some of the stemmers have updates which means some words can stem differently. In most cases only a small number of words are affected and building a database with 1.4.x and searching with 2.0.x (or vice versa) will work well (but using matched versions will work a bit better). An exception though is that the Dutch stemmer has switched to a completely different algorithm. In Dutch vowels may double in the singular form (e.g. manen = moons, maan = moon), and the old algorithm undoubles vowels (so stem is man) while the new one doubles them (so stem is maan). 1.4.x's Dutch stemmer is available in 2.0.x as dutch_porter which could be used for existing Dutch databases, or to postpone a switch. I wouldn't recommend hard wiring this as the Dutch stemmer for everything forever though - the new default Dutch stemmer is enough better it seemed worth the pain of switching.

As for new features you could make use of, these come to mind as perhaps interesting here:

• There's now support for arbitrary glob-like wildcards (single or multiple * and/or ? anywhere in a term rather then just * at the end of a word). Not on by default.
• A new fuzzy matching operator so e.g. foo~ expands to words within edit distance 2, foo~3 specifies the edit distance, or foo~0.2 as a fraction of the term length. Also not on by default.
• New support for using ICU's word-boundary finding for CJK and a few other languages (which is now an optional build-time dependency). Works in TermGenerator, QueryParser and snippet generation.

I also gave an overview of 2.0 at the start of the Xapian BoF session at Debconf earlier this year which covers some of the highlights (video | lower bitrate version).

One thing that's changed since then is we decided to postpone clustering and diversification until a 2.0.x point release (there's a bug in the diversification implementation and it seemed better to release without it than to release with a buggy version or to delay the release; diversification uses clustering behind the scenes and working on that raised some doubts about the current clustering API).

[kelson42]

Thank you for your detailed answer!

The intention is that the Xapian API should be compatible except for features marked as deprecated since 1.4.0 or earlier, so supporting both 1.4.x and 2.0.x should be easy.

This is very great.

The glass database format is compatible across 1.4.x and 2.0.x, but some of the stemmers have updates which means some words can stem differently. In most cases only a small number of words are affected and building a database with 1.4.x and searching with 2.0.x (or vice versa) will work well (but using matched versions will work a bit better).

Very important to us as user have to deal with "older" ZIM files embedding "older" Xapian indexes.

An exception though is that the Dutch stemmer has switched to a completely different algorithm. In Dutch vowels may double in the singular form (e.g. manen = moons, maan = moon), and the old algorithm undoubles vowels (so stem is man) while the new one doubles them (so stem is maan). 1.4.x's Dutch stemmer is available in 2.0.x as dutch_porter which could be used for existing Dutch databases, or to postpone a switch. I wouldn't recommend hard wiring this as the Dutch stemmer for everything forever though - the new default Dutch stemmer is enough better it seemed worth the pain of switching.

I guess here we will have to deal with a regression for people using "older" ZIM file and "newer" reader!

* There's now support for arbitrary glob-like wildcards (single or multiple `*` and/or `?` anywhere in a term rather then just `*` at the end of a word).  Not on by default.

* A new fuzzy matching operator so e.g. `foo~` expands to words within edit distance 2, `foo~3` specifies the edit distance, or `foo~0.2` as a fraction of the term length.  Also not on by default.

This is something we will need to have a look, as time to time users ask for this.

* New support for using ICU's word-boundary finding for CJK and a few other languages (which is now an optional build-time dependency).  Works in TermGenerator, QueryParser and snippet generation.

This would be very valuable for us if the tokenizer would be more efficient for CJK languages. FYI, Chinese is one of the top user language.

I also gave an overview of 2.0 at the start of the Xapian BoF session at Debconf earlier this year which covers some of the highlights (video | lower bitrate version).

Great, I watched it and better HTML parsing and Honey backend are definitely things of interest for us. BTW, one of the hard topic we always have is compilation on Windows with Microsoft compiler. Do we see any improvement on this front? Using Meson maybe?

One thing that's changed since then is we decided to postpone clustering and diversification until a 2.0.x point release (there's a bug in the diversification implementation and it seemed better to release without it than to release with a buggy version or to delay the release; diversification uses clustering behind the scenes and working on that raised some doubts about the current clustering API).

Diversification is a fairly new topic to me and - at this stage - I'm not sure exactly how this is going to work (even if the problem it solves is quite clear).

[kelson42]

@veloman-yunkan If this issue has been fully implemented by @ojwb, then I propose to close this issue and we are ready to use it when it will be released. It is important we move relatively early to Xapian 2.0 for the new/future cutting-edge Honey storage backend. Reason is that we need as much as possible Kiwix readers with libxapian 2.0 to be able then to move to a new storage backend (otherwise too many users will complain). Therefore I have created kiwix/kiwix-build#892 so we implement its support in kiwix-build.

[ojwb]

BTW, one of the hard topic we always have is compilation on Windows with Microsoft compiler. Do we see any improvement on this front? Using Meson maybe?

I think we should look at moving to Meson, but not for 2.0.x (because this is not the point at which to be adding significant new things to a release which has already taken far too long).

You should already be able to build with MSVC using the autotools build system though. INSTALL documents the dependencies and how to configure. It should work for Xapian 1.4.6 and later, and there are CI jobs which test it with various MSVC versions so it should actually still work.

The CI jobs don't currently test omega or the bindings though, mostly because nobody's cared enough to sort out how to install the dependencies: omega needs libmagic; the bindings need the target language development packages. I'd expect they should work with similar configure commands, though may need some minor code tweaks as MSVC does have some quirks.

Diversification is a fairly new topic to me and - at this stage - I'm not sure exactly how this is going to work (even if the problem it solves is quite clear).

Briefly, you would request a larger result set in the usual way (say 100 results instead of the 10 you want to show) and then ask for it to be diversified, which will identify which results are similar to others and demote all but the best of each group of similar results. Paging through the results should still show the same set overall, but in a different (and hopefully more useful) order.

The exact API still needs some work, as we want to avoid people having to implement extra machinery in their own code just to allow cleanly paging through diversified results. Ideally this sort of thing should work the same as for non-diversified.

It is important we move relatively early to Xapian 2.0 for the new/future cutting-edge Honey storage backend. Reason is that we need as much as possible Kiwix readers with libxapian 2.0 to be able then to move to a new storage backend (otherwise too many users will complain).

Note that currently Honey doesn't support update (you build a read-only Honey DB by converting a Glass DB via compaction with Xapian::DB_BACKEND_HONEY in the flags), so the patch in #1012 to get multiple spelling corrections won't work because it requires opening the database for update.

Honey is currently a kind of technology preview as much as anything. It is being used by endlessos so we'll need to continue to support current Honey databases in the future, but it's going to need to evolve to allow update. The index structure also needs to get more sophisticated to work well with very large databases.

If current Honey works well for libzim's use it's certainly reasonable to adopt it, but you definitely shouldn't think you have to move to it at this point. The glass backend is going to need to be supported in at least the release series after 2.0.x.

[kelson42]

@ojwb Thx for the clarifications, in particular regarding honey. Not sure what we will about though. Closing this issue as libzim is ready to deal with Xapian 2.0. I have release yesterday milestone 9.4.1 so from now are officially compatible.