Skip to content

refactor: compile against the forked OpenSearch package and drop the opensearch jar - #3439

Open
marevol wants to merge 2 commits into
mainfrom
fork/opensearch-migrate
Open

refactor: compile against the forked OpenSearch package and drop the opensearch jar#3439
marevol wants to merge 2 commits into
mainfrom
fork/opensearch-migrate

Conversation

@marevol

@marevol marevol commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

fesen-httpclient now carries the OpenSearch DTOs and query builders Fess needs under org.codelibs.fesen.opensearch.*, pruned from 8,578 classes to 3,751. This drops the org.opensearch:opensearch dependency and moves Fess onto them.

WEB-INF/lib goes from 229 jars / 146,843,781 bytes to 215 jars / 135,178,555 bytes — 11.67 MiB off every distribution. The 17.09 MiB server jar and its twelve small siblings leave, along with reactive-streams; fesen-httpclient grows from 0.44 MiB to 7.09 MiB. No org/opensearch/ class remains anywhere in the distribution — all 219 jars were swept.

All 13 Lucene modules stay at 10.5.0. The old exclusion block had been suppressing eight of them; the fork declares them directly, so they come back.

The change

264 files, 2,049 import lines. 142 of the 210 main files are checked-in ESFlute output, so 81% of the occurrences are generated code. src/main/webapp has none. Fess's own org.codelibs.fess.opensearch.* package is untouched — the rewrite was anchored so it could not match it.

Three things were not a rename:

  • QueryShardContext is gone from the fork — it was the bridge whose removal collapsed index/mapper/**, index/fielddata/** and search/internal/**. Four classes overrode a method taking it: KnnQueryBuilder, HybridQueryBuilder, StoredLtrQueryBuilder and DefaultQueryBuilder. All four bodies threw UnsupportedOperationException or delegated to one that did — these builders exist to be serialised to JSON, which still works — so the overrides are deleted. KnnQueryBuilderTest.test_doToQuery_isUnsupported called the removed method and could not compile; it is the only test removed.
  • The seeded Index Exporter job embeds org.opensearch.index.query.QueryBuilders in its script string. The seed is updated. See the upgrade note below.
  • Three plugin-facing core signatures move: IndexingHelper.deleteDocumentByQuery(SearchEngineClient, QueryBuilder), its four protected siblings, and LanguageHelper.createScript(Map, String) returning Script. Five plugin repositories follow them and ship together.

Two side effects of dropping the dependency, both fixed here:

  • SnakeYAML silently dropped from 2.6 to 2.5. ThemeManifest parses theme.yaml with it directly but never declared it — it was arriving through the opensearch jar, and without it jackson-dataformat-yaml decides the version. Now declared at the release it has been running.
  • The log4j2 rule silencing org.opensearch.bootstrap no longer names anything: the forked tree has no bootstrap package and the original jar is gone. Removed.

error_prone_annotations moves 2.45.0 → 2.36.0. Annotations only, unused at runtime, left alone.

Upgrade note for the persisted scheduled job

fess_config.scheduled_job/_id=index_exporter is reloaded at every webapp boot, but with OpType.CREATE, so an existing document conflicts and is skipped. The corrected seed therefore does not reach an upgraded installation, and Fess has no migration framework to hang a fix on.

The exposure is small and loud: the job ships available:false with an empty cron expression, so it never fires on its own. An administrator who had enabled it will see it fail with a ScriptEngineException in the job log rather than fail silently or corrupt anything. The fix is a one-line edit in Admin > Scheduler, replacing the package prefix. This is release-note material; rewriting the document at boot would silently overwrite a script an administrator may have customised.

Verification

  • mvn clean package and the full suite — 7,469 tests, 0 failures.
  • End to end against a real OpenSearch 3.8.0, from a virgin cluster: 40 indices created green, a crawl run through the crawler child process (which has its own classpath), search returning documents with highlighting, term and filter facet counts that match the indexed data, suggest indexing 27 documents and switching its alias, and _nodes/stats parsed with all 30 sections populated. Zero NoClassDefFoundError, ClassNotFoundException, NoSuchMethodError or ServiceConfigurationError across all 13 log files, and zero ERROR-level lines.

The facet result is the one that matters most: search/aggregations was cut from 1,042 classes to 334, and aggregation response parsers are only exercised at runtime.

Ordering

Requires codelibs/fesen-httpclient#43, codelibs/fess-suggest#98 and codelibs/fess-crawler#208 to be released first. The five plugin repositories that follow the core signatures above must be released with this, not after it.

…client

Fess shipped org.opensearch:opensearch (17.9 MB) purely for its request and
response types and its query builders; it never runs a node. Those classes are
now forked and pruned into org.codelibs.fesen.opensearch.* inside
fesen-httpclient, so the dependency and its exclusion block go away and every
org.opensearch import moves to the forked package. Apache Lucene stays as the
real org.apache.lucene and now arrives through fesen-httpclient, which keeps all
thirteen modules Fess resolved before. Fess's own org.codelibs.fess.opensearch.*
(DBFlute/ESFlute output) is unrelated and untouched.

Three things are not a plain rename:

- QueryShardContext is deliberately absent from the fork, so QueryBuilder no
  longer declares toQuery and AbstractQueryBuilder no longer declares doToQuery.
  The overrides in KnnQueryBuilder, HybridQueryBuilder, StoredLtrQueryBuilder and
  DefaultQueryBuilder only threw UnsupportedOperationException or delegated, so
  they are removed along with the now-unused imports. These builders exist to be
  serialised to JSON, which is unaffected. KnnQueryBuilderTest's
  test_doToQuery_isUnsupported covered a method that no longer exists and goes
  with it.

- The seeded Index Exporter job embeds a class name in its script body, so
  scheduled_job.bulk moves to org.codelibs.fesen.opensearch.index.query.
  QueryBuilders too. The startup reload is create-only, so an existing
  installation keeps its stale copy and needs a separate migration.

- IndexingHelper.deleteDocumentByQuery and LanguageHelper.createScript are part
  of the plugin-facing API and now take and return forked types; the plugins that
  call them migrate separately.

Removing the dependency drops opensearch-3.8.0.jar and its twelve siblings plus
reactive-streams, 17.84 MiB off the resolved runtime classpath (229 jars to 215).
Removing org.opensearch:opensearch took SnakeYAML off the resolved classpath at
2.6 and left jackson-dataformat-yaml to supply 2.5 instead. ThemeManifest parses
theme.yaml with SnakeYAML directly, so the version it gets should not be decided
by an unrelated Jackson module: declare it, at the release it has been running.

The log4j2 entry that silenced org.opensearch.bootstrap no longer names anything.
The forked tree has no bootstrap package - the node runtime was cut - and the
original jar is gone, so the logger cannot match. Remove it rather than leave a
rule pointing at a package that does not exist.

7469 tests pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant