[RFC] Jar dependencies consolidation

[andsel]

Abstract

Logstash is shaped as a Gradle multi-project, which contains multiple modules where the same dependency is defined across those different modules. So we end up declaring same dependency but with different versions in any project. For example consider assertj:

1. Logstash-core

   logstash/logstash-core/build.gradle

   Line 258 in 8c58270

   testImplementation 'org.assertj:assertj-core:3.11.1'

2. qa/integration

   logstash/qa/integration/build.gradle

   Line 34 in 8c58270

   testImplementation 'org.assertj:assertj-core:3.8.0'

3. x-pack

   logstash/x-pack/build.gradle

   Line 27 in 8c58270

   testImplementation 'org.assertj:assertj-core:3.8.0'

The Gradle’s way (since 7.4) is to define a Gradle Version Catalog in te main project, but Logstash already tracks jackson and jrjackson dependencies in versions.yml, which is a custom catalog that also define Logstash’s version, bundled JDK version and hash code of the JRuby artifact.

Goal

This document describes the two alternatives to make the team of it and discuss a possible solution to centralize dependencies versions.

### Existing versions.yml
This mechanism is already in place for jrjackson, and require to use Gradle ext properties and define a variable with the version in every project it’s used, for example:

def versionMap = gradle.ext.versions
String jacksonVersion = versionMap['jackson']


dependencies {
    implementation "com.fasterxml.jackson.core:jackson-core:${jacksonVersion}"
}

Gradle's Version Catalog

Create a TOML file gradle/libs.versions.toml that defines:

[versions]
assertj = "3.26.3"
jackson = "2.16.2"
log4j = "2.23.1"

[libraries]
# Testing
assertj-core = { module = "org.assertj:assertj-core", version.ref = "assertj" }

# Jackson family (sharing same version)
jackson-core = { module = "com.fasterxml.jackson.core:jackson-core", version.ref = "jackson" }
jackson-databind = { module = "com.fasterxml.jackson.core:jackson-databind", version.ref = "jackson" }
jackson-annotations = { module = "com.fasterxml.jackson.core:jackson-annotations", version.ref = "jackson" }

[bundles]
jackson = ["jackson-core", "jackson-databind", "jackson-annotations"]

o use the catalog definition Gradle automatically creates accesses to the the bundles and libraries for example, logstash-core/build.gradle could be result in:

dependencies {    
    implementation libs.bundles.jackson // this kicks in all the jackson libs
    testImplementation libs.assertj.core
}

While x-pack/build.gradle becomes:

dependencies {    
    testImplementation libs.assertj.core  // Same version as logstash-core!
}

Analysis

Using the already existing harness from versions.yml:

• ➕ is already in place and proven.
• ➖ no IDE support
• ➖ no standard way of doing things, so newcomers has to know it, instead of relay on standard mechanisms already well known.
• ➕ central point to handle all versioning related question for Logstash, but maybe increase too much it’s scope

Using the Gradle ‘s version catalog:

• ➕ standard ways on doing things
• ✔️ Requires Gradle 7.4+, but shouldn’t be a problem for us because we should have everything ported to Gradle 8
• ➕ well known mechanism
• ➖ could create confusion with versions.yml
• ➕ permit an incremental migration, having jar dependencies in this catalog and the rest in versions.yml.

[donoghuc]

For all the gradle-only shared components (jackson, jackson-databind, assertj-core, junit, junit-jupiter, log4j, guava, commons-io, commons-codec, snakeyaml, hamcrest, hamcrest-library, awaitility, jmh, httpclient, commons-compress, commons-lang3, commons-csv, javassist, janino, zstd-jni, google-java-format, securemock, json-unit, jopt-simple, elasticsearch-rest-client, wiremock) moving them to toml and sharing them across sub-projects is a huge improvment (seems like there is already evidence of drift). Anything that ruby or CI tooling needs from versions.yml will stay there, but I dont see a huge overlap. I think this is a good improvement and should be very low risk.