Generate OSGi infos in the manifest

[laeubi]

🤔 What's changed?

This enables to embed OSGi metadata into the manifest of the produced jar file.

⚡️ What's your motivation?

I use this in an Eclipse plugin and currently need to repack the original cucumber jars into an OSGi bundle, but adding the info at the source has much benefits:

1. Additional metadata in the manifest is ignored by other tools
2. At build time there is the most rich information about used packages and versions so better metadata can be generated
3. Using the original jar ensures it can later be identified as such e.g. with a security scanner
4. No risk that different people repack the same jar under the same name with different settings
5. only do the work once than on each build / consumer

[laeubi]

@mpkorstanje I actually would like to do that for other java based cucumber jars as well but though its good to start with something really simple, can you review and give feedback if it sounds feasible?

[mpkorstanje]

We've had OSGI meta info in the past, but because the project itself doesn't use OSGI and thus not test in an OSGI container, this resulted in a constant stream of broken packages. We simply didn't notice the problems until someone reported them (cucumber/cucumber-jvm#1404, cucumber/common#412).

So either:

• OSGI consumers repackage our jars in a way that suits them.
• We start testing in an OSGI container.

Both keep the feedback loop small. But at present I would prefer the first, because then it keeps the OSGI specific concerns away from an otherwise non-OSGI project.

That said, I'm not opposed to providing OSGI information all together. But I would like a more elegant, non-fragile, testable solution that also doesn't require knowing much, if anything about OSGI.

With Java 8 reaching EOL soon, we can start using the JPMS. Is there a way we can seamlessly translate that too an OSGI definition? Because we will test with the JPMS, that should also cover a programmatically derived OSGI definition.

[mpkorstanje]

This looks incredibly fragile. While for a single package module it is fine. I imagine that if there are multiple packages, this gets messy quickly.

Also, would it be possible to extract this to a file instead?

[laeubi]

This looks incredibly fragile. While for a single package module it is fine.

I must confess I don't understand what part of it?

Also, would it be possible to extract this to a file instead?

Yes one can extract it into a bnd.bnd file for example. I often find that a bit more disturbing for projects not using OSGi/bnd otherwise.

[mpkorstanje]

I must confess I don't understand what part of it?

• Not tested anywhere.
• Export-Package duplicates the automatic module name. That configuration now exists in two places. Same problem once JPMS is used.
• A magic -noimport:=true flag.
• A magic Import-Package: * statement.

[laeubi]

Not tested anywhere

I have reviewed the result and it all looks fine to me, and I trust the tool to do the right thing, so I'm not sure what kind of test is in your mind?

Export-Package duplicates the automatic module name

I really hope that you do not use io.cucumber.gherkin.utils.* as the module name, and as this relates to the root namespace of the packages this is also not related to that name in any case... I can name my module the-foo and have it contain packages named a.b.c.foo so I don't think it has to match anyways (but I'm not a JPMS expert!)

A magic -noimport:=true flag.
A magic Import-Package: * statement.

These all are bnd instructions and nothing "magic" so I do not really get why it is fragile to use the notation of a tool? e.g. everything in cucumber can be seen "magic" as well of course its all well defined?!?

[mpkorstanje]

These all are bnd instructions and nothing "magic"

That is exactly the magic I'm talking about. 😆

Anyway I've pushed 372d879 to remove the worst of the duplication.

This will work because the recommendations for following module names are from the top of my head:

• Use reverse DNS like package names.
• For a group of packages, use the name of the super package.

We can't follow these rules perfectly because io.cucumber.gherkin also exists, but there currently no overlap.

[mpkorstanje]

If applied to all projects, version management can be done through the cucumber-parent pom.

[laeubi]

If applied to all projects, version management can be done through the cucumber-parent pom.

Yes it would be good to have it at the parent.

[laeubi]

this resulted in a constant stream of broken packages

Can you give an example here? Of course I would expect people to report issues for some special cases, but that's usually not a problem. The biggest problem usually are split packages (what will be an issue with JPMS as well of course!)

We start testing in an OSGI container

I'm not sure what people expect from such test but from the linked issue it is way tooo complex and goes far beyond what I try to archive here. Running feature files with cucumber inside OSGi is of course an interesting part, but that's really something one should do in an own module/project and is also not an easy task. If that is of interest for cucumber community I could probably help with that but such feature would be more a plugin to cucumber than the other way round.

What we would need here is more a way to load and use the classes to allow such use case what is a much smaller scope and does not need a full fledged testing container. It would probably be nice that everything resolves, but nothing that a usual cucumber-devloper should need to worry about.

With Java 8 reaching EOL soon, we can start using the JPMS. Is there a way we can seamlessly translate that too an OSGI definition? Because we will test with the JPMS, that should also cover a programmatically derived OSGI definition.

If JPMS is interesting for you the could maybe benefit from both, as the bnd plugin is able to generate the JPMS infos as well: https://bnd.bndtools.org/chapters/330-jpms.html

I just didn't considered this as everything seems to be Java 8 at the moment.

[mpkorstanje]

Can you give an example here? Of course I would expect people to report issues for some special cases, but that's usually not a problem. The biggest problem usually are split packages (what will be an issue with JPMS as well of course!)

It's in the linked issues but you'll have to read between the lines a bit. We've had a module named pax-exam that used OSGI and constantly broke when a new dependency was added without OSGI or when something was refactored and not included in the OSGI manifest. Some examples:

• Upgrade tag expressions to 1.1.1 cucumber-jvm#1286
• Use gherkin 5.0.0 cucumber-jvm#1251
• Inviting maven project structure and JSONReporter now captures matches/results for each step def. cucumber-attic/gherkin2#188 (comment)
• Use gherkin 5.0.0 cucumber-jvm#1251

We've tried to maintain it for a bit, but telling people to ignore a certain type of failure isn't great either.

• SingleValueConverter registered globally does not get applied to XStream cucumber-jvm#1328 (comment)
• [Core] Refactor Runtime cucumber-jvm#1367 (comment)
• [WIP] Add --wip option cucumber-jvm#1380 (comment)

And of-course every module needs it, so that generates a stream of issues:

• (Java) Make the jar a bundle to support execution in OSGi containers. cucumber-attic/gherkin#221
• Package gherkin.jar as OSGi bundle cucumber-attic/gherkin2#166
• OSGi headers added to mainfest cucumber-attic/cucumber-jvm-deps#3
• Add osgi metadata cucumber-attic/cucumber-html#40
• tag-expressions: (Java) Make the jar a bundle to support osgi. common#99
• cucumber-expressions: (Java) Make the jar a bundle to support osgi. common#287

I hope that clarifies why I've soured on OSGI.

And to re-iterate, I'm not opposed to adding OSGI information all together. But I would like a more elegant, non-fragile, testable solution that also doesn't require knowing much, if anything about OSGI.

I'm not sure what people expect from such test but from the linked issue it is way tooo complex and goes far beyond what I try to archive here.

I'm currently looking for a is a solution that is amongst other things, testable. So if the OSGI definitions are not correct, I want to know that after running mvn verify. If that doesn't exist then I would prefer that cucumber-eclipse deals with the OSGI transformation.

If JPMS is interesting for you the could maybe benefit from both, as the bnd plugin is able to generate the JPMS infos as well: https://bnd.bndtools.org/chapters/330-jpms.html

I'm looking for something in the opposite direction.

I just didn't considered this as everything seems to be Java 8 at the moment.

It still is at the moment, but you can find the rationale for upgrading in cucumber/cucumber-jvm#2962.

[mpkorstanje]

I had a look at the JUnit 5 project to see how they resolved that problem. There seems to be a lot of custom programming done in Gradle to make that work nicely.

And after reading just the introduction https://bnd.bndtools.org I'm not expecting stellar Maven support. 😄

If you insist on using Maven, then you can start with the description of the Maven plugins.

You'd hope that people who want you to provide meta-information for their container system would make it easy and painless for you to provide it, not come at you with an attitude.

[laeubi]

It's in the linked issues but you'll have to read between the lines a bit. We've had a module named pax-exam that used OSGI and constantly broke when a new dependency was added without OSGI or when something was refactored and not included in the OSGI manifest.

So pax-exam is a tool for testing OSGi container and does not releate in any way to generation of OSGi metadata, but how does it "break packages"? I can understand that it might break the developers workflow in having to upgrade that thing, but as mentioned this is almost total overkill anyways for the thing that should be archived here... So you are actually facing different issues here that steems maybe from the fact how it was implemented was not that optimal in the past.

And to re-iterate, I'm not opposed to adding OSGI information all together. But I would like a more elegant, non-fragile, testable solution that also doesn't require knowing much, if anything about OSGI.

I think we can forget about the "testable" here, because I think it requires some kind of definition what/how to test.

So if you looking at my proposed solution here (and forget about the frustration in the past maybe) I have chosen one that:

1. Is as less intrusive as possible e.g. you do not need to change packaging types or otherwise modify how you build your jar. It only requires a little configuration of the jar-plugin but I think its something easy to grasp.
2. Is totally silent e.g. it does not yet fail if a dependency is not a bundle (what does not matter here) or require you to "know" anything about OSGi in general

So what is the worst can happen? There might need to be some things adjusted for the sake of OSGi compatibility and then you can ask the people to do that for you (as I did it here) and then again stop to care as it does not affects you (as a no OSGi user).

Given that I wonder, how much simpler a solution has to be to be considered here?

I'm currently looking for a is a solution that is amongst other things, testable. So if the OSGI definitions are not correct, I want to know that after running mvn verify.

If that would be the case, then the tool would be buggy. I think no one ever "verifies" class files... or jar files or ... produced by a tool because then you need to verify that the verifiers is correct and verify that and so on.

So I think at this point we need to make the assumption that the generated meta-data is correct (and I have no evidence from the thousands of people using that ever reported a complete failure of the tool).

I'm looking for something in the opposite direction.

As OSGi has richer metadata than JPMS and also things that can not be expressed in JPMS I really doubt something exits. Also often JPMS infos written manually are just wrong or incomplete.

I had a look at the JUnit 5 project to see how they resolved that problem. There seems to be a lot of custom programming done in Gradle to make that work nicely.

Well that's how gradle users supposed to do their work :-P
.. seriously JUnit is maybe not the best as an example to learn from. Still there are many projects using that in a much easier way for many years now. So you can find better example out there I think.

And after reading just the introduction https://bnd.bndtools.org I'm not expecting stellar Maven support. 😄

If you insist on using Maven, then you can start with the description of the Maven plugins.

You'd hope that people who want you to provide meta-information for their container system would make it easy and painless for you to provide it, not come at you with an attitude.

Well as a matter of fact, bndlib is most used in maven (e.g. Tycho, Felix Bundle Plugin and a like), but please don't take the mind or attitude of a project to be representative for OSGi (what is homed here https://www.osgi.org/) what is of course tool agnostic. Still bndlib is a good tool regardless of some its questionable statements in the documentation.

[mpkorstanje]

Given that I wonder, how much simpler a solution has to be to be considered here?

The complexity I'd prefer would be adding a single Maven plugin without configuration, then have everything work out of the box. If there is a JPMS module, then that is translated automatically. If it is a regular Jar, then import everything and export all the packages. And preferably all that is written to a manifest in OSGI-INF/MANIFEST.MF rather than added to the jar META-INF/MANIFEST.MF so we don't have to instruct the maven jar plugin.

Now I've played around with biz.aQute.bnd:bnd-maven-plugin a bit and it works much better than org.apache.felix:maven-bundle-plugin that we used in the past so I've got some confidence now that with the current state of the PR the tool will just work.

[mpkorstanje]

@laeubi any reason not to use v7.1?

[laeubi]

@laeubi any reason not to use v7.1?

This requires the build to use Java 17, I think that still work to build for java 8 then but don't wanted to change to much at once.

[mpkorstanje]

Okay, please upgrade and you can drop 11 from the matrix. We don't build against 8 for a long time already.

[laeubi]

If there is a JPMS module, then that is translated automatically.

BTW for reference

• [core] jpms support osgi/osgi#782

And preferably all that is written to a manifest in OSGI-INF/MANIFEST.MF rather than added to the jar META-INF/MANIFEST.MF so we don't have to instruct the maven jar plugin.

That's sadly a restriction of the maven-jar plugin and is required to have it more independent of each others.

[mpkorstanje]

I'm confident enough that this will work without too much fuss.

@laeubi please upgrade the plugin to the latest version and add a change long entry before merging. Semverwise, I think this would be an ### Added header, but correct me if you think I got that wrong.

[laeubi]

@mpkorstanje thanks for reviewing this, I think this is actually only a micro change, nothing changes API wise for consumers.

Also many thanks for consideration I know it could be a bit confusing for non-OSGi users and therefore I really like to understand how we can improve things here, I'm currently investigate writing a small tool that checks the dependencies of the produced metadata so we even have a little check.

And maybe first the cucumber-messages (that are the only dependency here) should be addressed so we get a proper version range automatically, so there is no need to merge this immediately, just let me know if a release is planned anyways, then it could be a good time to include this as well (maybe improve on it later on).

Some things to check:

1. Should I add the bnd-maven-plugin to the parent first?
2. Do you prefer the style with a separate file (we can place it anywhere you thing its suitable) or should we use the used inline style here?

[mpkorstanje]

Should I add the bnd-maven-plugin to the parent first?

That's probably sensible to do. But the ordering of operations isn't important though. It can be done before or after this PR is merged.

Do you prefer the style with a separate file (we can place it anywhere you thing its suitable) or should we use the used inline style here?

Inline style. Then we can reuse the Automatic Module name. That will allow easier copy/paste.

[mpkorstanje]

so there is no need to merge this immediately, just let me know if a release is planned anyways, then it could be a good time to include this as well (maybe improve on it later on).

Releases aren't planned. They happen ad-hoc, when there is something to release.

And maybe first the cucumber-messages (that are the only dependency here) should be addressed so we get a proper version range automatically,

I'm not 100% following this.

[laeubi]

I'm not 100% following this.

@mpkorstanje sorry for confusion.

So first I started on a small mojo that allows to verify some basic things (e.g. symbolic name is there and version is given) and then analyze the generated manifest if it can be consumed easily, this is something we can add after the next Tycho release as an additional way to get confidence:

• Add a new tycho-wrap:verify eclipse-tycho/tycho#5402

when I run this tool against my PR it currently gives me this report:

[INFO] --- tycho-wrap:6.0.0-SNAPSHOT:verify (default) @ gherkin-utils ---
[INFO] The Bundle-SymbolicName is: gherkin-utils
[WARNING] The OSGi specification recommends to use a reverse domain name for the 'Bundle-SymbolicName' but the current value do not seem to match!
See https://docs.osgi.org/specification/osgi.core/8.0.0/framework.module.html#d0e2086 for details.
[INFO] 10.0.1.SNAPSHOT
[INFO] It has 7 requirements:
[INFO]  ⚠ Import-Package: io.cucumber.messages.types (can be provided by io.cucumber:messages:jar:29.0.1 but artifact is not an OSGi bundle!)
[INFO]  ✓ Import-Package: java.lang (provided by the JVM)
[INFO]  ✓ Import-Package: java.lang.invoke (provided by the JVM)
[INFO]  ✓ Import-Package: java.util (provided by the JVM)
[INFO]  ✓ Import-Package: java.util.function (provided by the JVM)
[INFO]  ✓ Import-Package: java.util.regex (provided by the JVM)
[INFO]  ✓ Import-Package: java.util.stream (provided by the JVM)
[INFO] It provides 2 capabilities:
[INFO]  - Export-Package: io.cucumber.gherkin.utils; bundle-symbolic-name="gherkin-utils"; bundle-version="10.0.1.SNAPSHOT"; version="10.0.1"; uses:="io.cucumber.messages.types"
[INFO]  - Export-Package: io.cucumber.gherkin.utils.pretty; bundle-symbolic-name="gherkin-utils"; bundle-version="10.0.1.SNAPSHOT"; version="10.0.1"; uses:="io.cucumber.messages.types"

As you can see there is one warning because gherkin-utils requires something from messages but that jar is not a bundle. So it would be good to add the OSGi info first there (as it seems on the start of the food chain).

If I do that locally the output is as follows and all checks are green:

[INFO] It has 7 requirements:
[INFO]  ✓ Import-Package: io.cucumber.messages.types; version="[29.0.0,30.0.0)" (provided by io.cucumber:messages:jar:29.0.2-SNAPSHOT)
[INFO]  ✓ Import-Package: java.lang (provided by the JVM)
[INFO]  ✓ Import-Package: java.lang.invoke (provided by the JVM)
[INFO]  ✓ Import-Package: java.util (provided by the JVM)
[INFO]  ✓ Import-Package: java.util.function (provided by the JVM)
[INFO]  ✓ Import-Package: java.util.regex (provided by the JVM)
[INFO]  ✓ Import-Package: java.util.stream (provided by the JVM)
[INFO] It provides 2 capabilities:
[INFO]  - Export-Package: io.cucumber.gherkin.utils; bundle-symbolic-name="gherkin-utils"; bundle-version="10.0.1.SNAPSHOT"; version="10.0.1"; uses:="io.cucumber.messages.types"
[INFO]  - Export-Package: io.cucumber.gherkin.utils.pretty; bundle-symbolic-name="gherkin-utils"; bundle-version="10.0.1.SNAPSHOT"; version="10.0.1"; uses:="io.cucumber.messages.types"

As you see it now uses a version range for the package what makes it less likely to break if there is ever a major change in the requirement, so results in better metadata.

So to summarize it seems best for me:

1. Add the bnd-maven plugin to the parent
2. Add the metadata to messages
3. After that continue here (to use both of 1+2)

I hope it makes it more clear!

[mpkorstanje]

Yes! Thank you.

[laeubi]

For the first step I now created:

• Generate OSGi metadata cucumber-parent#61

[laeubi]

@mpkorstanje I tried to push branch directly but it says:

Can't connect to any repository: [email protected]:cucumber/gherkin-utils.git ([email protected]:cucumber/gherkin-utils.git: push not permitted)

I think this is because I have not yet comitted to this repo before?

[mpkorstanje]

Try again! Looks like some defaults that were applied to all repos are no longer applied by default.