CI: Adjust error matching for newer json gem / newer JRuby #161

sebastianrakel · 2025-07-26T18:40:27Z

A bit of backstory. I did a lot of JRuby debugging in #122. Yesterday @sebastianrakel and I did a bit of deep diving. Some numbers:

puppetserver 8.7.0, the last open source release from Perforce, ships JRuby 9.4.8.0
openvox-server 8.9.0 ships JRuby 9.4.8.0
puppet enterprise 2025.4.0 (latest version at 2025-07-27) ships pe-puppetserver JRuby 9.4.8.0
latest JRuby 9.4 is 9.4.13.0
JRuby 9.4 series is compatible with MRI Ruby 3.1
puppetserver 7 uses JRuby 9.3 which is compatible with Ruby 2.6
Perforce never tested JRuby 9.4.8.0 in CI, JRuby was pinned to 9.4.3.0

Since openvox runs tests, the Rspec JRuby pipeline fails. We cannot access the old perforce pipeline logs. Their last successful CI run I could fine was https://github.com/puppetlabs/puppet/actions/runs/11693993097. This was triggered by commit puppetlabs/puppet@4ea6ee2 on 2024-11-06.

In https://github.com/puppetlabs/puppet/actions/runs/12021455668 we see the first failed pipelline for puppetserver 8, triggered by puppetlabs/puppet@e967115 on 2024-11-24. The used JRuby verison in both jobs was 9.4.3.0

Something changed between 2024-11-06 and 2024-11-24 and I doubt it was related to puppetlabs/puppet@e967115 (puppetlabs/puppet#9523).

Whatever changed in that timeframe modified the error reporting. Basic example:

# manifests/site.pp
class a {}
class a {}

raises:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Class 'a' is already defined (file: /etc/puppetlabs/code/environments/server/manifests/site.pp, line: 1); cannot redefine (file: /etc/puppetlabs/code/environments/server/manifests/site.pp, line: 2) (file: /etc/puppetlabs/code/environments/server/manifests/site.pp, line: 2) on node puppettestinfra.voxpupu.li

After "cannot redefine", the file is listed twice. This issue is reproducible puppetserver 8.7.0, Puppet Enterprise 2025.4.0 & openvox-server 8.9.0.

After some testing in #122 I noticed that JRuby 9.4.3.0 indeed raises errors now, but others. I assume the JRuby issues are, at least in part, triggered by a webmock update. I don't know yet what causes the duplicate file lines in the output. The old rspec tests demand a single file line in the output.

In #164 I'm testing different Jruby Versions with a webmock pinned to 3.25.0. 3.25.0 was released on 2025-02-06. The version before that, 3.24.0, on 2024-09-30. That's what Perforce was using back then.

And after diffing CI output from JRuby 9.4.3, 9.4.4, 9.4.5 & 9.4.6 we can see that the duplicate line messages start with JRuby 9.4.6: https://gist.github.com/bastelfreak/025c23417f9dd7215b5c5af205fc8b76

bastelfreak · 2025-07-27T10:01:43Z

A bit of backstory. I did a lot of JRuby debugging in #122. Yesterday @sebastianrakel and I did a bit of deep diving. Some numbers:

puppetserver 8.7.0, the last open source release from Perforce, ships JRuby 9.4.8.0
openvox-server 8.9.0 ships JRuby 9.4.8.0
puppet enterprise 2025.4.0 (latest version at 2025-07-27) ships pe-puppetserver JRuby 9.4.8.0
latest JRuby 9.4 is 9.4.13.0
JRuby 9.4 series is compatible with MRI Ruby 3.1
puppetserver 7 uses JRuby 9.3 which is compatible with Ruby 2.6
Perforce never tested JRuby 9.4.8.0 in CI, JRuby was pinned to 9.4.3.0

Since openvox runs tests, the Rspec JRuby pipeline fails. We cannot access the old perforce pipeline logs. Their last successful CI run I could fine was https://github.com/puppetlabs/puppet/actions/runs/11693993097. This was triggered by commit puppetlabs/puppet@4ea6ee2 on 2024-11-06.

In https://github.com/puppetlabs/puppet/actions/runs/12021455668 we see the first failed pipelline for puppetserver 8, triggered by puppetlabs/puppet@e967115 on 2024-11-24. The used JRuby verison in both jobs was 9.4.3.0

Something changed between 2024-11-06 and 2024-11-24 and I doubt it was related to puppetlabs/puppet@e967115 (puppetlabs/puppet#9523).

Whatever changed in that timeframe modified the error reporting. Basic example:

# manifests/site.pp
class a {}
class a {}

raises:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Class 'a' is already defined (file: /etc/puppetlabs/code/environments/server/manifests/site.pp, line: 1); cannot redefine (file: /etc/puppetlabs/code/environments/server/manifests/site.pp, line: 2) (file: /etc/puppetlabs/code/environments/server/manifests/site.pp, line: 2) on node puppettestinfra.voxpupu.li

After "cannot redefine", the file is listed twice. This issue is reproducible puppetserver 8.7.0, Puppet Enterprise 2025.4.0 & openvox-server 8.9.0.

After some testing in #122 I noticed that JRuby 9.4.3.0 indeed raises errors now, but others. I assume the JRuby issues are, at least in part, triggered by a webmock update. I don't know yet what causes the duplicate file lines in the output. The old rspec tests demand a single file line in the output.

In #164 I'm testing different Jruby Versions with a webmock pinned to 3.25.0. 3.25.0 was released on 2025-02-06. The version before that, 3.24.0, on 2024-09-30. That's what Perforce was using back then.

And after diffing CI output from JRuby 9.4.3, 9.4.4, 9.4.5 & 9.4.6 we can see that the duplicate line messages start with JRuby 9.4.6: https://gist.github.com/bastelfreak/025c23417f9dd7215b5c5af205fc8b76

bastelfreak · 2025-07-27T10:15:28Z

spec/unit/http/service/compiler_spec.rb

@@ -594,7 +594,7 @@
      invalid_facts = Puppet::Node::Facts.new(certname, {'invalid_utf8_sequence' => "\xE2\x82".force_encoding('binary')})
      expect {
        subject.put_facts(certname, environment: 'production', facts: invalid_facts)
-      }.to raise_error(Puppet::HTTP::SerializationError, /Failed to serialize Puppet::Node::Facts to json: ("\\xE2" from ASCII-8BIT to UTF-8|partial character in source, but hit end)/)


I don't know yet what would have raised "\\xE2" from ASCII-8BIT to UTF-8. partial character in source, but hit end should come from the json gem java version: https://github.com/ruby/json/blob/master/java/src/json/ext/StringDecoder.java#L168-L176.

Maybe that exception just isn't raised anymore 🤔

bastelfreak · 2025-07-27T10:16:37Z

spec/unit/util/json_spec.rb

@@ -23,7 +23,7 @@
    it 'raises an error if JSON is invalid' do
      expect {
        Puppet::Util::Json.load('{ invalid')
-      }.to raise_error(Puppet::Util::Json::ParseError, /expected object key, got 'invalid' at line 1 column 3/)


This also looks like an error from the json lib:

bastelfreak@bastelfreak-nb ~/code/json $ git grep 'expected object key' ext/json/ext/parser/parser.c: raise_parse_error("expected object key, got %s", state); ext/json/ext/parser/parser.c: raise_parse_error("expected object key, got: %s", state); test/json/json_ext_parser_test.rb: assert_equal "expected object key, got EOF at line 1 column 5", ex.message bastelfreak@bastelfreak-nb ~/code/json $

Hahaha. While digging around I found ruby/json#754, which I raised some months ago. From the developers:

So it's probably best if your test only assume a specific error class, but doesn't try to match the message content.

Still it would be nice to figure out it this passes with an older json gem version.

jay7x · 2025-08-13T04:20:13Z

Had a look at jruby 9.4.6.0 changelog and they did some changes around exceptions handling and printing (jruby/jruby#8040 e.g.).
Also I did quick googling and found no reports about duplicate file & line position info in the exceptions.. so I guess it's something in the code we own now

@sebastianrakel

A bit of backstory. I did a lot of JRuby debugging in #122. Yesterday @sebastianrakel and I did a bit of deep diving. Some numbers: * puppetserver 8.7.0, the last open source release from Perforce, ships JRuby 9.4.8.0 * openvox-server 8.9.0 ships JRuby 9.4.8.0 * puppet enterprise 2025.4.0 (latest version at 2025-07-27) ships pe-puppetserver JRuby 9.4.8.0 * latest JRuby 9.4 is 9.4.13.0 * JRuby 9.4 series is compatible with MRI Ruby 3.1 * puppetserver 7 uses JRuby 9.3 which is compatible with Ruby 2.6 * Perforce never tested JRuby 9.4.8.0 in CI, JRuby was pinned to 9.4.3.0 Since openvox runs tests, the Rspec JRuby pipeline fails. We cannot access the old perforce pipeline logs. Their last successful CI run I could fine was https://github.com/puppetlabs/puppet/actions/runs/11693993097. This was triggered by commit puppetlabs/puppet@4ea6ee2 on 2024-11-06. In https://github.com/puppetlabs/puppet/actions/runs/12021455668 we see the first failed pipelline for puppetserver 8, triggered by puppetlabs/puppet@e967115 on 2024-11-24. The used JRuby verison in both jobs was 9.4.3.0 Something changed between 2024-11-06 and 2024-11-24 and I doubt it was related to puppetlabs/puppet@e967115 (puppetlabs/puppet#9523). Whatever changed in that timeframe modified the error reporting. Basic example: ``` class a {} class a {} ``` raises: ``` Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Class 'a' is already defined (file: /etc/puppetlabs/code/environments/server/manifests/site.pp, line: 1); cannot redefine (file: /etc/puppetlabs/code/environments/server/manifests/site.pp, line: 2) (file: /etc/puppetlabs/code/environments/server/manifests/site.pp, line: 2) on node puppettestinfra.voxpupu.li ``` After "cannot redefine", the file is listed twice. This issue is reproducible puppetserver 8.7.0, Puppet Enterprise 2025.4.0 & openvox-server 8.9.0. After some testing in #122 I noticed that JRuby 9.4.3.0 indeed raises errors now, but others. I assume the JRuby issues are, at least in part, triggered by a webmock update. I don't know yet what causes the duplicate file lines in the output. The old rspec tests demand a single file line in the output. In #164 I'm testing different Jruby Versions with a webmock pinned to 3.25.0. 3.25.0 was released on 2025-02-06. The version before that, 3.24.0, on 2024-09-30. That's what Perforce was using back then. And after diffing CI output from JRuby 9.4.3, 9.4.4, 9.4.5 & 9.4.6 we can see that the duplicate line messages start with JRuby 9.4.6: https://gist.github.com/bastelfreak/025c23417f9dd7215b5c5af205fc8b76

bastelfreak · 2025-08-24T14:40:58Z

@jay7x given this is also the current state in PE, I think it's fine to merge it.

sebastianrakel marked this pull request as draft July 26, 2025 18:40

sebastianrakel force-pushed the fix-jruby branch 7 times, most recently from 1f5ca5a to da79f36 Compare July 27, 2025 09:47

sebastianrakel force-pushed the fix-jruby branch from da79f36 to b243afd Compare July 27, 2025 10:08

bastelfreak reviewed Jul 27, 2025

View reviewed changes

bastelfreak force-pushed the fix-jruby branch from b243afd to bec8ad7 Compare August 24, 2025 14:39

bastelfreak marked this pull request as ready for review August 24, 2025 14:40

bastelfreak enabled auto-merge August 24, 2025 14:41

bastelfreak approved these changes Aug 24, 2025

View reviewed changes

bastelfreak changed the title ~~Fix jruby tests~~ CI: Adjust error message matching for newer json gem / newer JRuby Aug 24, 2025

bastelfreak changed the title ~~CI: Adjust error message matching for newer json gem / newer JRuby~~ CI: Adjust error matching for newer json gem / newer JRuby Aug 24, 2025

bastelfreak added the skip-changelog Excluded from CHANGELOG label Aug 24, 2025

bastelfreak merged commit 8e5f23b into main Aug 24, 2025
16 checks passed

bastelfreak deleted the fix-jruby branch August 24, 2025 15:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CI: Adjust error matching for newer json gem / newer JRuby #161

CI: Adjust error matching for newer json gem / newer JRuby #161

Uh oh!

sebastianrakel commented Jul 26, 2025 •

edited by bastelfreak

Loading

Uh oh!

bastelfreak commented Jul 27, 2025 •

edited

Loading

Uh oh!

bastelfreak Jul 27, 2025

Uh oh!

bastelfreak Jul 27, 2025

Uh oh!

bastelfreak Jul 27, 2025 •

edited

Loading

Uh oh!

jay7x commented Aug 13, 2025

Uh oh!

bastelfreak commented Aug 24, 2025

Uh oh!

Uh oh!

Uh oh!

CI: Adjust error matching for newer json gem / newer JRuby #161

CI: Adjust error matching for newer json gem / newer JRuby #161

Uh oh!

Conversation

sebastianrakel commented Jul 26, 2025 • edited by bastelfreak Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bastelfreak commented Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bastelfreak Jul 27, 2025

Choose a reason for hiding this comment

Uh oh!

bastelfreak Jul 27, 2025

Choose a reason for hiding this comment

Uh oh!

bastelfreak Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jay7x commented Aug 13, 2025

Uh oh!

bastelfreak commented Aug 24, 2025

Uh oh!

Uh oh!

Uh oh!

sebastianrakel commented Jul 26, 2025 •

edited by bastelfreak

Loading

bastelfreak commented Jul 27, 2025 •

edited

Loading

bastelfreak Jul 27, 2025 •

edited

Loading