Emit restart events when a job is enqueued #6816

d3flex · 2025-10-22T12:33:55Z

With this commit once the job goes to its end, it will check if it can restarted and if it fulfills the criteria, it will send the new job to the minion queue via enqueue_restart, where now it should populate the event.

issue: https://progress.opensuse.org/issues/190557

d3flex · 2025-10-22T12:37:25Z

I want to see where it will fail (some test failed on my side) and the codecov report
also I cant see where a test for this should go. there is t/10-jobs.t as an obvious candidate but it doesnt seem to cover the events

codecov · 2025-10-22T12:52:41Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.26%. Comparing base (1801a6d) to head (4300915).
⚠️ Report is 16 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6816   +/-   ##
=======================================
  Coverage   99.26%   99.26%           
=======================================
  Files         402      402           
  Lines       41396    41460   +64     
=======================================
+ Hits        41090    41154   +64     
  Misses        306      306

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

perlpunk · 2025-10-22T13:09:47Z

also I cant see where a test for this should go. there is t/10-jobs.t as an obvious candidate but it doesnt seem to cover the events

t/23-amqp.t looks right in this case. Take a job and call enqueue_restart on it

okurz

Needs tests

okurz

code looks good so far. I suggest you do a manual system test with a local openQA instance and ensure that no duplicate events are sent.

Also in your commit message I don't quite understand "With this commit once the job goes to its end, it will check if it can restarted and if it fulfills the criteria". What "end" and what "criteria"? Also "check if it can restarted" is grammatically incorrect

d3flex · 2025-10-23T11:43:31Z

code looks good so far. I suggest you do a manual system test with a local openQA instance and ensure that no duplicate events are sent.

Also in your commit message I don't quite understand "With this commit once the job goes to its end, it will check if it can restarted and if it fulfills the criteria". What "end" and what "criteria"? Also "check if it can restarted" is grammatically incorrect

d3flex · 2025-10-23T13:05:58Z

code looks good so far. I suggest you do a manual system test with a local openQA instance and ensure that no duplicate events are sent.

Also in your commit message I don't quite understand "With this commit once the job goes to its end, it will check if it can restarted and if it fulfills the criteria". What "end" and what "criteria"? Also "check if it can restarted" is grammatically incorrect

updated.

perlpunk · 2025-10-23T13:11:21Z

With this commit once the job goes to execute its final stage, it will
check if it can be restarted. If yes, it will send the new job to
the minion queue via enqueue_restart, where now it should populate
the event.

The only change in this commit is that, if the job is restarted via enqueue_restart, it now publishes a restart event.
The description sounds overly complicated ("goes to execute its final stage").
" it will check if it can be restarted" - that's already happening without your commit.

How about: "When a job is enqueued to be restarted via the RETRY feature, it now publishes a job.restart event."

d3flex · 2025-10-23T13:59:25Z

With this commit once the job goes to execute its final stage, it will
check if it can be restarted. If yes, it will send the new job to
the minion queue via enqueue_restart, where now it should populate
the event.

The only change in this commit is that, if the job is restarted via enqueue_restart, it now publishes a restart event. The description sounds overly complicated ("goes to execute its final stage"). " it will check if it can be restarted" - that's already happening without your commit.

How about: "When a job is enqueued to be restarted via the RETRY feature, it now publishes a job.restart event."

I avoid to reffer RETRY because is not the only path to enqueue_restart. are you good with When a job is enqueued to be restarted, it now publishes a job.restart event?

perlpunk · 2025-10-23T15:46:56Z

When a job is enqueued to be restarted as part of the $job->done handling, a job.restart event is now published

t/23-amqp.t

lib/OpenQA/Schema/Result/Jobs.pm

t/20-stale-job-detection.t

Martchus

As mentioned in the inline comments, this might be much harder to make actually work than we estimated. Maybe we should reevaluate the best way of implementing this and possibly also re-consider whether it is worth it.

Martchus · 2025-11-04T10:31:22Z

lib/OpenQA/Task/Job/Restart.pm

 sub restart_openqa_job ($minion_job, $openqa_job) {
    my $cloned_job_or_error = $openqa_job->auto_duplicate;
    my $is_ok = ref $cloned_job_or_error || $cloned_job_or_error =~ qr/(already.*clone|direct parent)/i;
+    OpenQA::Events->singleton->emit_event('job_restart', data => {id => $openqa_job->id}) if $is_ok;


To make this actually work we would probably need to make sure the AMQP plugin is loaded in the Gru service. It probably isn't so emitting this event probably has no effect. Now that I think about it, running the AMQP plugin within the Gru service and the processes it forks to run the particular Minion jobs is probably not so easy and could end up becoming quite a mess.

That's a good one. Lets take a step back.

It probably isn't so emitting this event probably has no effect

I am not sure about that but the emit_event in lib/OpenQA/Events.pm implies that this method is intended for non-controller contexts, such as Minion background tasks. right?

To make this actually work we would probably need to make sure the AMQP plugin is loaded in the Gru service.

Maybe is enough to make the Openqa::events::emit_events clever and enqueue an event from there? could that possibly work?

Yes, it is intended to be used from non-controller context but not Minion tasks. I think "non-controller context" mainly refers to DBIx result classes or some utility modules.

We probably don't need to make emit_events clever. We just need to make sure that in the process that runs the Minion task the AMQP plugin is initialized (and maybe also other similar plugins for the sake of consistency). The problem with that is just that it might get messy due to the involved forking in combination with having to manage a permanent connection to the AMQP server.

Of course we could actually make emit_event clever and make it invoke some internal route of the normal web UI service to emit the event in case it is running from the Gru service. That might be the easiest solution - even though it requires a new internal route to do IPC.

I guess this can closed as well?

Martchus · 2025-11-04T10:36:16Z

t/api/18-job-amqp-event.t

+    combined_like { $t->post_ok('/api/v1/jobs/99926/restart?force=1')->status_is(200) } qr/Job 99926 duplicated/,
+      'Job restarted successfully';


This route is not what your ticket is about. Of course it makes also sense to add an explicit test for this in case we don't already have one. However, the relevant route would be the one that ends with /done I suppose. And then we'd of course need to run the Minion job enqueued by that route. And then you'll probably run into the problem mentioned in my other review comment.

Yes, and I use this opportunity to remind about TDD (test driven development).
I took this PR, but only the new test and not the code changes.
The test passed.

It should be clear that this should not be the case. If you write a new test for a new or changed feature, then the test must fail until you add the actual code changes. If it passes, then you are not testing the new feature.

For checking this rule it isn't even important whether you started development with a test or not.

So for the future: This is an easy rule to check for yourself before you push something to a PR.

if I remember correctly, I added that change mainly because was not covered in the api tests. Apparently thats why it is added in t/api. it could be something else as well simultaneously. in a few words, this was not meant to cover the new case. The emission from the minion job should have implemented in 10-jobs.t:AMQP event emission for minion restarts

If this is covering an existing case that was not covered before, it should be in its own commit.

lib/OpenQA/Task/Job/Restart.pm

t/10-jobs.t

perlpunk · 2025-11-24T11:21:06Z

t/10-jobs.t

+    %published = ();
+    $job->done(result => FAILED);
+    stdout_like { perform_minion_jobs($minion) } qr/Job \d+ duplicated as \d+/;
+    my $expected_topic = 'suse.openqa.job.restart';


IMHO it's hard to read what is tested here, and some tests are redundant.
I suggest the following diff:

diff --git a/t/10-jobs.t b/t/10-jobs.t index d8a706c28..9719980d7 100644 --- a/t/10-jobs.t +++ b/t/10-jobs.t @@ -974,22 +974,20 @@ subtest 'AMQP event emission for minion restarts' => sub { %published = (); $job->done(result => FAILED); stdout_like { perform_minion_jobs($minion) } qr/Job \d+ duplicated as \d+/; - my $event = OpenQA::Test::Case::find_most_recent_event($schema, 'job_restart'); - my $expected_topic = 'suse.openqa.job.restart'; - my @restart_events = grep { $_ eq $expected_topic } keys %published; - ok exists $published{$expected_topic}, 'restart event published via AMQP'; - is $published{$expected_topic}{id}, $job_id, 'event contains original job ID'; - is $published{$expected_topic}{auto}, 1, 'event marked as auto restart'; - is scalar(@restart_events), 1, 'exactly one job restart event emitted'; - is $event_body[0], 'openqa_job_restart', 'event type is openqa_job_restart'; + is scalar keys %published, 1, 'exactly one job restart event emitted'; + my $event = $published{'suse.openqa.job.restart'}; + is $event->{id}, $job_id, 'event contains original job ID'; + is $event->{auto}, 1, 'event marked as auto restart'; + my ($user_id, $connection, $type, $data) = @{$event_body[1]}; is $user_id, undef, 'user_id is undef for Minion restart'; is $connection, undef, 'connection is undef for Minion restart'; is $type, 'openqa_job_restart', 'type matches event type'; - is $data->{id}, $job_id, 'data contains original job ID'; - is $data->{auto}, 1, 'data marked as auto restart'; - is ref($data->{result}), 'HASH', 'result should be a plain hash like API'; - is_deeply $data->{result}, {$job_id => $data->{result}{$job_id}}, 'result shows cloned job info'; + + is_deeply $data, $event, 'published event equals emitted event'; + ok exists $data->{result}->{$job_id}, 'old job id is in result'; + $job->discard_changes; + is $job->clone_id, $data->{result}->{$job_id}, 'clone_id points to reported id'; };

perlpunk · 2025-11-24T11:22:38Z

t/10-jobs.t

+    my $expected_topic = 'suse.openqa.job.restart';
+    my @restart_events = grep { $_ eq $expected_topic } keys %published;
+    ok exists $published{$expected_topic}, 'restart event published via AMQP';
+    is $published{$expected_topic}{id}, $job_id, 'event contains original job ID';
+    is $published{$expected_topic}{auto}, 1, 'event marked as auto restart';
+    is scalar(@restart_events), 1, 'exactly one job restart event emitted';
+    is $event_body[0], 'openqa_job_restart', 'event type is openqa_job_restart';
+    my ($user_id, $connection, $type, $data) = @{$event_body[1]};
+    is $user_id, undef, 'user_id is undef for Minion restart';
+    is $connection, undef, 'connection is undef for Minion restart';
+    is $type, 'openqa_job_restart', 'type matches event type';
+    is $data->{id}, $job_id, 'data contains original job ID';
+    is $data->{auto}, 1, 'data marked as auto restart';
+    is ref($data->{result}), 'HASH', 'result should be a plain hash like API';
+    is_deeply $data->{result}, {$job_id => $data->{result}{$job_id}}, 'result shows cloned job info';


Alternatively here as a suggestion:

Suggested change

my $expected_topic = 'suse.openqa.job.restart';

my @restart_events = grep { $_ eq $expected_topic } keys %published;

ok exists $published{$expected_topic}, 'restart event published via AMQP';

is $published{$expected_topic}{id}, $job_id, 'event contains original job ID';

is $published{$expected_topic}{auto}, 1, 'event marked as auto restart';

is scalar(@restart_events), 1, 'exactly one job restart event emitted';

is $event_body[0], 'openqa_job_restart', 'event type is openqa_job_restart';

my ($user_id, $connection, $type, $data) = @{$event_body[1]};

is $user_id, undef, 'user_id is undef for Minion restart';

is $connection, undef, 'connection is undef for Minion restart';

is $type, 'openqa_job_restart', 'type matches event type';

is $data->{id}, $job_id, 'data contains original job ID';

is $data->{auto}, 1, 'data marked as auto restart';

is ref($data->{result}), 'HASH', 'result should be a plain hash like API';

is_deeply $data->{result}, {$job_id => $data->{result}{$job_id}}, 'result shows cloned job info';

is scalar keys %published, 1, 'exactly one job restart event emitted';

my $event = $published{'suse.openqa.job.restart'};

is $event->{id}, $job_id, 'event contains original job ID';

is $event->{auto}, 1, 'event marked as auto restart';

my ($user_id, $connection, $type, $data) = @{$event_body[1]};

is $user_id, undef, 'user_id is undef for Minion restart';

is $connection, undef, 'connection is undef for Minion restart';

is $type, 'openqa_job_restart', 'type matches event type';

is_deeply $data, $event, 'published event equals emitted event';

ok exists $data->{result}->{$job_id}, 'old job id is in result';

$job->discard_changes;

is $job->clone_id, $data->{result}->{$job_id}, 'clone_id points to reported id';

I used the later

perlpunk

I don't like the third commit message subject much "Make restart_openqa_job emit event same as API" (what should "same as API" tell us?)
Also the message from the first commit wasn't fixed yet, it still contains the "TBD" thing from me.
Otherwise I'd approve.

d3flex · 2025-11-24T16:58:11Z

I don't like the third commit message subject much "Make restart_openqa_job emit event same as API" (what should "same as API" tell us?) Also the message from the first commit wasn't fixed yet, it still contains the "TBD" thing from me. Otherwise I'd approve.

I will look at the commit.
the TBD was not totally sure what was about. edit the commit or feature to be done. However I have modified it a bit.
Do not approve yet. I try to fix the codecov

That should make sure that the event is send to AMQP for jobs with `RETRY` once it is actually triggered. Using the `restart_openqa_job` from the Restart module in order to not duplicate the event which is emitted by the API, which also invoke `auto_duplicate`. It should also trigger the event when the job is processed and not while is queued. To do so, it loads the AMQP plugin to the Gru job. The `one_tick` is called to trigger the `next_tick` in the `register` function of the AMPQ module immediately, otherwise it seems like it is registered late and the event is not emitted. issue: https://progress.opensuse.org/issues/190557 Signed-off-by: Ioannis Bonatakis <[email protected]>

The plugin gets loaded automatically via OpenQA::Setup

Start Mojo::IOLoop and let it end when the AMQP plugin has finished (or failed) publishing the event

d3flex · 2025-11-24T17:44:42Z

I don't like the third commit message subject much "Make restart_openqa_job emit event same as API" (what should "same as API" tell us?) Also the message from the first commit wasn't fixed yet, it still contains the "TBD" thing from me. Otherwise I'd approve.

I will look at the commit. the TBD was not totally sure what was about. edit the commit or feature to be done. However I have modified it a bit. Do not approve yet. I try to fix the codecov

Commit messages were updated.
I didnt find a solution for the missing coverage (https://app.codecov.io/gh/os-autoinst/openQA/pull/6816?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=checks&utm_campaign=pr+comments&utm_term=os-autoinst). I marked it with # uncoverable statement. theoretically it can test it as a unit test. but in 10-jobs.t? is it worth it after all?

lib/OpenQA/Task/Job/Restart.pm

t/10-jobs.t

And create test coverage - Covers the emission of events on Minion restarts - Loads the AMQP plugin - Checks event body for consistency with events from API issue: https://progress.opensuse.org/issues/190557 Signed-off-by: Ioannis Bonatakis <[email protected]>

d3flex added the not-ready label Oct 22, 2025

perlpunk mentioned this pull request Oct 22, 2025

Emit restart events from _hook_script #6813

Closed

okurz requested changes Oct 23, 2025

View reviewed changes

d3flex force-pushed the feat/amqp_restart branch from 8632b55 to e5e5fd2 Compare October 23, 2025 07:44

okurz requested changes Oct 23, 2025

View reviewed changes

d3flex force-pushed the feat/amqp_restart branch from e5e5fd2 to 44e622e Compare October 23, 2025 08:12

d3flex force-pushed the feat/amqp_restart branch from 44e622e to 0b69cf5 Compare October 23, 2025 13:05

d3flex marked this pull request as ready for review October 23, 2025 13:06

d3flex removed the not-ready label Oct 23, 2025

Martchus requested changes Oct 24, 2025

View reviewed changes

t/23-amqp.t Outdated Show resolved Hide resolved

t/23-amqp.t Outdated Show resolved Hide resolved

lib/OpenQA/Schema/Result/Jobs.pm Outdated Show resolved Hide resolved

d3flex force-pushed the feat/amqp_restart branch 2 times, most recently from c3478ec to c8fe5b4 Compare October 25, 2025 04:12

okurz approved these changes Oct 25, 2025

View reviewed changes

perlpunk reviewed Oct 26, 2025

View reviewed changes

lib/OpenQA/Schema/Result/Jobs.pm Outdated Show resolved Hide resolved

Martchus reviewed Oct 27, 2025

View reviewed changes

t/20-stale-job-detection.t Outdated Show resolved Hide resolved

d3flex force-pushed the feat/amqp_restart branch 3 times, most recently from 0499fd0 to b987963 Compare November 4, 2025 09:38

Martchus requested changes Nov 4, 2025

View reviewed changes

perlpunk reviewed Nov 4, 2025

View reviewed changes

lib/OpenQA/Task/Job/Restart.pm Outdated Show resolved Hide resolved

d3flex added invalid stale labels Nov 4, 2025

d3flex added the not-ready label Nov 22, 2025

d3flex force-pushed the feat/amqp_restart branch from 7d16752 to 406afd4 Compare November 22, 2025 06:43

d3flex removed the not-ready label Nov 22, 2025

perlpunk reviewed Nov 23, 2025

View reviewed changes

lib/OpenQA/Task/Job/Restart.pm Outdated Show resolved Hide resolved

perlpunk reviewed Nov 23, 2025

View reviewed changes

t/10-jobs.t Outdated Show resolved Hide resolved

d3flex force-pushed the feat/amqp_restart branch 2 times, most recently from 6a94068 to 4d32ac5 Compare November 24, 2025 10:31

perlpunk reviewed Nov 24, 2025

View reviewed changes

t/10-jobs.t Outdated Show resolved Hide resolved

d3flex force-pushed the feat/amqp_restart branch from 4d32ac5 to 3060a83 Compare November 24, 2025 11:02

perlpunk reviewed Nov 24, 2025

View reviewed changes

d3flex force-pushed the feat/amqp_restart branch 2 times, most recently from 8e27e9a to 96463df Compare November 24, 2025 15:22

perlpunk reviewed Nov 24, 2025

View reviewed changes

d3flex force-pushed the feat/amqp_restart branch 2 times, most recently from 5298baf to bb1fc9d Compare November 24, 2025 16:54

d3flex and others added 3 commits November 24, 2025 18:31

Remove explicit loading AMQP plugin in Gru plugin

8bade67

The plugin gets loaded automatically via OpenQA::Setup

Enable gru tasks to emit AMQP messages

476bf89

Start Mojo::IOLoop and let it end when the AMQP plugin has finished (or failed) publishing the event

d3flex force-pushed the feat/amqp_restart branch 2 times, most recently from 35b4dad to 7f47099 Compare November 24, 2025 17:42

okurz approved these changes Nov 25, 2025

View reviewed changes

Martchus requested changes Nov 25, 2025

View reviewed changes

d3flex force-pushed the feat/amqp_restart branch from 7f47099 to 4300915 Compare November 25, 2025 12:20

perlpunk approved these changes Nov 25, 2025

View reviewed changes

Martchus approved these changes Nov 25, 2025

View reviewed changes

mergify bot merged commit 9eb96c2 into os-autoinst:master Nov 25, 2025
51 checks passed

		combined_like { $t->post_ok('/api/v1/jobs/99926/restart?force=1')->status_is(200) } qr/Job 99926 duplicated/,
		'Job restarted successfully';

Emit restart events when a job is enqueued #6816

Emit restart events when a job is enqueued #6816

Uh oh!

Conversation

d3flex commented Oct 22, 2025

Uh oh!

d3flex commented Oct 22, 2025

Uh oh!

codecov bot commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

perlpunk commented Oct 22, 2025

Uh oh!

okurz left a comment

Choose a reason for hiding this comment

Uh oh!

okurz left a comment

Choose a reason for hiding this comment

Uh oh!

d3flex commented Oct 23, 2025

Uh oh!

d3flex commented Oct 23, 2025

Uh oh!

perlpunk commented Oct 23, 2025

Uh oh!

d3flex commented Oct 23, 2025

Uh oh!

perlpunk commented Oct 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Martchus left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

perlpunk left a comment

Choose a reason for hiding this comment

Uh oh!

d3flex commented Nov 24, 2025

Uh oh!

d3flex commented Nov 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

codecov bot commented Oct 22, 2025 •

edited

Loading