Mojo::IOLoop->recurring not emitting in a minion worker process after updating Mojo (8.26 to 9.22) and Minion (9.13 to 10.22)

[mmoscher]

Hi @ALL!

I'm currently trying to get my application updated to the last mojo and minion updates. Sad to say that I'm currently stuck at Mojolicious version 8.26 and Minion version 9.13. However, directly updating the core packages to the latest versions (Mojolicious 9.22 and Minion 10.22) worked like a charm despite one thing, which I'm unable to solve right now.

The application in question can be started in two modes:
a) as daemon to serve the frontend (and its logic) perl myapp.pl daemon
b) as a minion worker to process async, long-running, jobs perl myapp.pl minion worker

The issue occurs in mode b. In the callback for the emitted worker event, I'm using Mojo::IOLoop->recurring to create a small function to regularly poke some resources. This worked fine in the old versions (8.26 and 9.13). However, it does nothing after the update :( - seems to be dead at all.

To clarify and nail down the problem, I wrote a small POC script.

use Mojolicious::Lite -signatures;
use Mojolicious;
use Mojo::IOLoop;
use Minion;
use Mojo::Log;

my $log = Mojo::Log->new;
my $mojoVersion = Mojolicious->VERSION;
my $minionVersion = Minion->VERSION;

my $uri = 'postgresql://';
$uri .= $ENV{POSTGRES_USER}             if $ENV{POSTGRES_USER};
$uri .= ':' . $ENV{POSTGRES_PASSWORD}   if $ENV{POSTGRES_PASSWORD};
$uri .= '@' . $ENV{POSTGRES_HOST}       if $ENV{POSTGRES_HOST};
$uri .= '/' . $ENV{POSTGRES_DB}         if $ENV{POSTGRES_DB};

$log->info("Mojolicous: $mojoVersion - Minion: $minionVersion");

Mojo::IOLoop->recurring(5 => sub ($ioloop) {
  $log->info('[Mojo] Outer IOLoop recurring all 5 sec.');
});

app->plugin( Minion => { Pg => $uri });
app->minion->on(worker => sub ($minion, $worker) {
  $log->info('[Minion] worker started.');
  my $id = Mojo::IOLoop->recurring(1 => sub ($ioloop) {
    $log->info('[Minion] Inner IOLoop recurring every sec.');
  });
});

app->start;

It is executed a) as daemon (service mojo) and b) as minion worker (service minion) using docker-compose

---
version: "3.9"

services:
  mojo:
    image: mojo-minion-old-version:latest
    environment:
      POSTGRES_HOST: "db-old"
      POSTGRES_DB: "minion"
      POSTGRES_USER: "postgres"
      POSTGRES_PASSWORD: "postgres123"
    depends_on:
      - "db-old"
    restart: on-failure
    entrypoint:
      - perl
    command:
      - poc.pl
      - daemon

  minion:
    image: mojo-minion-old-version:latest
    environment:
      POSTGRES_HOST: "db-old"
      POSTGRES_DB: "minion"
      POSTGRES_USER: "postgres"
      POSTGRES_PASSWORD: "postgres123"
    depends_on:
      - "db-old"
    restart: on-failure
    entrypoint:
      - perl
    command:
      - poc.pl
      - minion
      - worker

  db-old:
    image: postgres:12.8-alpine
    environment:
      POSTGRES_PASSWORD: postgres123
      POSTGRES_USER: postgres
      POSTGRES_DB: minion
    expose:
      - "5432"

Running this script on the old version it works as expected and the output is as follows:

Running against the new versions, it shows that Mojo::IOLoop is not executed at all in a minion worker process.

Both container (old [1] and upgraded [2]) are running the same OS and perl version.

Currently, I've no clue what's going wrong. Reading through the Mojolicious and Minion release notes (changes.md) didn't help me either. Hope someone can help :)

Thank's in advance for your support,
Cheer's 🍺

[mmoscher]

Okay, digged into the Minion release notes once again and found this change added in tag 10.14

mojolicious/minion@5cb6241

Seem's that this will reset all IOLoops when the worker starts running? Thus, also my registered IOLoop placed in the callback?

[fskale]

This is no good practice at all.
You might want to reload minion (e.g add queues) and keep the app running !
You've to seperate the worker service from the app and the poller.

E.g.: (my HWG SMS Gateway solution)
root 94845 0.0 2.7 277152 104520 ? Ss Oct26 4:23 HWG SMS Gateway UI
root 94846 0.0 3.1 644964 118560 ? Sl Oct26 1:32 _ HWG SMS Gateway UI
root 94847 0.0 3.1 644876 117864 ? Sl Oct26 1:32 _ HWG SMS Gateway UI
root 94848 0.0 3.4 657028 129396 ? Sl Oct26 1:32 _ HWG SMS Gateway UI
root 94849 0.0 3.1 645116 118288 ? Sl Oct26 1:46 _ HWG SMS Gateway UI
root 94850 0.0 3.1 645044 118532 ? Sl Oct26 1:34 _ HWG SMS Gateway UI
root 94861 0.5 2.6 623860 101128 ? Ssl Oct26 63:44 HWG SMS Gateway Poller
root 94866 0.0 3.0 268776 114944 ? Ss Oct26 5:03 Minion HWG Worker

The minion reference is worth reading though...

[mmoscher]

Thanks for your response!
App and Worker are yet already separated and can be reloaded independently. However, some kind of "poller" is currently present in the worker code/subroutine.

How did you manage to keep the poller in the same code base or are yours separated? If you have a single one, do you use custom mojolicious commands (https://docs.mojolicious.org/Mojolicious/Commands)?

Anyway, thanks for your response. Since Mojo::IOLoop is no longer "useable" in Minion::Worker, I'll shift the poller to a seperate command.

[mmoscher]

Since mojolicious/minion@5cb6241 Mojo::IOLoop is reset in Minion::Worker->run to prevent shared timers between Mojolicious and Minion, which breaks my previous implementations (seems it only worked due to a "bug" 😄 :/)

As mentioned by @fskale, app/worker and poller should be separated.

[fskale]

Hi,
for the poller i use a seperate start script, that's all.
Using Mojolicious::Command…
Mojolicious::Commands->start_app('SMSPoller', 'daemon', '-m', 'production',
@listen_opts);