Error message: Active version 7 is greater than the latest version 0

[yahermann]

Hello there,

I'm new to Mojolicious and I need a bit of help. Thanks in advance! Here's the issue.

We currently have a large and busy application running on Dancer(1). In order to process background tasks, we adopted Minion. This was done intentionally as we desire to eventually migrate the app to Mojolicious.

Minion has been wonderfully running minimal background tasks for the past several months. We recently decided to up the ante and give more work to Minion. The additional work is processing incoming webhooks.

After extensive testing, we put the upgrade into production, and it worked great for several days. Then, last night, without any updates whatsoever, we started to intermittently see the error Active version 7 is greater than the latest version 0 at /opt/perl-5.28.1/lib/site_perl/5.28.1/Mojo/mysql/Migrations.pm line 106. By intermittently, I would estimate 1 failure out 100 enqueues. More complete stack trace below.

Any ideas what's going on, and how to fix it? It's blowing my mind that we have made no changes whatsoever to our production configuration, or at least that we know of.

Thanks!

Active version 7 is greater than the latest version 0 at /opt/perl-5.28.1/lib/site_perl/5.28.1/Mojo/mysql/Migrations.pm line 106.
Mojo::mysql::Migrations::migrate(Mojo::mysql::Migrations=HASH(0x55ff95d88dc0)) called at /opt/perl-5.28.1/lib/site_perl/5.28.1/Minion/Backend/mysql.pm line 344
Minion::Backend::mysql::__ANON__(Mojo::mysql=HASH(0x55ff95da2ac8), DBI::db=HASH(0x55ff9754d848)) called at /opt/perl-5.28.1/lib/site_perl/5.28.1/Mojo/EventEmitter.pm line 35
Mojo::EventEmitter::__ANON__(Mojo::mysql=HASH(0x55ff95da2ac8), DBI::db=HASH(0x55ff9754d848)) called at /opt/perl-5.28.1/lib/site_perl/5.28.1/Mojo/EventEmitter.pm line 15
Mojo::EventEmitter::emit(Mojo::mysql=HASH(0x55ff95da2ac8), "connection", DBI::db=HASH(0x55ff9754d848)) called at /opt/perl-5.28.1/lib/site_perl/5.28.1/Mojo/mysql.pm line 137
Mojo::mysql::_dequeue(Mojo::mysql=HASH(0x55ff95da2ac8)) called at /opt/perl-5.28.1/lib/site_perl/5.28.1/Mojo/mysql.pm line 59
Mojo::mysql::db(Mojo::mysql=HASH(0x55ff95da2ac8)) called at /opt/perl-5.28.1/lib/site_perl/5.28.1/Minion/Backend/mysql.pm line 81
Minion::Backend::mysql::enqueue(Minion::Backend::mysql=HASH(0x55ff997a52a0), "message_svc_process_webhook_postmark", ARRAY(0x55ff9afccb80), HASH(0x55ff95d8ffa8)) called at /opt/perl-5.28.1/lib/site_perl/5.28.1/Minion.pm line 48
Minion::enqueue(Minion=HASH(0x55ff9ad2a678), "message_svc_process_webhook_postmark", ARRAY(0x55ff9afccb80), HASH(0x55ff95d8ffa8)) called at /home/app/lib/Local/Model/Svc/Minion.pm line 137

[kraih]

That error makes absolutely no sense to me. The migration is only checked the first time the database is accessed from a process, usually during application/worker startup. Of course i have no idea how your Dancer app is deployed, and i do not maintain the MySQL backend.

Given those constraints, my best guess would be to try $minion->backend->mysql->auto_migrate(0) with the Minion instance on the Dancer side. Your version upgrades will be a bit less seamless, but having migrations only run on the worker side should usually be good enough.

[yahermann]

Thanks so much for your quick and thoughtful response.

Here is code showing the basic flow of starting the worker and queuing jobs. I've removed all our app-specific implementation details for readability. We've wrapped Minion in a Moose class, and now I'm wondering if the problem might be multiple instances floating around and colliding into each other somehow. The two Dancer apps, shown toward the end, run separately.

This snippet may also be useful for anyone else implementing Minion within Moose & Dancer.

package Local::Model::Svc::Minion;
use Moose;
use namespace::autoclean;

use Try::Tiny;
use Minion;
use Minion::Backend::mysql;


has 'runner' => (    
  is => 'ro',
  isa => 'Minion',
  lazy => 1,
  builder => '_build_runner',
  init_arg => undef,
);


sub worker_run {
  my $self = shift;

  # start runner, repair if needed
  my $minion = $self->runner->repair;    ### is this a problem?  is Minion cloned?

  #
  # $minion->backend->mysql->auto_migrate(0);  ### add here??

  # register tasks
  $minion->add_task( ... )
  $minion->add_task( ... )

  my $worker = $minion->worker;
  ...
  $worker->run;  #forever
}


sub queue_add {
  my ( $self, $task_name, $task_args ) = @_;

  return $self->runner->enqueue( 
    $task_name => $task_args,
    {
      notes => { ... },
      queue => 'default',
    }
  );
}


sub _build_runner {
    my $self = shift;

  $ENV{ MOJO_PUBSUB_EXPERIMENTAL } = 1;  

  return Minion->new(
    mysql => 'mysql://...' 
  );
}


#######################################################################
# dancer app - to run background task
#
package main;
my $minion_svc = Local::Model::Svc::Minion->new();
$minion_svc->worker_run;



#######################################################################
# dancer app - foreground
#
package main;

...
my $job_id = $minion_svc->queue_add( 'some_task', { some => 'param' } );

[kraih]

You add the auto_migrate line to _build_runner, right after instantiating. And since you use the same constructor for both, you do the opposite ->auto_migrate(1) in worker_run (so it still migrates when needed on the worker side).

[kraih]

I'm assuming the error only happens inside your Dancer app. That means the Moose abstraction is fine, since the worker uses it too. And if i had to guess it's probably something about the server your deploy your app with that results in the migrations being cleared from the Mojo::mysql instance.

[kraih]

This is probably a Dancer/PSGI setup or Mojo::mysql specific problem.

[yahermann]

Hi Sebastian, thanks for the quick reply!

To add more to the mystery, the error messages suddenly stopped -- without any changes or updates whatsoever! We accumulated 53 enqueue exceptions (out of 1000+ enqueue attempts, most successful) over an 8-hour period.

At this point I have to think it's some sort of action-at-a-very-very-very-far-distance intermittent problem, maybe a setup or a memory leak or something along those lines. We will just let it be for now, and if the problem surfaces again, we'll immediately add the auto_migrate() you suggested and see if that changes anything.

Thanks again for looking into this!