Skip to content

Unknown/invalid phase type silently ignored — test always exits 0 #3757

Description

@tsushanth

Bug Description

In packages/core/lib/phases.ts, the async.series completion callback swallows errors silently. When a phase fails (e.g., because of an unknown mode value), the error is only debug()-logged — invisible unless DEBUG=phases is set — and then 'done' is unconditionally emitted regardless. The runner then aggregates stats normally and reports clean completion with exit code 0, giving no indication that anything went wrong.

Affected code (packages/core/lib/phases.ts, around line 95)

ee.run = () => {
  async.series(tasks, (err) => {
    if (err) {
      debug(err);
    }

    ee.emit('done');  // always fires, even on error
  });
};

Steps to Reproduce

  1. Create a script with an invalid phase mode:
config:
  target: "http://localhost:3000"
  phases:
    - duration: 10
      arrivalRate: 5
      mode: foobar   # invalid mode
scenarios:
  - flow:
      - get:
          url: "/"
  1. Run it:
artillery run script.yml
  1. Observed: exits 0, no error output (unless DEBUG=phases is set)
  2. Expected: exits non-zero with a visible error message about the unknown phase mode

Root Cause

createPhases iterates over the phase specs and pushes tasks into an array. For an unrecognised mode, it logs to console.log ("Unknown phase spec") but still returns undefined, contributing a no-op task. The async.series callback receives err only when a task itself calls back with an error — but even if it did, the current callback ignores that error path and always emits 'done'.

Suggested Fix

ee.run = () => {
  async.series(tasks, (err) => {
    if (err) {
      debug(err);
      ee.emit('error', err);
      return;
    }

    ee.emit('done');
  });
};

This ensures that any phase-level error surfaces through the event emitter so callers can handle it, log it visibly, and propagate a non-zero exit code.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions