What happens in CI?

[ggreif]

The continuous integration on GitHub repos dfinity/motoko is performing GitHub actions on every commit. At the moment there are 4 builders, which cover {linux, darwin} × {x64, arm64}.

The actions are

• .github/workflows/test.yml
• .github/workflows/update-hash.yml

Build pipeline

The test.yml action defines a sequence of steps that are required to execute without error. These are

• checking out the repo with the PR's current commit
• removing some junk form the builder
• installing nix and cachix
• building the attribute set with the tests (transitively builds if not found in the binary cache)
• calculating and uploading performance diffs as a PR comment

All tests are recorded in the nix store and synced to cachix.

The test/ directory

This contains the test files which are run themselves or just through some tool, capturing their textual outputs. Then this gets compared to the golden versions (.ok files).

• test/fail = moc typechecker only
• test/idl = moc compilation only, ingesting .did files
• test/mo-idl = moc compilation only, generating .did files
• test/run = (regular + EOP) × (release + debug) // simple wasmtime tests
• test/drun = (4 × GC + EOP) × (release + debug) // canister tests that need a replica
• test/trap = moc compilation and simple wasmtime tests, checking for intended traps
• test/ld = (regular + EOP) // linking tests
• test/repl = moc interactive mode
• test/perf = dApp benchmarks
• test/bench = codegen (micro-)benchmarks
• test/viper = moc with model-checker output and verification
• test/run-deser = binary Candid deserialiser

Build results

Following build artefacts are created

  /nix/store/cnra7vwcjdm0lhy6yjzhjh0xl62qawvc-test-bench  # only on specific builders
  /nix/store/3c4azcsbz501ngswcpi9lwnvzrhmg458-test-mo-idl-eop.drv
  /nix/store/66nm1yshbrywq3kl9djinimnhfgqs7js-test-drun-compacting-gc.drv
  /nix/store/7lqgm6qfjqwxbpwf6kx7k49b8bvp00ql-test-drun-dbg.drv
  /nix/store/88qlfz968xmfy60i8nzyw13bldafxlmh-test-idl.drv
  /nix/store/9zyisix6lwg20rsbnwkjdwrzvr4n62al-test-run-deser.drv
  /nix/store/cl3k8bnn72k8mm121mh6kzl23xhkzwx2-test-drun-eop-debug.drv
  /nix/store/d31fyxwfw2lgzxa7803c8j3vq500icf8-test-run-eop-debug.drv
  /nix/store/d8mcbr8ds5vf0l0yr4mm4fv2y55gpbmy-test-drun-generational-gc.drv
  /nix/store/fdnlcy5q37aradq6x5f5pbda4r57vczc-test-trap-eop.drv
  /nix/store/g46pwn1k09n2n1mm0m9vz68jxaw4w9p5-test-ld.drv
  /nix/store/hil22l0f140dphfjfzdzmy4wvyqx6kw4-test-fail.drv
  /nix/store/lgp8ivhmcgy3dnqq1xwd77xn443ybj18-test-drun-incremental-gc.drv
  /nix/store/m7aw3k1dn9wrx933bgwnf35ww2yh8a2h-test-run-dbg.drv
  /nix/store/m9rlff6jfhjrh40fkrfrd7d69w77gzv7-test-viper.drv
  /nix/store/p8v8drz7wjknafyjbjlv56dchdcd4gkv-test-drun.drv
  /nix/store/phyaz0fqzk8n9b27b70qgnhxm159qhq8-test-run.drv
  /nix/store/qvyq059sc0rnpvrlxqjyzw51j4zbssff-test-perf.drv
  /nix/store/qzrhafywxapxx3k0b9q2rrmia724bn2v-test-trap.drv
  /nix/store/r97a9adw1y82m0xlavfdr391qql9jvb5-test-repl.drv
  /nix/store/s8wlalqam64az9c2jrg5sw3jka3gzava-test-ld-eop.drv
  /nix/store/w7122gl0i4c3xd42im0pfp8q90d8wmfb-test-run-eop-release.drv
  /nix/store/x55nlkc1blz0m5qn5f4hyh60fb7fdc6l-test-drun-eop-release.drv
  /nix/store/zzvhj7ydi9bdpd3xzyjlx0w7zh9igmai-test-mo-idl.drv

test-coverage

This is a special target, which collects all the .mo files in the test/* directories and recompiles them using an instrumented version of the moc compiler (also the other tools?). This way a line-wise coverage map is generated and a visualisation is uploaded to the GH site.

test-unit

This target runs unit tests distilled from OCaml code.

check-error-codes

This target verifies that all error codes mentioned in src/lang_utils/error_codes.ml actually appear in some compiler diagnostic.

Other testing

We also test some other ways:

• property tests (test/random) for evaluating Motoko expressions
• test/lsp-int for testing the OCaml LSP executable