tests: Optimize CI test matrix to reduce workflow duration

[baconYao]

This PR restructures the CI test matrix to significantly reduce overall workflow duration while maintaining full test coverage and improving maintainability.

resolve: #2628

Changes Introduced

1. Introduced test-category split to divide test cases into logical groups:

   • core, storage, network, security, misc

   This reduces the number of tests executed per job, allowing them to complete faster.

2. Standardized full test runs to use the dir backend.

   • Ensures full functionality coverage with minimal overhead.

3. Storage backend tests are now isolated to:

   • suite: standalone

     • category: storage
     • backends: [btrfs, lvm, zfs, ceph, linstor, random]

   • suite: cluster

     • category: all
     • backends: [btrfs, lvm, zfs, ceph, linstor, random]

   This ensures backend behavior is validated without impacting other categories.

4. Ensured both ubuntu-24.04 and ubuntu-24.04-arm continue to be supported.

[stgraber]

Thanks a lot for the work!

I'll try to look more into this later today or tomorrow. I'll most likely be doing a few minor edits to the commit, splitting things up into distinct commits and trying a few tweaks to the Github workflow config to try and make it more readable.

[bensmrs]

Hi! Where has test_container_devices_disk gone? (and other devices, FWIW)

[baconYao]

Hi! Where has test_container_devices_disk gone? (and other devices, FWIW)

Hi @bensmrs

Thank you for helping identify the missing test cases — I think I really did overlook some of them.

Since I'm not very familiar with the overall Incus ecosystem, I proposed this PoC. I believe the most important part is categorizing the test cases. For now, I’ve only roughly grouped them into core, storage, network, security, and misc. However, I think these categories can be further refined, so I’d appreciate suggestions and ideas from everyone with more experience.

[bensmrs]

Well I don’t think it needs to be any more refined than that, so you just have to properly categorize the missing tests. However, container_devices_disk.sh definitely needs to be tested for each storage driver.

[baconYao]

Hi @stgraber, may I know is there any suggestion about this PR that I can improve? Thanks!

[stgraber]

@baconYao the current version appears to break all of the standalone, all tests.

But don't worry too much about that, I was going to track it down, patch up the PR and merge it. I've just been incredibly busy for the past week so haven't had time to do it yet, sorry.