Evaluation Criteria Inquiry

[danielan987]

Can we have further elaboration on what is expected in terms of test suite and security best practices from the evaluation criteria? Particularly if you are just using mostly Python.

[pamelafox]

For security best practices:

• Using managed identity/keyless auth instead of keys (for Azure services)
• No hard-coded keys in code, everything inside the environment variables

For testing, I don't honestly expect that most hacks will have time to add testing. I'd actually be more interested in seeing evaluation of agent quality versus unit tests, if there's time for any sort of testing.

For automated testing, I'll share my personal testing guidelines, but once again, I don't expect you to have time to add all this!

Python testing guidelines

• The repo should use either pytest or the framework-specific testing module.
• The repo should include the coverage library (typically via pytest-cov).
• The pyproject.toml file should configure pytest to check coverage for the application code, using --cov= option. See pyproject.toml
• The pyproject.toml file or CI workflow should configure coverage to fail if the coverage is below 100% threshold. See pyproject.toml, python.yaml
• A CI/CD workflow should run all the unit tests, and fail if they fail.
• Tests should use matrix to run against all supported Python versions (3.8+).
• Tests should use matrix to run against all supported OSes (Windows, Mac, Linux).

API tests

If the repo includes an API of some sort:

• Either schemathesis or hypothesis should be used to perform property-based testing of the API parameters. You may want to run those separately from unit tests for speed reasons. See property_based.py, python-check.yaml

Frontend tests

If the repo includes a frontend (HTML/CSS):

• Playwright tests should be used for end-to-end user flow tests. See playwright.py
• Playwright tests should be run in CI/CD (possibly in a different workflow from unit tests). See playwright.yaml
• Axe should be used to check for accessibility issues. Errors should either be uploaded to Code Scanning tab, displayed in the pull request, or break the workflow.

9.3.1 Load Tests

• If not using Azure Load Testing, use Locust to define the load tests.

[danielan987]

Thank you!