[Feat]: Align CI with pyproject extras, skill tests, and manifest dependencies

[rosspeili]

Feature Description

GitHub Actions CI (.github/workflows/ci.yml) has drifted from how Skillware is packaged and tested today. It still carries a stale “no setup.py yet” comment, redundant manual pip install lines, and only runs pytest tests/ — while docs/TESTING.md tells contributors to also run pytest skills/.

Recent skill PRs (e.g. #150) had to hardcode a skill test path and add web3 manually to CI. That pattern does not scale as the registry grows.

Goal: one maintainable CI layout that:

1. Installs dependencies from pyproject.toml (single source of truth), not ad hoc pip pins per PR
2. Runs framework tests and skill co-located tests in clear separate steps (or jobs)
3. Ensures skills with manifest.yaml requirements can be loaded and tested in CI without editing the workflow for each new skill
4. Stays simple and reliable — no flaky PR CI, no over-engineering

Current state (baseline)

Area	Today
CI install	pip install flake8 pytest pytest-mock + requirements.txt (-e ".[dev,all]") + manual pymupdf anthropic requests
CI tests	pytest tests/ only
Docs	TESTING.md step 5: pytest skills/
Co-located skill tests	skills/compliance/tos_evaluator/test_skill.py, skills/dev_tools/issue_resolver/test_skill.py, skills/data_engineering/novelty_extractor/test_skill.py — not run in CI
Framework skill tests	tests/skills/** — run in CI via tests/
Manifest deps not in [all]	e.g. fastembed, numpy (novelty_extractor); future web3 (defi)
Issuer/manifest checks	tests/test_skill_issuer.py in tests/ — already in CI

Problems to fix

• Per-PR edits to CI for each new skill dependency or test path
• Redundant installs (pymupdf, anthropic already in [all])
• Skill co-located tests invisible to CI unless duplicated under tests/
• manifest.yaml requirements and pyproject.toml optional extras can disagree
• requirements.txt duplicates/confuses the story (dev pointer vs CI truth)

Proposed direction (keep it simple)

A. Dependency install (one step, no per-skill pip lines)

• CI should use pip install -e ".[dev,all]" explicitly (or keep requirements.txt only if it remains a thin pointer to that — not a second dependency list)
• Extend [project.optional-dependencies] so [all] (or a new [ci] extra) includes all registry skill deps needed for CI today:
  • Existing: rich, SDKs, pymupdf, …
  • Missing today: fastembed, numpy, and future groups like web3>=6.0.0 under [defi] merged into [all] or [ci]
• Convention: when a skill adds requirements: in manifest, the same packages must appear in pyproject optional extras (document in CONTRIBUTING)
• Remove stale CI comment about missing setup.py; update comment to point at pyproject.toml

Avoid unless necessary: a dynamic “read all manifests and pip install” script in CI (nice later; not required for v1 if [all]/[ci] is kept current)

B. Two test steps (no hardcoded skill paths)

Step 1 — Framework / universal tests

pytest tests/

Covers loader, CLI, issuer rules, tests/skills/** integration tests. No per-skill paths.

Step 2 — Skill co-located tests

pytest skills/ --ignore=skills/**/__pycache__

Or equivalent glob: discovers every skills/**/test_skill.py automatically. No skills/defi/evm_tx_handler/test_skill.py hardcoding.

Optional: skip step 2 if no test_skill.py files exist (unlikely long term).

C. Env vars for CI

Keep dummy keys where skills expect them (ANTHROPIC_API_KEY, ETHERSCAN_API_KEY). Add placeholders for new skills as needed (e.g. AGENT_WALLET_PRIVATE_KEY for defi mocked tests) — generic dummy values in workflow env, not per-skill branches.

D. Docs sync

Update docs/TESTING.md and CONTRIBUTING CI section to match the two-step model. Clarify:

• Framework tests → tests/
• Skill bundle tests → skills/<category>/<name>/test_skill.py
• Adding a skill with external deps → update pyproject extras + manifest requirements

E. Out of scope (this issue)

• uv / lockfiles (separate RFC)
• Black in CI (optional follow-up; flake8-only is fine for now)
• Running every example script under examples/
• Live RPC / mainnet smoke tests

Acceptance criteria

• CI installs deps from pyproject only (no redundant manual package list except build tools if needed)
• [all] or [ci] includes deps for all skills that have co-located tests or manifest requirements not covered by core
• CI runs pytest tests/ and pytest skills/ as separate steps
• No hardcoded skill test paths in .github/workflows/ci.yml
• Co-located tests for existing skills (tos_evaluator, issue_resolver, novelty_extractor) pass in CI
• tests/test_skill_issuer.py still passes (manifest/issuer contract)
• TESTING.md / CONTRIBUTING updated to match
• New skill PRs do not need CI workflow edits for standard test + dep patterns

Related

• feat(defi): add evm_tx_handler PR1 (quote, preview, transfer) #150 — defi/evm_tx_handler (hold CI changes out until this lands)
• Feat]: Replace mandatory SDK dependencies with optional extras and remove requirements.txt #87 — optional extras model
• docs/TESTING.md pre-commit checklist

Rationale

CI is the contract between contributors and maintainers. Right now it passes for framework work but under-tests skills and invites per-PR workflow hacks. That creates review friction (every skill PR debates CI lines) and false confidence — docs say run pytest skills/, CI does not.

Centralizing deps in pyproject.toml and using pytest skills/ as a glob keeps CI stable as the registry grows, without the complexity of a custom dependency resolver or multi-job matrices per category.

We want this before or alongside merging large skills (e.g. #142 / #150) so contributors are not blocked on workflow edits.

Implementation Idea

Phase 1 — pyproject

Add optional groups and extend [all]:

[project.optional-dependencies]
defi = ["web3>=6.0.0"]
skills-embeddings = ["fastembed", "numpy"]  # or fold into all
ci = [...]  # optional alias: everything needed for pytest tests/ + pytest skills/
all = [..., "web3>=6.0.0", "fastembed", "numpy"]

Document in CONTRIBUTING: new skill with requirements: → update [all] (or relevant extra).

Phase 2 — ci.yml

- name: Install package and CI deps
  run: pip install -e ".[dev,all]"

- name: Framework tests
  run: pytest tests/
  env: { ... dummy keys ... }

- name: Skill co-located tests
  run: pytest skills/
  env: { ... same dummy keys ... }

Remove manual pip install pymupdf anthropic requests ... block and stale comment.

Phase 3 — verify locally

pip install -e ".[dev,all]"
pytest tests/
pytest skills/
flake8 .

Phase 4 — docs

One paragraph in TESTING.md + CONTRIBUTING linking manifest requirements ↔ pyproject extras ↔ CI.

Optional later: small script scripts/collect_manifest_requirements.py to fail CI if manifest deps are missing from [all] — only if manual [all] upkeep becomes painful.

[rosspeili]

Maintainer notes — scope boundaries and follow-ups for this issue.

In scope for this issue

• Single dependency source via pyproject.toml ([dev,all] or a dedicated [ci] extra)
• Two CI test steps: pytest tests/ then pytest skills/ (glob, no per-skill paths)
• Remove redundant manual pip install lines and the stale setup.py comment
• Sync docs/TESTING.md and CONTRIBUTING with what CI actually runs
• Convention: new skill manifest.yaml requirements must be reflected in pyproject optional extras

Gaps to keep in mind (not blockers for v1)

Two test homes today

We already have skill coverage in two places:

• tests/skills/** — framework integration tests (loader, execute paths)
• skills/**/test_skill.py — co-located bundle tests

Both are valid. This issue should run both layers (tests/ and skills/), not merge them into one directory.

Skills without co-located tests

Some registry skills only have coverage under tests/skills/ or no dedicated test file yet. That is acceptable for now. Optional policy later: skills with non-empty manifest.yaml requirements should ship a co-located test_skill.py.

Dummy env vars in CI

When pytest skills/ runs, some skills expect env at import or execute time. Keep generic dummy values in the workflow env block (e.g. ANTHROPIC_API_KEY, ETHERSCAN_API_KEY, and for defi AGENT_WALLET_PRIVATE_KEY with a fake hex key for mocked tests). No per-skill branches in the workflow.

Issuer / manifest contract

tests/test_skill_issuer.py already runs inside pytest tests/. No change needed — just ensure it keeps passing after dependency refactors.

Incoming skill PRs (#150 / #142)

Hold per-skill CI edits on open skill PRs until this lands. Contributors should not add hardcoded test paths or manual pip lines; we merge CI here once, then skill PRs only touch pyproject extras + manifest if they add deps.

Optional later (only if [all] upkeep gets painful)

A small check script that compares manifest.yaml requirements across skills/ against pyproject extras and fails if something is missing. Not required for the first pass if we document the manual convention clearly.

Follow-up issues (separate from this one)

Follow-up	Why separate
[Feat]: Add Black to CI	Docs already recommend Black; CI only runs flake8 today. Decide whether to gate on black --check or keep format as contributor-only.
[Docs]: Require co-located tests for skills with external deps	Policy change in CONTRIBUTING / skill proposal template — not a CI wiring change.
[Feat]: scripts/check_manifest_deps.py (manifest vs pyproject drift)	Only worth it if manual [all] maintenance becomes error-prone.
[Feat]: CI dummy env registry	Single doc or small module listing required dummy env vars per skill category for local dev and Actions — optional polish after the two-step CI lands.

Success check

After this issue merges, a new skill PR should only need:

1. Skill bundle + co-located tests (if applicable)
2. manifest.yaml requirements
3. Matching entry in pyproject optional extras / [all]

No edits to .github/workflows/ci.yml for a standard skill.

[rosspeili]

Scope adjustment for this fix

The original issue proposed pytest skills/ in CI alongside pytest tests/. After maintainer review we are keeping CI on pytest tests/ only — same effective behavior as today, but with a clean install path.

Why not pytest skills/ in CI

• Co-located skills/**/test_skill.py stays a local pre-PR step for skill authors (already documented in CONTRIBUTING).
• Avoids per-skill pip install lines in the workflow as the registry grows (web3, fastembed, etc.).
• Framework coverage remains in tests/ including tests/skills/** integration tests and test_skill_issuer.py.
• New skill PRs (e.g. feat(defi): add evm_tx_handler PR1 (quote, preview, transfer) #150) do not need CI workflow edits.

What this PR still delivers from #151

• Single dependency source: pip install -e ".[dev,all]" from pyproject.toml
• Removed redundant manual pip pins and stale comments
• Docs (TESTING.md, CONTRIBUTING.md) aligned with CI vs local skill tests
• Black excluded for now (16 files would fail black --check); format pass + CI gate is a follow-up

Not in scope

• Extending [all] with every manifest requirements entry (skill deps stay in manifest; authors install locally for skill tests)
• uv / lockfiles