Docs: add llms.txt support and agent guidance

Author: xsedla1o

AGENTS.md

@@ -0,0 +1,47 @@
+# Repository Guidelines
+
+## Project Structure & Module Organization
+- `dp3/`: core Python package (processing engine, API, CLI utilities).
+- `modules/`: example app-specific modules/plugins.
+- `config/`: example/default configuration.
+- `install/` and `docker/`: deployment and container setup.
+- `tests/`: unit/integration tests (`tests/test_common`, `tests/test_api`, `tests/test_config`).
+- `docs/` and `mkdocs.yml`: documentation sources.
+
+## Build, Test, and Development Commands
+- `python -m venv venv && source venv/bin/activate`: local virtualenv.
+- `pip install --editable ".[dev]"`: install for platform development.
+- `pre-commit install`: enable formatting/linting hooks.
+- `docker compose up -d --build`: start MongoDB, RabbitMQ, Redis for local runs/tests.
+- `APP_NAME=my_app CONF_DIR=config dp3 api`: run API process.
+- `dp3 worker my_app config 0`: run a worker process.
+- `dp3 check <config_dir>`: validate configuration with detailed errors.
+
+## Documentation
+- When adding or changing user-visible features, configuration, CLI behavior, deployment flow, or extension points, update the relevant docs in `docs/` and `mkdocs.yml` as part of the same change.
+- Prefer keeping how-to guides, concept pages, and configuration reference pages aligned with code changes.
+
+## Coding Style & Naming Conventions
+- Formatting: Black with 100-char line length.
+- Linting: Ruff (with auto-fix in pre-commit).
+- Keep module/function names descriptive; prefer `snake_case` for Python identifiers.
+- Follow existing package layout under `dp3/` and test naming under `tests/`.
+
+## Testing Guidelines
+- Framework: `unittest`.
+- Common tests:  
+  `python -m unittest discover -s tests/test_common -v`
+- API tests (with config):  
+  `CONF_DIR=tests/test_config python -m unittest discover -s tests/test_api -v`
+- Add tests next to related area (`tests/test_common`, `tests/test_api`, or `tests/modules`).
+
+## Commit & Pull Request Guidelines
+- Commit messages use concise scope + summary, often `Scope: message`, e.g.  
+  `Docs: update callback registrar docstring` or `DB: fix unpacking of fulltext filters`.
+- Use `Breaking` or similar marker when removing/deprecating behavior, e.g.  
+  `API - Breaking: Removed deprecated parameter.`
+- PRs should include: summary, rationale, test evidence, linked issues/PRs when relevant, and a note on documentation updates when user-visible behavior changed.
+
+## Configuration & Services
+- Local runtime depends on MongoDB, RabbitMQ, and Redis (provided via `docker-compose.yml`).
+- Use `CONF_DIR` to point to config sets (e.g. `tests/test_config` for test runs).

docs/howto/extending.md

@@ -22,7 +22,7 @@ To see the changes made to the documentation page locally, a local instance of `
 You can install all the required packages using:
 
 ```shell
-pip install -r requirements.doc.txt
+pip install -r requirements.dev.txt
 ```
 
 After installing, you can use the following `mkdocs` commands:

dp3/template/app/AGENTS.md

@@ -0,0 +1,18 @@
+# Guidelines
+
+This is an app built on the DP3 platform; for guides, concepts, best practices
+- docs index for quick reference: https://cesnet.github.io/dp3/llms.txt
+- human-readable docs at: https://cesnet.github.io/dp3/
+
+## Common commands
+
+- Start local services: `docker compose up -d --build`
+- Run the API: `APP_NAME={{DP3_APP}} CONF_DIR=config dp3 api`
+- Run a worker: `dp3 worker {{DP3_APP}} config 0`
+- Validate configuration: `dp3 check config`
+
+## Project notes
+
+- Application configuration lives in `config/`.
+- Custom modules can be added under `modules/`.
+- When changing configuration, validate it with `dp3 check config`.

mkdocs.yml

@@ -137,6 +137,29 @@ plugins:
             merge_init_into_class: true
             show_signature_annotations: true
             separate_signature: true
+  # Generate llms.txt and llms-full.txt for LLM-friendly documentation access
+  - llmstxt:
+      markdown_description: >-
+        DP3 is a configurable data processing platform for building applications
+        around entities, attributes, modules, hooks, and processing pipelines.
+        The documentation covers getting started, extending deployments and
+        modules, architecture concepts, and configuration reference material.
+      sections:
+        Overview:
+          - index.md: Introduction to DP3
+        How-to guides:
+          - howto/*.md
+        Concepts:
+          - architecture.md
+          - data_model.md
+          - history_management.md
+          - modules.md
+          - hooks.md
+        Reference:
+          - api.md
+          - configuration/*.md
+          - scripts.md
+          - grafana_plugin.md
 
 watch:
   - dp3

requirements.dev.txt

@@ -1,6 +1,7 @@
-mkdocs>=1.0.3
+mkdocs>=1.6
 mkdocs-gen-files>=0.4
 mkdocs-literate-nav>=0.6
+mkdocs-llmstxt
 mkdocs-macros-plugin==1.5.0
 mkdocs-material==9.0.12
 mkdocs-section-index>=0.3