This repository is an AI research codebase for a micro-GPT and deep-learning visualization laboratory. Treat it as source code for a serious research paper: claims must be reproducible, implementation details must be explicit, and visualizations must explain real learning dynamics rather than decorative diagrams.
- Do not present a model as trained, benchmarked, or state of the art unless the exact command, config, dataset, hardware, seed, commit, and metric artifact exist in the repo.
- Prefer small deterministic demonstrations, dry runs, and unit tests over long-running experiments unless the user explicitly asks to train.
- Keep the from-scratch path separate from Hugging Face baselines. The from-scratch path uses PyTorch tensor primitives; the Hugging Face path is for comparison, dataset tooling, TRL recipes, and future managed jobs.
- Every new algorithm module should expose inspectable intermediate values for visualization: activations, gradients, losses, attention maps, policy/value maps, or token probabilities.
- When adding research claims, cite papers or official documentation in
docs/literature_review.mdor the relevant document.
- Build algorithms from first principles using tensor operations before reaching for high-level trainer abstractions.
- Keep long training disabled by default. New training entrypoints must support a dry-run mode that completes quickly on CPU.
- Add tests for mathematical behavior before implementation: gradients, shapes, masks, return calculations, optimizer updates, and deterministic config parsing.
- Preserve the existing GPT-2 summarization fine-tuning path as a baseline unless the user explicitly asks to remove it.
- Avoid notebook-only logic. Notebooks may be added later, but the canonical implementation belongs in importable Python modules.
- The local web app should visualize actual tensors generated by repository code.
- Prefer compact interactive views that expose knobs researchers care about: learning rate, optimizer, depth, activation, sequence length, discount factor, context length, and sampling parameters.
- Do not use visualizations as screenshots of static theory. Each visualization should be tied to an executable demo, dry run, or unit-tested primitive.
Run the lightweight checks before committing:
./venv/bin/python -m unittest
./venv/bin/python -m py_compile main.py src/prepare_data.py src/finetune_model.py src/algorithms/*.py src/micro_gpt/*.py src/research_lab/*.py
./venv/bin/python -m src.micro_gpt.train --config configs/micro_gpt/tiny_debug.json --dry-run
git diff --checkIf Streamlit dependencies are installed, also smoke-test the app:
./venv/bin/python -m streamlit run src/research_lab/app.py --server.headless trueDo not launch full model training or Hugging Face Jobs without explicit user approval.