ultimate-utils (uutils)

A comprehensive Python utility library for machine learning research, spanning PyTorch training infrastructure, statistical analysis, meta-learning, LLM evaluation, and experiment management.

Built and maintained by Brando Miranda (Stanford, MIT, UIUC).

pip install ultimate-utils

import uutils
uutils.hello()

What's Inside

Module	Description
uutils	Core utilities — file I/O, argument parsing, serialization (dill/pickle/JSON), git helpers, seed management, progress bars
uutils.torch_uu	PyTorch training loops, distributed training, checkpointing, optimizers, learning rate schedulers
uutils.torch_uu.models	Model utilities and HuggingFace integrations
uutils.torch_uu.dataloaders	Dataloaders for meta-learning (miniImageNet, CIFAR-FS), standard vision, and multi-dataset sampling
uutils.torch_uu.metrics	CCA/PWCCA/DCCA similarity, model complexity, task diversity (Task2Vec), accuracy with confidence intervals
uutils.torch_uu.meta_learners	Meta-learning algorithms (MAML, Prototypical Networks, etc.)
uutils.stats_uu	Hypothesis testing, p-values, effect sizes (Cohen's d), power analysis, confidence intervals, ANOVA, regression
uutils.plot	Plotting with error bands, heatmaps, bar charts, LaTeX table export
uutils.hf_uu	HuggingFace training utilities — full fine-tuning, QLoRA/Unsloth, causal and seq2seq LM training
uutils.evals	LLM evaluation — math benchmarks (MATH, Putnam, OlympiadBench), API inference (Claude, OpenAI, vLLM), answer extraction
uutils.dspy_uu	DSPy-based synthetic data generation for in-context learning and fine-tuning
uutils.jax_uu	JAX multi-head attention, layer norm, flash attention implementations
uutils.numpy_uu	Statistical moments, confidence intervals, matrix utilities
uutils.logging_uu	Weights & Biases integration — setup, logging, sweeps, model watching
uutils.emailing	SMTP email + Stanford Outlook (AppleScript) notifications with attachments
uutils.discord_uu	One-way Discord notifications via webhooks (text, embeds, file uploads)
uutils.whatsapp_uu	One-way WhatsApp notifications via Meta Cloud API or Twilio

Notifications vs. Interactive Agents

The messaging modules above (emailing, discord_uu, whatsapp_uu) are one-way programmatic notification senders — small libraries you import from scripts, schedulers, and watchers to push alerts like "job finished", "GPU idle", or "deploy complete". They are not chat agents; there is no conversation loop, no inbound message handling, and no LLM attached.

If you want an interactive AI assistant that lives inside WhatsApp / Discord / Telegram / iMessage (reads your messages, replies, executes tasks), that is a different category of tool. Projects like OpenClaw (hosted via myclaw.ai) exist for that use case — they bundle Baileys, grammY, Discord.py, AppleScript bridges, etc., and wire them into an agent loop.

Rule of thumb:

Need	Use
Script on my cluster pings me when training finishes	uutils.emailing / discord_uu / whatsapp_uu
I want to chat with an agent from my phone and have it triage email, reply to admin tasks, run commands	OpenClaw or similar (not this library)

Today Brando's default notification channel is email (uutils.emailing); Discord and WhatsApp modules are available but optional. For the interactive-agent use case, see issue #41. The executable plan lives at experiments/01_self_hosted_openclaw/cc_prompt.md here, with the canonical home being ~/agents-config/experiments/01_self_hosted_openclaw/cc_prompt.md (mirror it back into agents-config when convenient).

Installation

Note: PyTorch must be installed separately (with CUDA if you need GPU support).

Development install (recommended)

conda create -n uutils python=3.11 -y
conda activate uutils
pip install -e ~/ultimate-utils

pip install from PyPI

pip install ultimate-utils

Verify installation

python -c "import uutils; uutils.hello()"
python -c "import uutils; uutils.torch_uu.gpu_test_torch_any_device()"

Quick Examples

Seed everything for reproducibility

from uutils import seed_everything
seed_everything(42)

Save and load with dill

from uutils import save_with_dill, load_with_dill
save_with_dill(my_object, '~/data/my_object.pkl')
obj = load_with_dill('~/data/my_object.pkl')

Plot with error bands

from uutils.plot import plot_with_error_bands
plot_with_error_bands(x, y, yerr, xlabel='Steps', ylabel='Loss', title='Training Loss')

Statistical testing with effect size

from uutils.stats_uu.effect_size import stat_test_with_effect_size_as_emphasis
stat_test_with_effect_size_as_emphasis(group1_data, group2_data)

W&B logging

from uutils.logging_uu.wandb_logging.common import setup_wandb, log_2_wandb
setup_wandb(args)
log_2_wandb(metrics_dict, step=step)

Publishing to PyPI

# Bump version in setup.py, then:
cd ~/ultimate-utils && bash scripts/publish_to_pypi.sh

Citation

If you use ultimate-utils in your research, please cite:

@software{miranda2024uutils,
  author       = {Brando Miranda},
  title        = {ultimate-utils: A Comprehensive Utility Library for Machine Learning Research},
  year         = {2024},
  publisher    = {PyPI},
  url          = {https://github.com/brando90/ultimate-utils},
  note         = {Available at \url{https://pypi.org/project/ultimate-utils/}}
}

You can also find the author's publications on Google Scholar.

Related Publications

This library has supported research in the following publications (among others):

• Beyond Scale: the Diversity Coefficient as a Data Quality Metric Demonstrates LLMs are Not Yet More Human than Human
• Does MAML Only Work via Feature Re-use? A Data Set Centric Perspective

License

Apache-2.0