feat: publish optimization SGD and optimizer-zoo notebooks

From-scratch gradient descent / SGD notebook and the momentum-to-Adam optimizer
zoo, both executed in torch.dev.gpu, plus registry entries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: pantelis

notebooks/notebook-database.yml

@@ -533,3 +533,17 @@ notebooks:
   description: SGD on the regularized ridge loss for degree-9 polynomial regression, lambda anchored to the closed-form optimum
   last_executed: 2026-05-30
   duration_seconds: 17.7
+- source: aiml-common/lectures/optimization/sgd/index.ipynb
+  notebook: optimization/sgd/index.ipynb
+  code_cells: 10
+  environment: torch.dev.gpu
+  description: Gradient descent, SGD and mini-batch from scratch in NumPy with a PyTorch coda
+  last_executed: 2026-05-30
+  duration_seconds: 5.4
+- source: aiml-common/lectures/optimization/optimizers/index.ipynb
+  notebook: optimization/optimizers/index.ipynb
+  code_cells: 8
+  environment: torch.dev.gpu
+  description: Momentum, Nesterov, RMSProp and Adam from scratch on ravine and saddle landscapes, with a PyTorch coda
+  last_executed: 2026-05-31
+  duration_seconds: 4.8

notebooks/optimization/optimizers/index.ipynb

@@ -0,0 +1,297 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Plain gradient descent, the subject of the [previous notebook](/aiml-common/lectures/optimization/sgd), struggles on two landscape features that are everywhere in deep learning. In a **ravine**, where the loss curves far more sharply in one direction than another, it zigzags across the steep walls while creeping along the shallow floor. Near a **saddle point**, where the gradient nearly vanishes, it stalls. This notebook builds the standard fixes from scratch, momentum, Nesterov, RMSProp, and Adam, compares their trajectories on both landscapes, and then shows the same optimizers through `torch.optim`. Throughout, $\\boldsymbol{\\theta}$ is the parameter vector and $\\mathbf{g} = \\nabla_{\\boldsymbol{\\theta}} L$ is the gradient."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {
+    "tags": [
+     "remove-cell"
+    ]
+   },
+   "execution_count": null,
+   "source": [
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "import seaborn as sns\n",
+    "sns.set_theme()"
+   ],
+   "outputs": []
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## A ravine\n",
+    "\n",
+    "Take an anisotropic quadratic that is steep in one coordinate and shallow in the other,\n",
+    "\n",
+    "$$L(\\boldsymbol{\\theta}) = \\tfrac{1}{2}\\big(\\theta_0^2 + \\kappa\\,\\theta_1^2\\big), \\qquad \\nabla_{\\boldsymbol{\\theta}} L = (\\theta_0,\\; \\kappa\\,\\theta_1).$$\n",
+    "\n",
+    "The Hessian has eigenvalues $1$ and $\\kappa$, so the **condition number** is $\\kappa$. Gradient descent is stable only while $\\eta < 2/\\kappa$, set by the steep direction, but then the shallow direction contracts by just $(1 - \\eta)$ per step. With $\\kappa$ large the steep coordinate oscillates while the shallow one barely moves: the familiar zigzag down a narrow valley."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "execution_count": null,
+   "source": [
+    "KAPPA = 100.0\n",
+    "def L(theta):\n",
+    "    return 0.5 * (theta[0]**2 + KAPPA * theta[1]**2)\n",
+    "def grad(theta):\n",
+    "    return np.array([theta[0], KAPPA * theta[1]])\n",
+    "\n",
+    "start = np.array([-9.0, -1.0])     # common starting point; minimum is at the origin\n",
+    "print(f\"condition number kappa = {KAPPA:.0f}\")"
+   ],
+   "outputs": []
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Momentum\n",
+    "\n",
+    "Momentum accumulates a running **velocity** $\\mathbf{v}$ that averages successive gradients. Oscillating components (the steep direction) cancel, while the consistent component (the shallow floor) builds up, so the iterate accelerates along the valley instead of bouncing across it:\n",
+    "\n",
+    "$$\\mathbf{v}_t = \\mu\\,\\mathbf{v}_{t-1} + \\mathbf{g}_t, \\qquad \\boldsymbol{\\theta}_t = \\boldsymbol{\\theta}_{t-1} - \\eta\\,\\mathbf{v}_t,$$\n",
+    "\n",
+    "with momentum coefficient $\\mu \\in [0,1)$. **Nesterov** momentum evaluates the gradient at a look-ahead point $\\boldsymbol{\\theta} + \\mu\\mathbf{v}$, which corrects overshoot and usually converges a little faster."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Adaptive methods: RMSProp and Adam\n",
+    "\n",
+    "A different fix rescales each coordinate by its own recent gradient magnitude, so steep directions take smaller steps and shallow directions larger ones automatically. **RMSProp** keeps an exponential average of squared gradients $\\mathbf{s}$ and divides by its root,\n",
+    "\n",
+    "$$\\mathbf{s}_t = \\rho\\,\\mathbf{s}_{t-1} + (1-\\rho)\\,\\mathbf{g}_t^2, \\qquad \\boldsymbol{\\theta}_t = \\boldsymbol{\\theta}_{t-1} - \\frac{\\eta}{\\sqrt{\\mathbf{s}_t} + \\epsilon}\\,\\mathbf{g}_t.$$\n",
+    "\n",
+    "**Adam** combines this with momentum, tracking averages of both the gradient ($\\mathbf{m}$) and its square ($\\mathbf{v}$), each bias-corrected,\n",
+    "\n",
+    "$$\\mathbf{m}_t = \\beta_1\\mathbf{m}_{t-1} + (1-\\beta_1)\\mathbf{g}_t, \\quad \\mathbf{v}_t = \\beta_2\\mathbf{v}_{t-1} + (1-\\beta_2)\\mathbf{g}_t^2, \\quad \\boldsymbol{\\theta}_t = \\boldsymbol{\\theta}_{t-1} - \\eta\\,\\frac{\\hat{\\mathbf{m}}_t}{\\sqrt{\\hat{\\mathbf{v}}_t} + \\epsilon}.$$\n",
+    "\n",
+    "Each optimizer below is a small step function that reads and updates its own state; a shared runner iterates it and records the path."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "execution_count": null,
+   "source": [
+    "def run(step, grad, theta0, n=120):\n",
+    "    theta, state, path = np.array(theta0, float), {}, [np.array(theta0, float)]\n",
+    "    for t in range(1, n + 1):\n",
+    "        theta = step(theta, grad, state, t)\n",
+    "        path.append(theta.copy())\n",
+    "    return np.array(path)\n",
+    "\n",
+    "def sgd(lr):\n",
+    "    def step(th, grad, s, t):\n",
+    "        return th - lr * grad(th)\n",
+    "    return step\n",
+    "\n",
+    "def momentum(lr, mu=0.9):\n",
+    "    def step(th, grad, s, t):\n",
+    "        s[\"v\"] = mu * s.get(\"v\", 0.0) + grad(th)\n",
+    "        return th - lr * s[\"v\"]\n",
+    "    return step\n",
+    "\n",
+    "def nesterov(lr, mu=0.9):\n",
+    "    def step(th, grad, s, t):\n",
+    "        v = s.get(\"v\", np.zeros_like(th))\n",
+    "        v = mu * v - lr * grad(th + mu * v)\n",
+    "        s[\"v\"] = v\n",
+    "        return th + v\n",
+    "    return step\n",
+    "\n",
+    "def rmsprop(lr, rho=0.99, eps=1e-8):\n",
+    "    def step(th, grad, s, t):\n",
+    "        g = grad(th)\n",
+    "        s[\"s\"] = rho * s.get(\"s\", 0.0) + (1 - rho) * g**2\n",
+    "        return th - lr * g / (np.sqrt(s[\"s\"]) + eps)\n",
+    "    return step\n",
+    "\n",
+    "def adam(lr, b1=0.9, b2=0.999, eps=1e-8):\n",
+    "    def step(th, grad, s, t):\n",
+    "        g = grad(th)\n",
+    "        s[\"m\"] = b1 * s.get(\"m\", 0.0) + (1 - b1) * g\n",
+    "        s[\"v\"] = b2 * s.get(\"v\", 0.0) + (1 - b2) * g**2\n",
+    "        mhat, vhat = s[\"m\"] / (1 - b1**t), s[\"v\"] / (1 - b2**t)\n",
+    "        return th - lr * mhat / (np.sqrt(vhat) + eps)\n",
+    "    return step"
+   ],
+   "outputs": []
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "execution_count": null,
+   "source": [
+    "paths = {\n",
+    "    \"SGD\":      run(sgd(0.018),      grad, start),\n",
+    "    \"Momentum\": run(momentum(0.012, 0.85), grad, start),\n",
+    "    \"Nesterov\": run(nesterov(0.008, 0.85), grad, start),\n",
+    "    \"RMSProp\":  run(rmsprop(0.15),   grad, start),\n",
+    "    \"Adam\":     run(adam(0.5),       grad, start),\n",
+    "}\n",
+    "for name, p in paths.items():\n",
+    "    print(f\"{name:9s} final L = {L(p[-1]):.3g}\")"
+   ],
+   "outputs": []
+  },
+  {
+   "cell_type": "code",
+   "metadata": {
+    "tags": [
+     "hide-input"
+    ]
+   },
+   "execution_count": null,
+   "source": [
+    "g0 = np.linspace(-10, 10, 240); g1 = np.linspace(-2.2, 2.2, 240)\n",
+    "G0, G1 = np.meshgrid(g0, g1)\n",
+    "Z = 0.5 * (G0**2 + KAPPA * G1**2)\n",
+    "plt.figure(figsize=[11, 7])\n",
+    "plt.contour(G0, G1, Z, levels=np.logspace(-0.5, 2.5, 22), cmap=\"Greys\", alpha=0.5)\n",
+    "for name, p in paths.items():\n",
+    "    plt.plot(p[:, 0], p[:, 1], \"-\", lw=1.5, label=name)\n",
+    "plt.scatter([0], [0], color=\"g\", zorder=5, label=\"minimum\")\n",
+    "plt.xlabel(r\"$\\theta_0$\"); plt.ylabel(r\"$\\theta_1$\")\n",
+    "plt.legend(); plt.title(r\"optimizer paths down a ravine ($\\kappa = 100$)\")\n",
+    "plt.show()"
+   ],
+   "outputs": []
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Saddle points\n",
+    "\n",
+    "In high dimensions most critical points where the gradient vanishes are not minima but **saddles**, low along some directions and high along others. A clean two-dimensional model is\n",
+    "\n",
+    "$$L(\\boldsymbol{\\theta}) = \\tfrac{1}{2}\\big(\\theta_0^2 - \\theta_1^2\\big), \\qquad \\nabla_{\\boldsymbol{\\theta}} L = (\\theta_0,\\; -\\theta_1),$$\n",
+    "\n",
+    "with a saddle at the origin. Starting almost on the ridge ($\\theta_1 \\approx 0$) the gradient in the escape direction is tiny, so plain gradient descent and momentum dawdle near the origin, while the per-coordinate scaling in RMSProp and Adam amplifies the weak direction and breaks away sooner."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "execution_count": null,
+   "source": [
+    "def L_saddle(theta):\n",
+    "    return 0.5 * (theta[0]**2 - theta[1]**2)\n",
+    "def grad_saddle(theta):\n",
+    "    return np.array([theta[0], -theta[1]])\n",
+    "\n",
+    "start_s = np.array([-1.8, 1e-2])     # almost on the ridge\n",
+    "paths_s = {\n",
+    "    \"SGD\":      run(sgd(0.08),      grad_saddle, start_s, n=35),\n",
+    "    \"Momentum\": run(momentum(0.04), grad_saddle, start_s, n=35),\n",
+    "    \"RMSProp\":  run(rmsprop(0.03),  grad_saddle, start_s, n=35),\n",
+    "    \"Adam\":     run(adam(0.05),     grad_saddle, start_s, n=35),\n",
+    "}\n",
+    "for name, p in paths_s.items():\n",
+    "    print(f\"{name:9s} |theta_1| after 35 steps = {abs(p[-1, 1]):.3f}\")"
+   ],
+   "outputs": []
+  },
+  {
+   "cell_type": "code",
+   "metadata": {
+    "tags": [
+     "hide-input"
+    ]
+   },
+   "execution_count": null,
+   "source": [
+    "g0 = np.linspace(-2, 2, 200); g1 = np.linspace(-3, 3, 200)\n",
+    "G0, G1 = np.meshgrid(g0, g1)\n",
+    "Z = 0.5 * (G0**2 - G1**2)\n",
+    "plt.figure(figsize=[9, 8])\n",
+    "plt.contour(G0, G1, Z, levels=25, cmap=\"Greys\", alpha=0.5)\n",
+    "for name, p in paths_s.items():\n",
+    "    plt.plot(p[:, 0], p[:, 1], \"-o\", ms=3, lw=1.2, label=name)\n",
+    "plt.scatter([0], [0], color=\"r\", marker=\"x\", zorder=5, label=\"saddle\")\n",
+    "plt.xlim(-2, 2); plt.ylim(-3, 3)\n",
+    "plt.xlabel(r\"$\\theta_0$\"); plt.ylabel(r\"$\\theta_1$\")\n",
+    "plt.legend(); plt.title(\"escaping a saddle point\"); plt.show()"
+   ],
+   "outputs": []
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## The same optimizers in PyTorch\n",
+    "\n",
+    "`torch.optim` ships these as one-line choices. Vanilla SGD takes a `momentum` argument, and Adam is its own class. Optimizing the ravine through autograd reproduces what you built by hand."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "execution_count": null,
+   "source": [
+    "import torch\n",
+    "\n",
+    "def torch_descend(make_opt, n=120):\n",
+    "    theta = torch.tensor([-9.0, -1.0], requires_grad=True)\n",
+    "    opt = make_opt([theta])\n",
+    "    loss = None\n",
+    "    for _ in range(n):\n",
+    "        opt.zero_grad()\n",
+    "        loss = 0.5 * (theta[0]**2 + KAPPA * theta[1]**2)\n",
+    "        loss.backward()\n",
+    "        opt.step()\n",
+    "    return theta.detach().numpy(), loss.item()\n",
+    "\n",
+    "for name, make_opt in {\n",
+    "    \"SGD\":          lambda p: torch.optim.SGD(p, lr=0.018),\n",
+    "    \"SGD+momentum\": lambda p: torch.optim.SGD(p, lr=0.012, momentum=0.85),\n",
+    "    \"Adam\":         lambda p: torch.optim.Adam(p, lr=0.5),\n",
+    "}.items():\n",
+    "    theta, loss = torch_descend(make_opt)\n",
+    "    print(f\"{name:13s} final L = {loss:.3g}   theta = {np.round(theta, 3)}\")"
+   ],
+   "outputs": []
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Takeaways\n",
+    "\n",
+    "- Plain gradient descent is limited by the **steepest** direction, so on an ill-conditioned ravine it zigzags and the shallow direction crawls.\n",
+    "- **Momentum** averages gradients into a velocity that cancels the oscillation and accelerates along the valley; **Nesterov** sharpens this with a look-ahead gradient.\n",
+    "- **Adaptive** methods (RMSProp, Adam) rescale each coordinate by its own gradient history, which both fixes the conditioning and helps escape saddle points where one direction is nearly flat.\n",
+    "- **Adam** is momentum plus per-coordinate scaling with bias correction, the common default; well-tuned SGD with momentum often matches or beats it on large problems.\n",
+    "- `torch.optim` provides all of these; the update rules are exactly the ones implemented here by hand.\n",
+    "\n",
+    "**Key references**: [@Kingma2014-ua; @Ruder2016-overview; @Goodfellow2014-ub]"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}