Skip to content

Commit 765732d

Browse files
Goldokpaobielinclaude
authored
feat(governance): add regional bias audit framework for model fairness (#30)
- Add BiasAuditor class with demographic parity, equalized odds, and predictive parity metrics - Implement run_bias_audit() for evaluating model fairness across regions - Add check_fairness_gate() for CI/CD integration - Create scripts/audit_model.py CLI tool for running audits - Add notebooks/07_bias_audit.ipynb with visualization examples - Support Amazon, Congo, Southeast Asia, and Boreal forest regions Closes #23 Co-authored-by: Linda Oraegbunam <obielinda@gmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 833e08a commit 765732d

4 files changed

Lines changed: 1138 additions & 0 deletions

File tree

‎notebooks/07_bias_audit.ipynb‎

Lines changed: 374 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,374 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# ClimateVision Regional Bias Audit\n",
8+
"\n",
9+
"This notebook demonstrates how to evaluate model fairness across geographic regions.\n",
10+
"Ensuring equitable predictions is critical for NGOs operating in different parts of the world.\n",
11+
"\n",
12+
"**Author:** Linda Oraegbunam (@obielin) \n",
13+
"**Module:** `src/climatevision/governance/bias_audit.py`"
14+
]
15+
},
16+
{
17+
"cell_type": "code",
18+
"execution_count": null,
19+
"metadata": {},
20+
"outputs": [],
21+
"source": [
22+
"import sys\n",
23+
"sys.path.insert(0, '..')\n",
24+
"\n",
25+
"import numpy as np\n",
26+
"import matplotlib.pyplot as plt\n",
27+
"from pathlib import Path\n",
28+
"\n",
29+
"from climatevision.governance import (\n",
30+
" run_bias_audit,\n",
31+
" BiasAuditor,\n",
32+
" BiasReport,\n",
33+
" check_fairness_gate,\n",
34+
" SUPPORTED_REGIONS,\n",
35+
")"
36+
]
37+
},
38+
{
39+
"cell_type": "markdown",
40+
"metadata": {},
41+
"source": [
42+
"## 1. Understanding Regional Bias\n",
43+
"\n",
44+
"Climate models trained primarily on Amazon data may underperform on Congo Basin imagery due to:\n",
45+
"- Different forest types and canopy structures\n",
46+
"- Varying cloud patterns and seasonal effects\n",
47+
"- Different satellite viewing angles and atmospheric conditions\n",
48+
"\n",
49+
"This audit ensures NGOs in all regions receive equally reliable predictions."
50+
]
51+
},
52+
{
53+
"cell_type": "code",
54+
"execution_count": null,
55+
"metadata": {},
56+
"outputs": [],
57+
"source": [
58+
"# View supported regions\n",
59+
"print(\"Supported Regions for Bias Audit:\")\n",
60+
"print(\"=\" * 50)\n",
61+
"for key, info in SUPPORTED_REGIONS.items():\n",
62+
" print(f\"\\n{info['name']} ({key})\")\n",
63+
" print(f\" Bounding Box: {info['bbox']}\")\n",
64+
" print(f\" Description: {info['description']}\")"
65+
]
66+
},
67+
{
68+
"cell_type": "markdown",
69+
"metadata": {},
70+
"source": [
71+
"## 2. Creating a Bias Auditor"
72+
]
73+
},
74+
{
75+
"cell_type": "code",
76+
"execution_count": null,
77+
"metadata": {},
78+
"outputs": [],
79+
"source": [
80+
"# Create auditor with 85% fairness threshold\n",
81+
"auditor = BiasAuditor(model=None, threshold=0.85)\n",
82+
"\n",
83+
"# Simulate regional prediction data\n",
84+
"# In production, this would be real model outputs on test sets\n",
85+
"np.random.seed(42)\n",
86+
"\n",
87+
"regions_data = {\n",
88+
" 'amazon': {'accuracy': 0.92, 'forest_ratio': 0.70},\n",
89+
" 'congo': {'accuracy': 0.85, 'forest_ratio': 0.65},\n",
90+
" 'southeast_asia': {'accuracy': 0.88, 'forest_ratio': 0.55},\n",
91+
"}\n",
92+
"\n",
93+
"for region, params in regions_data.items():\n",
94+
" n_samples = 1000\n",
95+
" \n",
96+
" # Ground truth based on regional forest coverage\n",
97+
" ground_truth = (np.random.random(n_samples) < params['forest_ratio']).astype(int)\n",
98+
" \n",
99+
" # Predictions based on regional accuracy\n",
100+
" correct = np.random.random(n_samples) < params['accuracy']\n",
101+
" predictions = np.where(correct, ground_truth, 1 - ground_truth)\n",
102+
" \n",
103+
" auditor.add_region_data(region, predictions, ground_truth)\n",
104+
" print(f\"Added {n_samples} samples for {region}\")"
105+
]
106+
},
107+
{
108+
"cell_type": "markdown",
109+
"metadata": {},
110+
"source": [
111+
"## 3. Computing Fairness Metrics"
112+
]
113+
},
114+
{
115+
"cell_type": "code",
116+
"execution_count": null,
117+
"metadata": {},
118+
"outputs": [],
119+
"source": [
120+
"# Run full bias audit\n",
121+
"report = auditor.run_audit(\n",
122+
" metric='equalized_odds',\n",
123+
" model_path='models/demo_model.pth',\n",
124+
" model_version='v1.0-demo',\n",
125+
" analysis_type='deforestation',\n",
126+
")\n",
127+
"\n",
128+
"print(f\"Fairness Score: {report.fairness_score:.4f}\")\n",
129+
"print(f\"Threshold: {report.threshold}\")\n",
130+
"print(f\"Passed: {'✅' if report.passed else '❌'}\")\n",
131+
"print(f\"\\nDisparity Regions: {report.disparity_regions or 'None'}\")"
132+
]
133+
},
134+
{
135+
"cell_type": "code",
136+
"execution_count": null,
137+
"metadata": {},
138+
"outputs": [],
139+
"source": [
140+
"# View per-region metrics\n",
141+
"print(\"Per-Region Metrics:\")\n",
142+
"print(\"=\" * 60)\n",
143+
"\n",
144+
"for metrics in report.region_metrics:\n",
145+
" print(f\"\\n{metrics.region_name} ({metrics.region}):\")\n",
146+
" print(f\" Samples: {metrics.n_samples}\")\n",
147+
" print(f\" IoU: {metrics.iou:.4f}\")\n",
148+
" print(f\" F1: {metrics.f1:.4f}\")\n",
149+
" print(f\" Precision: {metrics.precision:.4f}\")\n",
150+
" print(f\" Recall: {metrics.recall:.4f}\")\n",
151+
" print(f\" TPR: {metrics.true_positive_rate:.4f}\")\n",
152+
" print(f\" FPR: {metrics.false_positive_rate:.4f}\")"
153+
]
154+
},
155+
{
156+
"cell_type": "markdown",
157+
"metadata": {},
158+
"source": [
159+
"## 4. Visualizing Regional Disparities"
160+
]
161+
},
162+
{
163+
"cell_type": "code",
164+
"execution_count": null,
165+
"metadata": {},
166+
"outputs": [],
167+
"source": [
168+
"# Prepare data for visualization\n",
169+
"regions = [m.region_name for m in report.region_metrics]\n",
170+
"ious = [m.iou for m in report.region_metrics]\n",
171+
"f1s = [m.f1 for m in report.region_metrics]\n",
172+
"tprs = [m.true_positive_rate for m in report.region_metrics]\n",
173+
"\n",
174+
"x = np.arange(len(regions))\n",
175+
"width = 0.25\n",
176+
"\n",
177+
"fig, ax = plt.subplots(figsize=(12, 6))\n",
178+
"\n",
179+
"bars1 = ax.bar(x - width, ious, width, label='IoU', color='#3498db')\n",
180+
"bars2 = ax.bar(x, f1s, width, label='F1 Score', color='#2ecc71')\n",
181+
"bars3 = ax.bar(x + width, tprs, width, label='True Positive Rate', color='#e74c3c')\n",
182+
"\n",
183+
"ax.set_ylabel('Score')\n",
184+
"ax.set_title('Model Performance by Region')\n",
185+
"ax.set_xticks(x)\n",
186+
"ax.set_xticklabels(regions)\n",
187+
"ax.legend()\n",
188+
"ax.set_ylim(0, 1.1)\n",
189+
"ax.axhline(y=0.85, color='gray', linestyle='--', label='Threshold')\n",
190+
"\n",
191+
"plt.tight_layout()\n",
192+
"plt.show()"
193+
]
194+
},
195+
{
196+
"cell_type": "code",
197+
"execution_count": null,
198+
"metadata": {},
199+
"outputs": [],
200+
"source": [
201+
"# Radar chart for multi-metric comparison\n",
202+
"from math import pi\n",
203+
"\n",
204+
"categories = ['IoU', 'F1', 'Precision', 'Recall', 'TPR']\n",
205+
"N = len(categories)\n",
206+
"\n",
207+
"angles = [n / float(N) * 2 * pi for n in range(N)]\n",
208+
"angles += angles[:1]\n",
209+
"\n",
210+
"fig, ax = plt.subplots(figsize=(8, 8), subplot_kw=dict(polar=True))\n",
211+
"\n",
212+
"colors = ['#3498db', '#2ecc71', '#e74c3c']\n",
213+
"for i, metrics in enumerate(report.region_metrics):\n",
214+
" values = [metrics.iou, metrics.f1, metrics.precision, metrics.recall, metrics.true_positive_rate]\n",
215+
" values += values[:1]\n",
216+
" ax.plot(angles, values, 'o-', linewidth=2, label=metrics.region_name, color=colors[i % len(colors)])\n",
217+
" ax.fill(angles, values, alpha=0.25, color=colors[i % len(colors)])\n",
218+
"\n",
219+
"ax.set_xticks(angles[:-1])\n",
220+
"ax.set_xticklabels(categories)\n",
221+
"ax.set_ylim(0, 1)\n",
222+
"ax.legend(loc='upper right', bbox_to_anchor=(1.3, 1.0))\n",
223+
"ax.set_title('Regional Performance Comparison', y=1.08)\n",
224+
"\n",
225+
"plt.tight_layout()\n",
226+
"plt.show()"
227+
]
228+
},
229+
{
230+
"cell_type": "markdown",
231+
"metadata": {},
232+
"source": [
233+
"## 5. Comparing Fairness Metrics"
234+
]
235+
},
236+
{
237+
"cell_type": "code",
238+
"execution_count": null,
239+
"metadata": {},
240+
"outputs": [],
241+
"source": [
242+
"# Compare different fairness metrics\n",
243+
"metrics_to_test = ['demographic_parity', 'equalized_odds', 'predictive_parity']\n",
244+
"results = {}\n",
245+
"\n",
246+
"for metric in metrics_to_test:\n",
247+
" report = auditor.run_audit(metric=metric)\n",
248+
" results[metric] = {\n",
249+
" 'score': report.fairness_score,\n",
250+
" 'passed': report.passed,\n",
251+
" 'disparity_regions': report.disparity_regions,\n",
252+
" }\n",
253+
"\n",
254+
"print(\"Fairness Metrics Comparison:\")\n",
255+
"print(\"=\" * 50)\n",
256+
"for metric, result in results.items():\n",
257+
" status = '✅' if result['passed'] else '❌'\n",
258+
" print(f\"\\n{metric}:\")\n",
259+
" print(f\" Score: {result['score']:.4f} {status}\")\n",
260+
" if result['disparity_regions']:\n",
261+
" print(f\" Disparity in: {', '.join(result['disparity_regions'])}\")"
262+
]
263+
},
264+
{
265+
"cell_type": "markdown",
266+
"metadata": {},
267+
"source": [
268+
"## 6. Using the High-Level API"
269+
]
270+
},
271+
{
272+
"cell_type": "code",
273+
"execution_count": null,
274+
"metadata": {},
275+
"outputs": [],
276+
"source": [
277+
"# For real usage with trained models:\n",
278+
"# result = run_bias_audit(\n",
279+
"# model_path='models/unet_deforestation.pth',\n",
280+
"# regions=['amazon', 'congo', 'southeast_asia'],\n",
281+
"# metric='equalized_odds',\n",
282+
"# threshold=0.85,\n",
283+
"# )\n",
284+
"# \n",
285+
"# print(f\"Score: {result['score']}\")\n",
286+
"# print(f\"Passed: {result['passed']}\")\n",
287+
"# print(f\"Report: {result['report_path']}\")\n",
288+
"\n",
289+
"print(\"See run_bias_audit() for production usage\")"
290+
]
291+
},
292+
{
293+
"cell_type": "markdown",
294+
"metadata": {},
295+
"source": [
296+
"## 7. CI/CD Integration"
297+
]
298+
},
299+
{
300+
"cell_type": "code",
301+
"execution_count": null,
302+
"metadata": {},
303+
"outputs": [],
304+
"source": [
305+
"# CI gate function for automated checks\n",
306+
"# This would be called in GitHub Actions or similar\n",
307+
"\n",
308+
"# passed = check_fairness_gate(\n",
309+
"# model_path='models/best_model.pth',\n",
310+
"# regions=['amazon', 'congo', 'southeast_asia'],\n",
311+
"# threshold=0.85,\n",
312+
"# )\n",
313+
"# \n",
314+
"# if not passed:\n",
315+
"# sys.exit(1) # Fail the CI build\n",
316+
"\n",
317+
"print(\"Use check_fairness_gate() in CI/CD pipelines\")\n",
318+
"print(\"Command: python scripts/audit_model.py --model models/best.pth --ci-gate\")"
319+
]
320+
},
321+
{
322+
"cell_type": "markdown",
323+
"metadata": {},
324+
"source": [
325+
"## 8. Recommendations"
326+
]
327+
},
328+
{
329+
"cell_type": "code",
330+
"execution_count": null,
331+
"metadata": {},
332+
"outputs": [],
333+
"source": [
334+
"# Get recommendations from the audit\n",
335+
"print(\"Recommendations:\")\n",
336+
"print(\"=\" * 50)\n",
337+
"for rec in report.recommendations:\n",
338+
" print(f\"\\n• {rec}\")"
339+
]
340+
},
341+
{
342+
"cell_type": "markdown",
343+
"metadata": {},
344+
"source": [
345+
"## Summary\n",
346+
"\n",
347+
"This notebook demonstrated:\n",
348+
"\n",
349+
"1. **BiasAuditor** - Core class for fairness evaluation\n",
350+
"2. **Fairness Metrics** - Demographic parity, equalized odds, predictive parity\n",
351+
"3. **Regional Analysis** - Per-region IoU, F1, precision, recall\n",
352+
"4. **Visualization** - Bar charts and radar plots for stakeholder reports\n",
353+
"5. **CI/CD Integration** - `check_fairness_gate()` for automated checks\n",
354+
"\n",
355+
"For production use:\n",
356+
"- Run `python scripts/audit_model.py --model <path> --regions amazon,congo`\n",
357+
"- Add `--ci-gate` flag to fail builds with poor fairness scores"
358+
]
359+
}
360+
],
361+
"metadata": {
362+
"kernelspec": {
363+
"display_name": "Python 3",
364+
"language": "python",
365+
"name": "python3"
366+
},
367+
"language_info": {
368+
"name": "python",
369+
"version": "3.11.0"
370+
}
371+
},
372+
"nbformat": 4,
373+
"nbformat_minor": 4
374+
}

0 commit comments

Comments
 (0)