Skip to content

Commit d28f038

Browse files
sadiqkhznsadiqkhzn
committed
Add LangChain Integration Health Framework documentation
- Add comprehensive integration testing and monitoring framework - Provides compatibility matrix for LangChain integrations - Includes MLXPipeline bind_tools fix example - Web dashboard for real-time monitoring - CLI tool for automated testing
1 parent 238ecd0 commit d28f038

File tree

2 files changed

+291
-0
lines changed

2 files changed

+291
-0
lines changed

‎docs/docs/integrations/tools/langchain_integration_health.ipynb‎

Lines changed: 285 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,285 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# LangChain Integration Health\n",
8+
"\n",
9+
"This notebook demonstrates how to use the LangChain Integration Health Dashboard & Testing Framework to monitor and test LangChain integration compatibility.\n",
10+
"\n",
11+
"## Overview\n",
12+
"\n",
13+
"The LangChain Integration Health framework addresses critical pain points in the LangChain ecosystem:\n",
14+
"- Missing `.bind_tools()` methods in integrations like MLXPipeline\n",
15+
"- Inconsistent API support across different model providers\n",
16+
"- Poor integration testing leading to breaking changes\n",
17+
"- No visibility into which integrations support which features\n",
18+
"\n",
19+
"## Installation\n",
20+
"\n",
21+
"```bash\n",
22+
"pip install langchain-integration-health\n",
23+
"```\n",
24+
"\n",
25+
"## Features\n",
26+
"\n",
27+
"1. **Integration Compatibility Testing Framework** - Automated testing suite that validates LangChain integrations\n",
28+
"2. **Real-time Integration Health Dashboard** - Web-based dashboard showing integration compatibility status\n",
29+
"3. **Integration Template Generator** - Standardized templates for new integrations\n",
30+
"4. **MLXPipeline Fix Example** - Complete example of how to fix missing bind_tools functionality\n",
31+
"\n",
32+
"## Quick Start\n",
33+
"\n",
34+
"### CLI Usage"
35+
]
36+
},
37+
{
38+
"cell_type": "code",
39+
"execution_count": null,
40+
"metadata": {},
41+
"outputs": [],
42+
"source": [
43+
"# Install the package first\n",
44+
"# pip install langchain-integration-health\n",
45+
"\n",
46+
"# Run from command line:\n",
47+
"# langchain-health discover # Discover available integrations\n",
48+
"# langchain-health test # Run integration tests\n",
49+
"# langchain-health dashboard # Launch the dashboard\n",
50+
"# langchain-health report # Generate compatibility report"
51+
]
52+
},
53+
{
54+
"cell_type": "markdown",
55+
"metadata": {},
56+
"source": [
57+
"### Programmatic Usage\n",
58+
"\n",
59+
"Test a specific integration:"
60+
]
61+
},
62+
{
63+
"cell_type": "code",
64+
"execution_count": null,
65+
"metadata": {},
66+
"outputs": [],
67+
"source": [
68+
"import asyncio\n",
69+
"from langchain_integration_health.testers import LLMIntegrationTester\n",
70+
"from langchain_integration_health.utils.config import Config\n",
71+
"\n",
72+
"# Test a specific integration\n",
73+
"async def test_integration():\n",
74+
" config = Config.from_env()\n",
75+
" \n",
76+
" # Example: Test OpenAI integration\n",
77+
" from langchain_openai import ChatOpenAI\n",
78+
" \n",
79+
" tester = LLMIntegrationTester(ChatOpenAI, config.get_integration_config(\"ChatOpenAI\"))\n",
80+
" result = await tester.run_all_tests()\n",
81+
" \n",
82+
" print(f\"Compatibility Score: {result.compatibility_score}\")\n",
83+
" print(f\"bind_tools Support: {result.bind_tools_support}\")\n",
84+
" print(f\"Streaming Support: {result.streaming_support}\")\n",
85+
" print(f\"Structured Output Support: {result.structured_output_support}\")\n",
86+
"\n",
87+
"# asyncio.run(test_integration())"
88+
]
89+
},
90+
{
91+
"cell_type": "markdown",
92+
"metadata": {},
93+
"source": [
94+
"### MLXPipeline Fix Example\n",
95+
"\n",
96+
"The framework includes a complete example of how to fix the MLXPipeline `bind_tools` issue:"
97+
]
98+
},
99+
{
100+
"cell_type": "code",
101+
"execution_count": null,
102+
"metadata": {},
103+
"outputs": [],
104+
"source": [
105+
"# from langchain_integration_health.examples.mlx_pipeline_fix import create_mlx_wrapper\n",
106+
"# from langchain.tools import tool\n",
107+
"\n",
108+
"# # Original MLXPipeline (missing bind_tools)\n",
109+
"# mlx = MLXPipeline(model_name=\"mlx-community/Llama-3.2-1B-Instruct-4bit\")\n",
110+
"\n",
111+
"# # Wrap for LangChain compatibility\n",
112+
"# langchain_mlx = create_mlx_wrapper(mlx)\n",
113+
"\n",
114+
"# @tool\n",
115+
"# def calculator(a: int, b: int) -> int:\n",
116+
"# \"\"\"Add two numbers\"\"\"\n",
117+
"# return a + b\n",
118+
"\n",
119+
"# # Now you can use bind_tools!\n",
120+
"# mlx_with_tools = langchain_mlx.bind_tools([calculator])\n",
121+
"# response = mlx_with_tools.invoke(\"What is 5 + 3?\")\n",
122+
"# print(response)"
123+
]
124+
},
125+
{
126+
"cell_type": "markdown",
127+
"metadata": {},
128+
"source": [
129+
"### Launch Dashboard\n",
130+
"\n",
131+
"Start the real-time dashboard to monitor integration health:"
132+
]
133+
},
134+
{
135+
"cell_type": "code",
136+
"execution_count": null,
137+
"metadata": {},
138+
"outputs": [],
139+
"source": [
140+
"# Launch dashboard from Python\n",
141+
"# from langchain_integration_health.dashboard.app import main\n",
142+
"# main()\n",
143+
"\n",
144+
"# Or run from command line:\n",
145+
"# streamlit run -m langchain_integration_health.dashboard.app"
146+
]
147+
},
148+
{
149+
"cell_type": "markdown",
150+
"metadata": {},
151+
"source": [
152+
"### Batch Testing Multiple Integrations"
153+
]
154+
},
155+
{
156+
"cell_type": "code",
157+
"execution_count": null,
158+
"metadata": {},
159+
"outputs": [],
160+
"source": [
161+
"from langchain_integration_health.utils.discovery import IntegrationDiscovery\n",
162+
"from langchain_integration_health.utils.reporters import CompatibilityReporter\n",
163+
"\n",
164+
"async def test_all_integrations():\n",
165+
" # Discover all installed integrations\n",
166+
" discovery = IntegrationDiscovery()\n",
167+
" integrations = discovery.discover_integrations()\n",
168+
" \n",
169+
" print(f\"Found {len(integrations)} integrations to test\")\n",
170+
" \n",
171+
" # Test each integration\n",
172+
" results = []\n",
173+
" for integration in integrations:\n",
174+
" tester = LLMIntegrationTester(integration.class_type)\n",
175+
" result = await tester.run_all_tests()\n",
176+
" results.append(result)\n",
177+
" \n",
178+
" # Generate report\n",
179+
" reporter = CompatibilityReporter()\n",
180+
" report = reporter.generate_compatibility_matrix(results)\n",
181+
" \n",
182+
" return report\n",
183+
"\n",
184+
"# asyncio.run(test_all_integrations())"
185+
]
186+
},
187+
{
188+
"cell_type": "markdown",
189+
"metadata": {},
190+
"source": [
191+
"## Dashboard Features\n",
192+
"\n",
193+
"The web dashboard provides:\n",
194+
"\n",
195+
"- **Compatibility Matrix**: Color-coded grid showing which integrations support which features\n",
196+
"- **Integration Details**: Expandable sections with full test results\n",
197+
"- **Search/Filter**: Find integrations by provider, feature support, or compatibility score\n",
198+
"- **Historical Data**: Track compatibility changes over time\n",
199+
"- **Export Results**: JSON/CSV export of test results\n",
200+
"\n",
201+
"## CI/CD Integration\n",
202+
"\n",
203+
"The framework includes GitHub Actions workflow for automated testing. Add this to your repository:\n",
204+
"\n",
205+
"```yaml\n",
206+
"name: LangChain Integration Health Check\n",
207+
"on:\n",
208+
" push:\n",
209+
" pull_request:\n",
210+
" schedule:\n",
211+
" - cron: '0 2 * * *' # Daily at 2 AM\n",
212+
"\n",
213+
"jobs:\n",
214+
" test-integrations:\n",
215+
" runs-on: ubuntu-latest\n",
216+
" steps:\n",
217+
" - uses: actions/checkout@v4\n",
218+
" - uses: actions/setup-python@v4\n",
219+
" with:\n",
220+
" python-version: '3.11'\n",
221+
" - run: pip install langchain-integration-health\n",
222+
" - run: langchain-health test\n",
223+
" - run: langchain-health report\n",
224+
"```\n",
225+
"\n",
226+
"## Configuration\n",
227+
"\n",
228+
"Configure via environment variables or config file:"
229+
]
230+
},
231+
{
232+
"cell_type": "code",
233+
"execution_count": null,
234+
"metadata": {},
235+
"outputs": [],
236+
"source": [
237+
"import os\n",
238+
"\n",
239+
"# Environment variables\n",
240+
"os.environ['LIH_DATABASE_URL'] = 'sqlite:///integration_health.db'\n",
241+
"os.environ['LIH_TEST_TIMEOUT'] = '30'\n",
242+
"os.environ['LIH_PARALLEL_TESTS'] = 'true'\n",
243+
"os.environ['LIH_MOCK_MODE'] = 'false'\n",
244+
"\n",
245+
"# Optional: API keys for real testing\n",
246+
"# os.environ['OPENAI_API_KEY'] = 'your_key_here'\n",
247+
"# os.environ['ANTHROPIC_API_KEY'] = 'your_key_here'"
248+
]
249+
},
250+
{
251+
"cell_type": "markdown",
252+
"metadata": {},
253+
"source": [
254+
"## More Information\n",
255+
"\n",
256+
"- **GitHub**: [sadiqkhzn/langchain-integration-health](https://github.com/sadiqkhzn/langchain-integration-health)\n",
257+
"- **PyPI**: [langchain-integration-health](https://pypi.org/project/langchain-integration-health/)\n",
258+
"- **Documentation**: See the GitHub README for comprehensive documentation\n",
259+
"\n",
260+
"This framework helps ensure LangChain integrations are reliable, compatible, and well-tested, making the entire ecosystem more robust for developers."
261+
]
262+
}
263+
],
264+
"metadata": {
265+
"kernelspec": {
266+
"display_name": "Python 3 (ipykernel)",
267+
"language": "python",
268+
"name": "python3"
269+
},
270+
"language_info": {
271+
"codemirror_mode": {
272+
"name": "ipython",
273+
"version": 3
274+
},
275+
"file_extension": ".py",
276+
"mimetype": "text/x-python",
277+
"name": "python",
278+
"nbconvert_exporter": "python",
279+
"pygments_lexer": "ipython3",
280+
"version": "3.11.0"
281+
}
282+
},
283+
"nbformat": 4,
284+
"nbformat_minor": 4
285+
}

‎libs/packages.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -733,4 +733,10 @@ packages:
733733
- name: langchain-scrapeless
734734
repo: scrapeless-ai/langchain-scrapeless
735735
path: .
736+
- name: langchain-integration-health
737+
name_title: Integration Health Framework
738+
path: .
739+
repo: sadiqkhzn/langchain-integration-health
740+
downloads: 1000
741+
downloads_updated_at: '2025-08-10T21:38:36.795416+00:00'
736742

0 commit comments

Comments
 (0)