Skip to content

feat: Add HTTP connection pooling for Lighthouse API calls #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Patrick-Ehimen merged 2 commits into from
Feb 13, 2026
Merged

feat: Add HTTP connection pooling for Lighthouse API calls #75

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8fea2c0
feat: integrate HTTP connection pooling for Lighthouse API calls
Patrick-Ehimen Feb 11, 2026
ad8a7b9
Merge origin/main into feat/connection-pooling
Patrick-Ehimen Feb 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions apps/mcp-server/src/config/server-config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ import { MultiTenancyConfig, OrganizationSettings, UsageQuota } from "@lighthous
import * as path from "path";
import * as os from "os";

export interface ConnectionPoolServerConfig {
maxConnections: number;
idleTimeoutMs: number;
requestTimeoutMs: number;
keepAlive: boolean;
Comment on lines +10 to +14
Copy link

@sourcery-ai sourcery-ai bot Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): ConnectionPoolServerConfig shape doesn’t align with ConnectionPoolConfig used by the SDK

This type uses idleTimeoutMs / requestTimeoutMs, while LighthouseConfig.pool is ConnectionPoolConfig, which (per LighthouseAISDK usage) expects idleTimeout, requestTimeout, acquireTimeout, and maxSockets. If ServerConfig.connectionPool is passed through to LighthouseService/SDK, your timeout values will be ignored and required fields may be missing. Please either align this interface with ConnectionPoolConfig or explicitly map it to a ConnectionPoolConfig before calling LighthouseAISDK.

}

export interface ServerConfig {
name: string;
version: string;
Expand All @@ -19,6 +26,7 @@ export interface ServerConfig {
authentication?: AuthConfig;
performance?: PerformanceConfig;
multiTenancy?: MultiTenancyConfig;
connectionPool?: ConnectionPoolServerConfig;
}

/**
Expand Down Expand Up @@ -73,6 +81,13 @@ export const DEFAULT_PERFORMANCE_CONFIG: PerformanceConfig = {
concurrentRequestLimit: 100,
};

export const DEFAULT_CONNECTION_POOL_CONFIG: ConnectionPoolServerConfig = {
maxConnections: parseInt(process.env.LIGHTHOUSE_POOL_MAX_CONNECTIONS || "10", 10),
idleTimeoutMs: parseInt(process.env.LIGHTHOUSE_POOL_IDLE_TIMEOUT || "60000", 10),
requestTimeoutMs: parseInt(process.env.LIGHTHOUSE_POOL_REQUEST_TIMEOUT || "30000", 10),
keepAlive: process.env.LIGHTHOUSE_POOL_KEEP_ALIVE !== "false",
};

export const DEFAULT_ORGANIZATION_SETTINGS: OrganizationSettings = {
defaultStorageQuota: 10 * 1024 * 1024 * 1024, // 10GB
defaultRateLimit: 1000, // 1000 requests per minute
Expand Down Expand Up @@ -131,6 +146,7 @@ export function getDefaultServerConfig(): ServerConfig {
authentication: getDefaultAuthConfig(),
performance: DEFAULT_PERFORMANCE_CONFIG,
multiTenancy: DEFAULT_MULTI_TENANCY_CONFIG,
connectionPool: DEFAULT_CONNECTION_POOL_CONFIG,
};
}

Expand All @@ -149,6 +165,7 @@ export const DEFAULT_SERVER_CONFIG: ServerConfig = {
authentication: DEFAULT_AUTH_CONFIG,
performance: DEFAULT_PERFORMANCE_CONFIG,
multiTenancy: DEFAULT_MULTI_TENANCY_CONFIG,
connectionPool: DEFAULT_CONNECTION_POOL_CONFIG,
};

/**
Expand Down
5 changes: 4 additions & 1 deletion apps/mcp-server/src/services/LighthouseService.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import {
LighthouseAISDK,
EnhancedAccessCondition,
ConnectionPoolConfig,
BatchUploadOptions,
BatchDownloadOptions,
BatchOperationResult,
Expand All @@ -29,7 +30,7 @@ export class LighthouseService implements ILighthouseService {
private fileCache: Map<string, StoredFile> = new Map();
private datasetCache: Map<string, Dataset> = new Map();

constructor(apiKey: string, logger?: Logger, dbPath?: string) {
constructor(apiKey: string, logger?: Logger, dbPath?: string, poolConfig?: ConnectionPoolConfig) {
this.logger = logger || Logger.getInstance({ level: "info", component: "LighthouseService" });
this.dbPath = dbPath;

Expand All @@ -41,6 +42,7 @@ export class LighthouseService implements ILighthouseService {
timeout: 30000,
maxRetries: 3,
debug: false,
pool: poolConfig,
});

// Set up event listeners for progress tracking
Expand Down Expand Up @@ -450,6 +452,7 @@ export class LighthouseService implements ILighthouseService {
activeOperations: this.sdk.getActiveOperations(),
errorMetrics: this.sdk.getErrorMetrics(),
circuitBreaker: this.sdk.getCircuitBreakerStatus(),
connectionPool: this.sdk.getConnectionPoolStats(),
};
}

Expand Down
75 changes: 72 additions & 3 deletions packages/sdk-wrapper/src/LighthouseAISDK.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,14 @@ import { EventEmitter } from "eventemitter3";
import lighthouse from "@lighthouse-web3/sdk";
import { readFileSync, createWriteStream, promises as fsPromises } from "fs";
import { dirname } from "path";
import axios from "axios";
import axios, { AxiosRequestConfig, AxiosResponse } from "axios";
import { AuthenticationManager } from "./auth/AuthenticationManager";
import { ProgressTracker } from "./progress/ProgressTracker";
import { ErrorHandler } from "./errors/ErrorHandler";
import { CircuitBreaker } from "./errors/CircuitBreaker";
import { EncryptionManager } from "./encryption/EncryptionManager";
import { RateLimiter } from "./utils/RateLimiter";
import { ConnectionPool, ConnectionPoolConfig } from "./pool";
import {
LighthouseConfig,
UploadOptions,
Expand Down Expand Up @@ -78,6 +79,7 @@ export class LighthouseAISDK extends EventEmitter {
private circuitBreaker: CircuitBreaker;
private encryption: EncryptionManager;
private rateLimiter: RateLimiter;
private connectionPool: ConnectionPool | null;
private memoryManager: MemoryManager;
private config: LighthouseConfig;

Expand All @@ -101,6 +103,22 @@ export class LighthouseAISDK extends EventEmitter {
autoCleanup: true,
});

// Initialize connection pool (unless explicitly disabled)
if (config.pool === false) {
this.connectionPool = null;
} else {
const poolConfig: ConnectionPoolConfig = {
maxConnections: 10,
acquireTimeout: 5000,
idleTimeout: 60000,
requestTimeout: config.timeout || 30000,
keepAlive: true,
maxSockets: 50,
...(typeof config.pool === "object" ? config.pool : {}),
};
this.connectionPool = new ConnectionPool(poolConfig);
}

// Forward authentication events
this.auth.on("auth:error", (error) => this.emit("auth:error", error));
this.auth.on("auth:refresh", () => this.emit("auth:refresh"));
Expand Down Expand Up @@ -145,6 +163,15 @@ export class LighthouseAISDK extends EventEmitter {
this.emit("encryption:access:control:error", event),
);

// Forward connection pool events
if (this.connectionPool) {
this.connectionPool.on("acquire", (event) => this.emit("pool:acquire", event));
this.connectionPool.on("create", (event) => this.emit("pool:create", event));
this.connectionPool.on("queue", (event) => this.emit("pool:queue", event));
this.connectionPool.on("release", (event) => this.emit("pool:release", event));
this.connectionPool.on("cleanup", (event) => this.emit("pool:cleanup", event));
}

// Forward memory manager events
this.memoryManager.on("backpressure:start", (event) =>
this.emit("memory:backpressure:start", event),
Expand Down Expand Up @@ -363,6 +390,25 @@ Maximum file size may be exceeded. Try uploading a smaller file.`);
]);
}

/**
* Execute an HTTP request using the connection pool if available,
* otherwise fall back to a direct axios call.
*/
private async executeHttpRequest<T = any>(config: AxiosRequestConfig): Promise<AxiosResponse<T>> {
if (this.connectionPool) {
const instance = await this.connectionPool.acquire();
try {
return await instance.request<T>(config);
} finally {
this.connectionPool.release(instance);
}
} else {
const axiosLib: { request: (config: AxiosRequestConfig) => Promise<AxiosResponse<T>> } =
Comment on lines +397 to +406
Copy link

@sourcery-ai sourcery-ai bot Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (performance): Dynamic axios require in executeHttpRequest is repeated on every non-pooled call

In the non-pooled branch, axios is dynamically loaded via eval("require") on every call. Since executeHttpRequest is now on a hot path, this adds unnecessary overhead and can confuse bundlers. Consider caching the loaded axios instance (e.g., in a private field or module-level variable) so the dynamic require runs only once while preserving the indirection.

Suggested implementation:

let cachedAxiosLib:
  | { request: (config: AxiosRequestConfig) => Promise<AxiosResponse<any>> }
  | null = null;

function getAxiosLib<T = any>(): {
  request: (config: AxiosRequestConfig) => Promise<AxiosResponse<T>>;
} {
  if (!cachedAxiosLib) {
    // Use dynamic require to avoid bundler issues while caching the result
    cachedAxiosLib = eval("require")("axios");
  }
  return cachedAxiosLib as {
    request: (config: AxiosRequestConfig) => Promise<AxiosResponse<T>>;
  };
}

/**
   * Execute an HTTP request using the connection pool if available,
   * otherwise fall back to a direct axios call.
   */
  private async executeHttpRequest<T = any>(config: AxiosRequestConfig): Promise<AxiosResponse<T>> {
    } else {
      const axiosLib = getAxiosLib<T>();
      return axiosLib.request(config);
    }

None required, assuming AxiosRequestConfig and AxiosResponse are already imported in this file (they are referenced in the existing code). The helper and cache are module-level so they will be shared across all uses in this module without further changes.

eval("require")("axios");
return axiosLib.request(config);
}
}

/**
* Upload file via direct API call as fallback when SDK fails
*/
Expand All @@ -375,12 +421,14 @@ Maximum file size may be exceeded. Try uploading a smaller file.`);
// when the standard SDK fails (usually due to node.lighthouse.storage being down)

const FormData = eval("require")("form-data");
const axios = eval("require")("axios");

const formData = new FormData();
formData.append("file", fileBuffer, fileName);

const response = await axios.post("https://api.lighthouse.storage/api/v0/add", formData, {
const response = await this.executeHttpRequest({
method: "POST",
url: "https://api.lighthouse.storage/api/v0/add",
data: formData,
headers: {
...formData.getHeaders(),
Authorization: `Bearer ${apiKey}`,
Expand Down Expand Up @@ -932,6 +980,24 @@ Check your internet connection and try again.`);
return this.circuitBreaker.getMetrics();
}

/**
* Get connection pool statistics.
* Returns null if the connection pool is disabled.
*/
getConnectionPoolStats(): {
totalConnections: number;
activeConnections: number;
idleConnections: number;
queuedRequests: number;
totalRequests: number;
averageWaitTime: number;
} | null {
if (!this.connectionPool) {
return null;
}
return this.connectionPool.getStats();
}

/**
* Reset error metrics
*/
Expand Down Expand Up @@ -1627,6 +1693,9 @@ Check your internet connection and try again.`);
this.auth.destroy();
this.progress.cleanup();
this.encryption.destroy();
if (this.connectionPool) {
this.connectionPool.destroy();
}
this.memoryManager.destroy();
this.removeAllListeners();
}
Expand Down
Loading
Loading