Skip to content

http2 endpoint #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

http2 endpoint #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cbeef4a
http2 endpoint
Jun 27, 2025
8dc84eb
Merge branch 'main' into http2
lmangani Jun 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 102 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,86 @@ This project provides tools to convert HEP packets into GigAPI Line Protocol.
- Maintains the same column structure and table as hep
- Parses SIP payload data to extract useful fields
- Can be used as a library or as a standalone server
- Supports both TCP and UDP for HEP packet reception
- Supports all of the following for HEP packet reception:
- UDP
- TCP
- HTTP/2 (optional, POST endpoint for raw HEPv3 binary)
- Batch processing for efficient InfluxDB writes
- Optional file output for debugging or offline processing

## Quickstart: All Protocols (Local Development)

```js
import HepToInfluxDBServer from './hep-server.js';

const server = new HepToInfluxDBServer({
hepPort: 9060, // UDP/TCP port
hepBindAddress: '0.0.0.0',
influxDbUrl: 'http://localhost:7971',
influxDbDatabase: 'hep',
batchSize: 1000,
flushInterval: 5000,
debug: true,
// Enable HTTP/2 HEPv3 ingestion (optional)
http2Port: 8080, // HTTP/2 port
http2BindAddress: '0.0.0.0', // HTTP/2 bind address
http2Endpoint: '/test/api', // HTTP/2 POST endpoint
});

server.initialize().catch(console.error);
```

- **UDP/TCP**: Send HEPv3 packets to `hepPort` (default: 9060)
- **HTTP/2**: POST raw HEPv3 binary to `http://localhost:8080/test/api` (see below)

### HTTP/2 HEPv3 Ingestion Example

If enabled, you can POST raw HEPv3 binary packets to the configured HTTP/2 endpoint:

```http
POST /test/api HTTP/2
Host: localhost:8080
Content-Type: application/octet-stream

<raw HEPv3 binary>
```

Example with curl (using HTTP/2):

```bash
curl --http2-prior-knowledge -X POST \
-H "Content-Type: application/octet-stream" \
--data-binary @packet.bin \
http://localhost:8080/test/api
```

## Configuration Options

| Option | Description | Default |
|------------------|-----------------------------------------------------|-------------|
| hepPort | Port to listen for HEP packets (UDP/TCP) | 9060 |
| hepBindAddress | Address to bind HEP server (UDP/TCP) | 0.0.0.0 |
| influxDbUrl | InfluxDB server URL | http://localhost:7971 |
| influxDbDatabase | InfluxDB database name | hep |
| batchSize | Number of records to batch before sending | 1000 |
| flushInterval | Maximum time between flushes (ms) | 5000 |
| maxBufferSize | Maximum buffer size before forced flush | 10000 |
| debug | Enable debug logging | false |
| writeToFile | Save Line Protocol to files | false |
| outputDir | Directory for output files | ./data |
| http2Port | Port for HTTP/2 HEPv3 ingestion (optional) | undefined |
| http2BindAddress | Address for HTTP/2 server (optional) | undefined |
| http2Endpoint | HTTP/2 POST endpoint path (optional) | undefined |

- **To enable HTTP/2 ingestion, set all three: `http2Port`, `http2BindAddress`, and `http2Endpoint`.**
- If not set, HTTP/2 server will not start.

## Testing All Sockets Locally

- **UDP**: Use a HEP generator (e.g., hepgen) or a tool that can send HEPv3 packets to `localhost:9060` (UDP).
- **TCP**: Use a HEP generator or netcat to send HEPv3 packets to `localhost:9060` (TCP).
- **HTTP/2**: Use the curl example above, or any HTTP/2 client, to POST a raw HEPv3 binary to `http://localhost:8080/test/api`.

## Components

1. **hep-proto**: Core library for converting HEP packets to Line Protocol
Expand Down Expand Up @@ -121,13 +197,36 @@ const server = new HepToInfluxDBServer({
influxDbDatabase: 'hep',
batchSize: 1000,
flushInterval: 5000,
debug: true
debug: true,
// Enable HTTP/2 HEPv3 ingestion (optional)
http2Port: 8080, // HTTP/2 port
http2BindAddress: '0.0.0.0', // HTTP/2 bind address
http2Endpoint: '/test/api', // HTTP/2 POST endpoint
});

server.initialize().catch(console.error);
```

## Configuration Options
### HTTP/2 HEPv3 Ingestion Example

If enabled, you can POST raw HEPv3 binary packets to the configured HTTP/2 endpoint:

```http
POST /test/api HTTP/2
Host: localhost:8080
Content-Type: application/octet-stream

<raw HEPv3 binary>
```

Example with curl (using HTTP/2):

```bash
curl --http2-prior-knowledge -X POST \
-H "Content-Type: application/octet-stream" \
--data-binary @packet.bin \
http://localhost:8080/test/api
```

### HepToInfluxDBServer

Expand Down
93 changes: 93 additions & 0 deletions bun.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
{
"lockfileVersion": 1,
"workspaces": {
"": {
"name": "hep2gig",
"dependencies": {
"@duckdb/node-api": "^1.2.2-alpha.18",
"axios": "^1.8.4",
"hep-js": "^1.0.22",
"parsip": "^1.0.7",
},
},
},
"packages": {
"@duckdb/node-api": ["@duckdb/[email protected]", "", { "dependencies": { "@duckdb/node-bindings": "1.2.2-alpha.19" } }, "sha512-qH/PqK+Kp7TDZJU5SdS9+2jrYOJGHME4NcDsgrcbZk7fKTlXsDI8f8F0CnPxGCsBD35B+BEgphz9TTLoj1/vNA=="],

"@duckdb/node-bindings": ["@duckdb/[email protected]", "", { "optionalDependencies": { "@duckdb/node-bindings-darwin-arm64": "1.2.2-alpha.19", "@duckdb/node-bindings-darwin-x64": "1.2.2-alpha.19", "@duckdb/node-bindings-linux-arm64": "1.2.2-alpha.19", "@duckdb/node-bindings-linux-x64": "1.2.2-alpha.19", "@duckdb/node-bindings-win32-x64": "1.2.2-alpha.19" } }, "sha512-EZpUPCPNJaQVmI3jWa/SIrs7lV9SJYS5a1bV3SRSQ6ZBJYFMrr2Mik2RjCVepA9cIwD2KIrXaWV7jiSVM29zJA=="],

"@duckdb/node-bindings-darwin-arm64": ["@duckdb/[email protected]", "", { "os": "darwin", "cpu": "arm64" }, "sha512-r+MDv7XFxl4eCZEthmHWV2mxnczUFVc+w2cH0KzdTWmCgtjT3jpDhEulDekOG9SP9lFvfsTRTRgKatE2TTqu6w=="],

"@duckdb/node-bindings-darwin-x64": ["@duckdb/[email protected]", "", { "os": "darwin", "cpu": "x64" }, "sha512-WibHtKNQVDFr8AQumML9HLdgkYIRiC0B+MSrLvfojcrYmupoHB0Nwyn/T2UoTXNnTsn3NOrZECHUZGwRFmmF9g=="],

"@duckdb/node-bindings-linux-arm64": ["@duckdb/[email protected]", "", { "os": "linux", "cpu": "arm64" }, "sha512-TfQ/k4YMAsDFpI5Y/nvA7yXs7KqzKsOZX4XzCyqWdfNPDQvheW+wRCxJVUE8kWqniMep62EkvbuX94yFtTZb8A=="],

"@duckdb/node-bindings-linux-x64": ["@duckdb/[email protected]", "", { "os": "linux", "cpu": "x64" }, "sha512-KDGD50H/sA/HWZFVuZhZTVZEKB3N8+ZPhHxxPs67p46WNm/fiQ6/FlaLasQ/gEmOY+I3bWcwjrba4h6LhFXnAQ=="],

"@duckdb/node-bindings-win32-x64": ["@duckdb/[email protected]", "", { "os": "win32", "cpu": "x64" }, "sha512-jsHu/tORVBfJ6QQ02c1vVKEJM6aVSNwwmFtd1G7n9HGjJeE1Wclewpfp3k7R7qU9vxtQtHoAMHeTVsnC2d3ymQ=="],

"asynckit": ["[email protected]", "", {}, "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="],

"axios": ["[email protected]", "", { "dependencies": { "follow-redirects": "^1.15.6", "form-data": "^4.0.0", "proxy-from-env": "^1.1.0" } }, "sha512-/1xYAC4MP/HEG+3duIhFr4ZQXR4sQXOIe+o6sdqzeykGLx6Upp/1p8MHqhINOvGeP7xyNHe7tsiJByc4SSVUxw=="],

"binary-parser": ["[email protected]", "", { "dependencies": { "fast-text-encoding": "^1.0.3" } }, "sha512-aQvTapiRIV/x7bAaR2KT3Ptyjff4R8i1zxljgw/Je2kUt0babNPUdtuo7AKN4CWklyzPK4u5e99Z9/3Ib03u9w=="],

"call-bind-apply-helpers": ["[email protected]", "", { "dependencies": { "es-errors": "^1.3.0", "function-bind": "^1.1.2" } }, "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ=="],

"combined-stream": ["[email protected]", "", { "dependencies": { "delayed-stream": "~1.0.0" } }, "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg=="],

"debug": ["[email protected]", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-KcKCqiftBJcZr++7ykoDIEwSa3XWowTfNPo92BYxjXiyYEVrUQh2aLyhxBCwww+heortUFxEJYcRzosstTEBYQ=="],

"delayed-stream": ["[email protected]", "", {}, "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ=="],

"dunder-proto": ["[email protected]", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.1", "es-errors": "^1.3.0", "gopd": "^1.2.0" } }, "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A=="],

"es-define-property": ["[email protected]", "", {}, "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g=="],

"es-errors": ["[email protected]", "", {}, "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw=="],

"es-object-atoms": ["[email protected]", "", { "dependencies": { "es-errors": "^1.3.0" } }, "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA=="],

"es-set-tostringtag": ["[email protected]", "", { "dependencies": { "es-errors": "^1.3.0", "get-intrinsic": "^1.2.6", "has-tostringtag": "^1.0.2", "hasown": "^2.0.2" } }, "sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA=="],

"fast-text-encoding": ["[email protected]", "", {}, "sha512-VhXlQgj9ioXCqGstD37E/HBeqEGV/qOD/kmbVG8h5xKBYvM1L3lR1Zn4555cQ8GkYbJa8aJSipLPndE1k6zK2w=="],

"follow-redirects": ["[email protected]", "", {}, "sha512-gew4GsXizNgdoRyqmyfMHyAmXsZDk6mHkSxZFCzW9gwlbtOW44CDtYavM+y+72qD/Vq2l550kMF52DT8fOLJqQ=="],

"form-data": ["[email protected]", "", { "dependencies": { "asynckit": "^0.4.0", "combined-stream": "^1.0.8", "es-set-tostringtag": "^2.1.0", "hasown": "^2.0.2", "mime-types": "^2.1.12" } }, "sha512-qsITQPfmvMOSAdeyZ+12I1c+CKSstAFAwu+97zrnWAbIr5u8wfsExUzCesVLC8NgHuRUqNN4Zy6UPWUTRGslcA=="],

"function-bind": ["[email protected]", "", {}, "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA=="],

"get-intrinsic": ["[email protected]", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.2", "es-define-property": "^1.0.1", "es-errors": "^1.3.0", "es-object-atoms": "^1.1.1", "function-bind": "^1.1.2", "get-proto": "^1.0.1", "gopd": "^1.2.0", "has-symbols": "^1.1.0", "hasown": "^2.0.2", "math-intrinsics": "^1.1.0" } }, "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ=="],

"get-proto": ["[email protected]", "", { "dependencies": { "dunder-proto": "^1.0.1", "es-object-atoms": "^1.0.0" } }, "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g=="],

"gopd": ["[email protected]", "", {}, "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg=="],

"has-symbols": ["[email protected]", "", {}, "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ=="],

"has-tostringtag": ["[email protected]", "", { "dependencies": { "has-symbols": "^1.0.3" } }, "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw=="],

"hasown": ["[email protected]", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="],

"hep-js": ["[email protected]", "", { "dependencies": { "binary-parser": "^1.3.2", "mixin-deep": "^2.0.1" } }, "sha512-o5VQpr50FLFjJlFIEaq0voYMItHVWUEASnGyVf7bWxV1Ew9agFFEtSjeMDVby1UJEHf2RZkpnferGHPIP6QGDA=="],

"jwt-decode": ["[email protected]", "", {}, "sha512-+KJGIyHgkGuIq3IEBNftfhW/LfWhXUIY6OmyVWjliu5KH1y0fw7VQ8YndE2O4qZdMSd9SqbnC8GOcZEy0Om7sA=="],

"math-intrinsics": ["[email protected]", "", {}, "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g=="],

"mime-db": ["[email protected]", "", {}, "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg=="],

"mime-types": ["[email protected]", "", { "dependencies": { "mime-db": "1.52.0" } }, "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw=="],

"mixin-deep": ["[email protected]", "", {}, "sha512-imbHQNRglyaplMmjBLL3V5R6Bfq5oM+ivds3SKgc6oRtzErEnBUUc5No11Z2pilkUvl42gJvi285xTNswcKCMA=="],

"ms": ["[email protected]", "", {}, "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="],

"parsip": ["[email protected]", "", { "dependencies": { "debug": "^4.3.4", "jwt-decode": "^4.0.0", "sdp-transform": "^2.4.0" } }, "sha512-XFIQaFKVu1vMBYF36mQRn08Ns8RYW+D2NQZqqxDE5ju7CQp5OItmL+g4fxCHW57PV3uvPkck/9aX9A5IVZfIgg=="],

"proxy-from-env": ["[email protected]", "", {}, "sha512-D+zkORCbA9f1tdWRK0RaCR3GPv50cMxcrz4X8k5LTSUD1Dkw47mKJEZQNunItRTkWwgtaUSo1RVFRIG9ZXiFYg=="],

"sdp-transform": ["[email protected]", "", { "bin": { "sdp-verify": "checker.js" } }, "sha512-KrOH82c/W+GYQ0LHqtr3caRpM3ITglq3ljGUIb8LTki7ByacJZ9z+piSGiwZDsRyhQbYBOBJgr2k6X4BZXi3Kw=="],
}
}
122 changes: 122 additions & 0 deletions example.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,125 @@ async function main() {

// Run the example
main().catch(console.error);

// --- Smarter Socket Test Section ---
async function isPortOpen(port, host, protocol = 'tcp') {
if (protocol === 'tcp') {
const net = await import('net');
return await new Promise((resolve) => {
const socket = net.createConnection({ port, host });
socket.setTimeout(1000);
socket.on('connect', () => {
socket.destroy();
resolve(true);
});
socket.on('timeout', () => {
socket.destroy();
resolve(false);
});
socket.on('error', () => {
resolve(false);
});
});
} else if (protocol === 'udp') {
// UDP is connectionless, so we just try to send
return true;
}
return false;
}

async function testAllSocketsSmart() {
const packet = createTestHepPacket();
const udpPort = 9060;
const tcpPort = 9060;
const http2Port = 8080;
const http2Endpoint = '/test/api';
const host = '127.0.0.1';
const results = [];

// UDP Test
try {
const udpOpen = await isPortOpen(udpPort, host, 'udp');
if (!udpOpen) {
results.push({ protocol: 'UDP', port: udpPort, status: 'DOWN', details: 'UDP port not open' });
} else {
const dgram = await import('dgram');
const udpClient = dgram.createSocket('udp4');
await new Promise((resolve, reject) => {
udpClient.send(packet, udpPort, host, (err) => {
if (err) reject(err);
else resolve();
});
});
udpClient.close();
results.push({ protocol: 'UDP', port: udpPort, status: 'SENT', details: 'Packet sent (no response expected)' });
}
} catch (err) {
results.push({ protocol: 'UDP', port: udpPort, status: 'FAIL', details: err.message });
}

// TCP Test
try {
const tcpOpen = await isPortOpen(tcpPort, host, 'tcp');
if (!tcpOpen) {
results.push({ protocol: 'TCP', port: tcpPort, status: 'DOWN', details: 'TCP port not open' });
} else {
const net = await import('net');
await new Promise((resolve, reject) => {
const client = net.createConnection({ port: tcpPort, host }, () => {
client.write(packet);
client.end();
resolve();
});
client.on('error', reject);
});
results.push({ protocol: 'TCP', port: tcpPort, status: 'SENT', details: 'Packet sent (no response expected)' });
}
} catch (err) {
results.push({ protocol: 'TCP', port: tcpPort, status: 'FAIL', details: err.message });
}

// HTTP/2 Test
try {
const http2Open = await isPortOpen(http2Port, host, 'tcp');
if (!http2Open) {
results.push({ protocol: 'HTTP/2', port: http2Port, status: 'DOWN', details: 'HTTP/2 port not open' });
} else {
let fetchImpl = globalThis.fetch;
if (!fetchImpl) fetchImpl = (await import('node-fetch')).default;
const url = `http://${host}:${http2Port}${http2Endpoint}`;
const res = await fetchImpl(url, {
method: 'POST',
headers: { 'Content-Type': 'application/octet-stream' },
body: packet
});
const body = await res.text();
if (res.status === 200) {
results.push({ protocol: 'HTTP/2', port: http2Port, status: 'OK', details: `200 OK: ${body}` });
} else {
results.push({ protocol: 'HTTP/2', port: http2Port, status: 'FAIL', details: `${res.status}: ${body}` });
}
}
} catch (err) {
results.push({ protocol: 'HTTP/2', port: http2Port, status: 'FAIL', details: err.message });
}

// Print summary table
console.log('\n=== Socket Test Summary ===');
console.log('| Protocol | Port | Status | Details');
console.log('|----------|-------|--------|------------------------------------------');
for (const r of results) {
console.log(`| ${r.protocol.padEnd(8)} | ${String(r.port).padEnd(5)} | ${r.status.padEnd(6)} | ${r.details}`);
}
console.log('===========================================\n');
}

// If run with 'test' argument, run the smarter socket test
if (process.argv.includes('test')) {
testAllSocketsSmart().then(() => {
process.exit(0);
}).catch((err) => {
console.error('Socket test error:', err);
process.exit(1);
});
}
Loading