diff --git a/README.md b/README.md index 3a52e56..ff2d719 100644 --- a/README.md +++ b/README.md @@ -9,18 +9,20 @@ The client requires Node.js v20 or newer version. ```shell # With npm npm i -s @questdb/nodejs-client + # With yarn yarn add @questdb/nodejs-client + # With pnpm pnpm add @questdb/nodejs-client ``` ## Compatibility table -| QuestDB version | Node.js client version | HTTP Agent | -| --------------- | ---------------------- | ------------ | -| ^4.0.0 | >=v20.X.X | Undici Agent | -| ^3.0.0 | =v20.X.X | Undici Agent | +| ^3.0.0 | { // create a sender using HTTPS protocol with bearer token authentication const sender: Sender = Sender.fromConfig( - "https::addr=127.0.0.1:9000;token=Xyvd3er6GF87ysaHk;", + "https::addr=127.0.0.1:9000;token=Xyvd3er6GF87ysaHk", ); // send the data over the authenticated and secure connection diff --git a/docs/HttpTransport.html b/docs/HttpTransport.html new file mode 100644 index 0000000..a40ec94 --- /dev/null +++ b/docs/HttpTransport.html @@ -0,0 +1,445 @@ + + + + + JSDoc: Class: HttpTransport + + + + + + + + + + +
+ +

Class: HttpTransport

+ + + + + + +
+ +
+ +

HttpTransport(options)

+ +
HTTP transport implementation using Node.js built-in http/https modules.
+Supports both HTTP and HTTPS protocols with configurable authentication.
+ + +
+ +
+
+ + + + +

Constructor

+ + + +

new HttpTransport(options)

+ + + + + + +
+ Creates a new HttpTransport instance using Node.js HTTP modules. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
options + + +SenderOptions + + + + Sender configuration object containing connection details
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + +
Throws:
+ + + +
+ + Error if the protocol is not 'http' or 'https' + +
+ + + + + + + + + + + +
+ + + + + + + + + + + + + + + + +

Methods

+ + + + + + + +

send(data, retryBegin, retryInterval)

+ + + + + + +
+ Sends data to QuestDB using HTTP POST. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
data + + +Buffer + + + + Buffer containing the data to send
retryBegin + + +number + + + + Internal parameter for tracking retry start time
retryInterval + + +number + + + + Internal parameter for tracking retry intervals
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + +
Throws:
+ + + +
+ + Error if request fails after all retries or times out + +
+ + + + + +
Returns:
+ + +
+ Promise resolving to true if data was sent successfully +
+ + + + + + + + + + + + + + + +
+ +
+ + + + +
+ + + +
+ +
+ Documentation generated by JSDoc 4.0.4 on Thu Aug 07 2025 12:45:55 GMT+0100 (British Summer Time) +
+ + + + + \ No newline at end of file diff --git a/docs/HttpTransportBase.html b/docs/HttpTransportBase.html new file mode 100644 index 0000000..e0e0464 --- /dev/null +++ b/docs/HttpTransportBase.html @@ -0,0 +1,548 @@ + + + + + JSDoc: Class: HttpTransportBase + + + + + + + + + + +
+ +

Class: HttpTransportBase

+ + + + + + +
+ +
+ +

HttpTransportBase(options)

+ +
Abstract base class for HTTP-based transport implementations.
+Provides common configuration and functionality for HTTP and HTTPS protocols.
+ + +
+ +
+
+ + + + +

Constructor

+ + + +

new HttpTransportBase(options)

+ + + + + + +
+ Creates a new HttpTransportBase instance. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
options + + +SenderOptions + + + + Sender configuration options including connection and authentication details
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + +
Throws:
+ + + +
+ + Error if required protocol or host options are missing + +
+ + + + + + + + + + + +
+ + + + + + + + + + + + + + + + +

Methods

+ + + + + + + +

(async) close()

+ + + + + + +
+ HTTP transport does not require explicit connection closure. +
+ + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + +
Returns:
+ + +
+ Promise that resolves immediately +
+ + + + + + + + + + + + + + + +

connect()

+ + + + + + +
+ HTTP transport does not require explicit connection establishment. +
+ + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + +
Throws:
+ + + +
+ + Error indicating connect is not required for HTTP transport + +
+ + + + + + + + + + + + + + + + +

getDefaultAutoFlushRows() → {number}

+ + + + + + +
+ Gets the default auto-flush row count for HTTP transport. +
+ + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + +
Returns:
+ + +
+ Default number of rows that trigger auto-flush +
+ + + +
+
+ Type +
+
+ +number + + +
+
+ + + + + + + + + + + + + +
+ +
+ + + + +
+ + + +
+ +
+ Documentation generated by JSDoc 4.0.4 on Thu Aug 07 2025 12:45:55 GMT+0100 (British Summer Time) +
+ + + + + \ No newline at end of file diff --git a/docs/Sender.html b/docs/Sender.html index 0d8d510..09ab939 100644 --- a/docs/Sender.html +++ b/docs/Sender.html @@ -30,9 +30,19 @@

Class: Sender

Sender(options)

-
The QuestDB client's API provides methods to connect to the database, ingest data, and close the connection. -The supported protocols are HTTP and TCP. HTTP is preferred as it provides feedback in the HTTP response.
-Based on benchmarks HTTP also provides higher throughput, if configured to ingest data in bigger batches. +
The QuestDB client's API provides methods to connect to the database, ingest data, and close the connection.
+The client supports multiple transport protocols. +

+Transport Options: +

    +
  • HTTP: Uses standard HTTP requests for data ingestion. Provides immediate feedback via HTTP response codes. +Recommended for most use cases due to superior error handling and debugging capabilities. Uses Undici library by default for high performance.
  • +
  • HTTPS: Secure HTTP transport with TLS encryption. Same benefits as HTTP but with encrypted communication. +Supports certificate validation and custom CA certificates.
  • +
  • TCP: Direct TCP connection, provides persistent connections. Uses JWK token-based authentication.
  • +
  • TCPS: Secure TCP transport with TLS encryption.
  • +
+

The client supports authentication.
Authentication details can be passed to the Sender in its configuration options.
@@ -48,19 +58,41 @@

Sender

+The client supports multiple protocol versions for data serialization. Protocol version 1 uses text-based +serialization, while version 2 uses binary encoding for doubles and supports array columns for improved +performance. The client can automatically negotiate the protocol version with the server when using HTTP/HTTPS +by setting the protocol_version to 'auto' (default behavior). +

+

The client uses a buffer to store data. It automatically flushes the buffer by sending its content to the server. Auto flushing can be disabled via configuration options to gain control over transactions. Initial and maximum buffer sizes can also be set.

It is recommended that the Sender is created by using one of the static factory methods, -Sender.fromConfig(configString, extraOptions) or Sender.fromEnv(extraOptions)). +Sender.fromConfig(configString, extraOptions) or Sender.fromEnv(extraOptions). If the Sender is created via its constructor, at least the SenderOptions configuration object should be initialized from a configuration string to make sure that the parameters are validated.
Detailed description of the Sender's configuration options can be found in the SenderOptions documentation.

+Transport Configuration Examples: +

    +
  • HTTP: Sender.fromConfig("http::addr=localhost:9000")
  • +
  • HTTPS with authentication: Sender.fromConfig("https::addr=localhost:9000;username=admin;password=secret")
  • +
  • TCP: Sender.fromConfig("tcp::addr=localhost:9009")
  • +
  • TCPS with authentication: Sender.fromConfig("tcps::addr=localhost:9009;username=user;token=private_key")
  • +
+

+

+HTTP Transport Implementation:
+By default, HTTP/HTTPS transport uses the high-performance Undici library for connection management and request handling. +For compatibility or specific requirements, you can enable the standard HTTP transport using Node.js built-in modules +by setting stdlib_http=on in the configuration string. The standard HTTP transport provides the same functionality +but uses Node.js http/https modules instead of Undici. +

+

Extra options can be provided to the Sender in the extraOptions configuration object.
A custom logging function and a custom HTTP(S) agent can be passed to the Sender in this object.
The logger implementation provides the option to direct log messages to the same place where the host application's @@ -147,7 +179,7 @@

Parameters:
Sender configuration object.
-See SenderOptions documentation for detailed description of configuration options.
+See SenderOptions documentation for detailed description of configuration options. @@ -188,7 +220,7 @@
Parameters:
Source:
@@ -244,6 +276,203 @@

Methods

+

arrayColumn(name, value) → {Sender}

+ + + + + + +
+ Writes an array column with its values into the buffer of the sender. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
name + + +string + + + + Column name
value + + +Array.<unknown> + + + + Array values to write (currently supports double arrays)
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + +
Throws:
+ + + +
+ + Error if arrays are not supported by the buffer implementation, or array validation fails: +- value is not an array +- or the shape of the array is irregular: the length of sub-arrays are different +- or the array is not homogeneous: its elements are not all the same type + +
+ + + + + +
Returns:
+ + +
+ Returns with a reference to this sender. +
+ + + +
+
+ Type +
+
+ +Sender + + +
+
+ + + + + + + + + + + + +

(async) at(timestamp, unitopt)

@@ -252,7 +481,7 @@

(async) at - Closing the row after writing the designated timestamp into the buffer of the sender. + Closes the row after writing the designated timestamp into the buffer of the sender.

@@ -403,7 +632,7 @@
Parameters:
Source:
@@ -447,7 +676,7 @@

(async) atNow - Closing the row without writing designated timestamp into the buffer of the sender.
+ Closes the row without writing designated timestamp into the buffer of the sender.
Designated timestamp will be populated by the server on this record.

@@ -492,7 +721,7 @@

(async) atNowSource:
@@ -536,7 +765,8 @@

booleanC
- Write a boolean column with its value into the buffer of the sender. + Writes a boolean column with its value into the buffer of the sender.
+Use it to insert into BOOLEAN columns.
@@ -652,7 +882,7 @@

Parameters:
Source:
@@ -718,7 +948,7 @@

(async) close - Closes the TCP connection to the database.
+ Closes the connection to the database.
Data sitting in the Sender's buffer will be lost unless flush() is called before close(). @@ -763,7 +993,7 @@

(async) closeSource:
@@ -799,7 +1029,7 @@

(async) closeconnect(connectOptions) → {Promise.<boolean>}

+

connect() → {Promise.<boolean>}

@@ -818,58 +1048,6 @@

connectParameters:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NameTypeDescription
connectOptions - - -net.NetConnectOpts -| - -tls.ConnectionOptions - - - - Connection options, host and port are required.
- - @@ -903,7 +1081,7 @@
Parameters:
Source:
@@ -969,7 +1147,8 @@

floatColum
- Write a float column with its value into the buffer of the sender. + Writes a 64-bit floating point value into the buffer of the sender.
+Use it to insert into DOUBLE or FLOAT database columns.
@@ -1085,7 +1264,7 @@

Parameters:
Source:
@@ -1151,7 +1330,7 @@

(async) flush - Sends the buffer's content to the database and compacts the buffer. + Sends the content of the sender's buffer to the database and compacts the buffer. If the last row is not finished it stays in the sender's buffer. @@ -1196,7 +1375,7 @@

(async) flushSource:
@@ -1225,7 +1404,7 @@
Returns:
- Resolves to true when there was data in the buffer to send. + Resolves to true when there was data in the buffer to send, and it was sent successfully.
@@ -1262,7 +1441,8 @@

intColumn - Write an integer column with its value into the buffer of the sender. + Writes a 64-bit signed integer into the buffer of the sender.
+Use it to insert into LONG, INT, SHORT and BYTE columns. @@ -1378,7 +1558,7 @@
Parameters:
Source:
@@ -1401,6 +1581,18 @@
Parameters:
+
Throws:
+ + + +
+ + Error if the value is not an integer + +
+ + +
Returns:
@@ -1444,8 +1636,8 @@

reset - Resets the buffer, data added to the buffer will be lost.
-In other words it clears the buffer and sets the writing position to the beginning of the buffer. + Resets the sender's buffer, data sitting in the buffer will be lost.
+In other words it clears the buffer, and sets the writing position to the beginning of the buffer. @@ -1489,7 +1681,7 @@

resetSource:
@@ -1541,145 +1733,6 @@
Returns:
- - - - - - -

resize(bufferSize)

- - - - - - -
- Extends the size of the sender's buffer.
-Can be used to increase the size of buffer if overflown. -The buffer's content is copied into the new buffer. -
- - - - - - - - - -
Parameters:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NameTypeDescription
bufferSize - - -number - - - - New size of the buffer used by the sender, provided in bytes.
- - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - -
Source:
-
- - - - - - - -
- - - - - - - - - - - - - - - - - - - - @@ -1694,7 +1747,8 @@

stringCol
- Write a string column with its value into the buffer of the sender. + Writes a string column with its value into the buffer of the sender.
+Use it to insert into VARCHAR and STRING columns.
@@ -1810,7 +1864,7 @@

Parameters:
Source:
@@ -1876,7 +1930,8 @@

symbol - Write a symbol name and value into the buffer of the sender. + Writes a symbol name and value into the buffer of the sender.
+Use it to insert into SYMBOL columns. @@ -1941,7 +1996,7 @@
Parameters:
-any +unknown @@ -1951,7 +2006,7 @@
Parameters:
- Symbol value, toString() will be called to extract the actual symbol value from the parameter. + Symbol value, toString() is called to extract the actual symbol value from the parameter. @@ -1992,7 +2047,7 @@
Parameters:
Source:
@@ -2058,7 +2113,7 @@

table - Write the table name into the buffer of the sender. + Writes the table name into the buffer of the sender of the sender. @@ -2151,7 +2206,7 @@
Parameters:
Source:
@@ -2217,7 +2272,8 @@

timest
- Write a timestamp column with its value into the buffer of the sender. + Writes a timestamp column with its value into the buffer of the sender.
+Use it to insert into TIMESTAMP columns.
@@ -2403,7 +2459,7 @@

Parameters:
Source:
@@ -2461,7 +2517,7 @@
Returns:
-

(static) fromConfig(configurationString, extraOptions) → {Sender}

+

(async, static) fromConfig(configurationString, extraOptions) → {Sender}

@@ -2469,7 +2525,7 @@

(static) f
- Creates a Sender options object by parsing the provided configuration string. + Creates a Sender object by parsing the provided configuration string.
@@ -2547,8 +2603,8 @@

Parameters:
Optional extra configuration.
- 'log' is a logging function used by the Sender.
Prototype: (level: 'error'|'warn'|'info'|'debug', message: string) => void.
-- 'agent' is a custom Undici agent used by the Sender when http/https transport is used.
-A undici.Agent object is expected. +- 'agent' is a custom http/https agent used by the Sender when http/https transport is used.
+Depends on which transport implementation and protocol used, one of the followings expected: undici.Agent, http.Agent or https.Agent. @@ -2589,7 +2645,7 @@
Parameters:
Source:
@@ -2647,7 +2703,7 @@
Returns:
-

(static) fromEnv(extraOptions) → {Sender}

+

(async, static) fromEnv(extraOptions) → {Sender}

@@ -2655,7 +2711,7 @@

(static) from
- Creates a Sender options object by parsing the configuration string set in the QDB_CLIENT_CONF environment variable. + Creates a Sender object by parsing the configuration string set in the QDB_CLIENT_CONF environment variable.
@@ -2710,8 +2766,8 @@

Parameters:
Optional extra configuration.
- 'log' is a logging function used by the Sender.
Prototype: (level: 'error'|'warn'|'info'|'debug', message: string) => void.
-- 'agent' is a custom Undici agent used by the Sender when http/https transport is used.
-A undici.Agent object is expected. +- 'agent' is a custom http/https agent used by the Sender when http/https transport is used.
+Depends on which transport implementation and protocol used, one of the followings expected: undici.Agent, http.Agent or https.Agent. @@ -2752,7 +2808,7 @@
Parameters:
Source:
@@ -2820,13 +2876,13 @@
Returns:

- Documentation generated by JSDoc 4.0.4 on Mon Dec 09 2024 02:16:59 GMT+0100 (hora estándar de Europa central) + Documentation generated by JSDoc 4.0.4 on Thu Aug 07 2025 12:45:55 GMT+0100 (British Summer Time)
diff --git a/docs/SenderBufferBase.html b/docs/SenderBufferBase.html new file mode 100644 index 0000000..8b6d627 --- /dev/null +++ b/docs/SenderBufferBase.html @@ -0,0 +1,2247 @@ + + + + + JSDoc: Class: SenderBufferBase + + + + + + + + + + +
+ +

Class: SenderBufferBase

+ + + + + + +
+ +
+ +

SenderBufferBase(options)

+ +
Abstract base class for SenderBuffer implementations.
+Provides common functionality for writing data into the buffer.
+ + +
+ +
+
+ + + + +

Constructor

+ + + +

new SenderBufferBase(options)

+ + + + + + +
+ Creates an instance of SenderBufferBase. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
options + + +SenderOptions + + + + Sender configuration object.
+See SenderOptions documentation for detailed description of configuration options.
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + +

Methods

+ + + + + + + +

at(timestamp, unitopt)

+ + + + + + +
+ Closes the row after writing the designated timestamp into the buffer. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeAttributesDefaultDescription
timestamp + + +number +| + +bigint + + + + + + + + + + + + Designated epoch timestamp, accepts numbers or BigInts.
unit + + +string + + + + + + <optional>
+ + + + + +
+ + us + + Timestamp unit. Supported values: 'ns' - nanoseconds, 'us' - microseconds, 'ms' - milliseconds. Defaults to 'us'.
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +

atNow()

+ + + + + + +
+ Closes the row without writing designated timestamp into the buffer.
+Designated timestamp will be populated by the server on this record. +
+ + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +

booleanColumn(name, value) → {SenderBuffer}

+ + + + + + +
+ Writes a boolean column with its value into the buffer.
+Use it to insert into BOOLEAN columns. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
name + + +string + + + + Column name.
value + + +boolean + + + + Column value, accepts only boolean values.
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + +
Returns:
+ + +
+ Returns with a reference to this buffer. +
+ + + +
+
+ Type +
+
+ +SenderBuffer + + +
+
+ + + + + + + + + + + + + +

checkCapacity(data, base)

+ + + + + + +
+ Checks if the buffer has sufficient capacity for additional data and resizes if needed. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDefaultDescription
data + + + + Array of strings to calculate the required capacity for
base + + + + 0 + + Base number of bytes to add to the calculation
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +

currentPosition()

+ + + + + + +
+ Returns the current position of the buffer.
+New data will be written into the buffer starting from this position. +
+ + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +

intColumn(name, value) → {SenderBuffer}

+ + + + + + +
+ Writes a 64-bit signed integer into the buffer.
+Use it to insert into LONG, INT, SHORT and BYTE columns. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
name + + +string + + + + Column name.
value + + +number + + + + Column value, accepts only number values.
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + +
Throws:
+ + + +
+ + Error if the value is not an integer + +
+ + + + + +
Returns:
+ + +
+ Returns with a reference to this buffer. +
+ + + +
+
+ Type +
+
+ +SenderBuffer + + +
+
+ + + + + + + + + + + + + +

reset() → {SenderBuffer}

+ + + + + + +
+ Resets the buffer, data sitting in the buffer will be lost.
+In other words it clears the buffer, and sets the writing position to the beginning of the buffer. +
+ + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + +
Returns:
+ + +
+ Returns with a reference to this buffer. +
+ + + +
+
+ Type +
+
+ +SenderBuffer + + +
+
+ + + + + + + + + + + + + +

stringColumn(name, value) → {SenderBuffer}

+ + + + + + +
+ Writes a string column with its value into the buffer.
+Use it to insert into VARCHAR and STRING columns. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
name + + +string + + + + Column name.
value + + +string + + + + Column value, accepts only string values.
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + +
Returns:
+ + +
+ Returns with a reference to this buffer. +
+ + + +
+
+ Type +
+
+ +SenderBuffer + + +
+
+ + + + + + + + + + + + + +

symbol(name, value) → {SenderBuffer}

+ + + + + + +
+ Writes a symbol name and value into the buffer.
+Use it to insert into SYMBOL columns. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
name + + +string + + + + Symbol name.
value + + +unknown + + + + Symbol value, toString() is called to extract the actual symbol value from the parameter.
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + +
Returns:
+ + +
+ Returns with a reference to this buffer. +
+ + + +
+
+ Type +
+
+ +SenderBuffer + + +
+
+ + + + + + + + + + + + + +

table(table) → {SenderBuffer}

+ + + + + + +
+ Writes the table name into the buffer. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
table + + +string + + + + Table name.
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + +
Returns:
+ + +
+ Returns with a reference to this buffer. +
+ + + +
+
+ Type +
+
+ +SenderBuffer + + +
+
+ + + + + + + + + + + + + +

timestampColumn(name, value, unitopt) → {SenderBuffer}

+ + + + + + +
+ Writes a timestamp column with its value into the buffer.
+Use it to insert into TIMESTAMP columns. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeAttributesDefaultDescription
name + + +string + + + + + + + + + + + + Column name.
value + + +number +| + +bigint + + + + + + + + + + + + Epoch timestamp, accepts numbers or BigInts.
unit + + +string + + + + + + <optional>
+ + + + + +
+ + us + + Timestamp unit. Supported values: 'ns' - nanoseconds, 'us' - microseconds, 'ms' - milliseconds. Defaults to 'us'.
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + +
Returns:
+ + +
+ Returns with a reference to this buffer. +
+ + + +
+
+ Type +
+
+ +SenderBuffer + + +
+
+ + + + + + + + + + + + + +

toBufferNew() → {Buffer}

+ + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + +
Returns:
+ + +
+ Returns a cropped buffer ready to send to the server, or null if there is nothing to send.
+The returned buffer is a copy of this buffer. +It also compacts the buffer. +
+ + + +
+
+ Type +
+
+ +Buffer + + +
+
+ + + + + + + + + + + + + +

toBufferView() → {Buffer}

+ + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + +
Returns:
+ + +
+ Returns a cropped buffer, or null if there is nothing to send.
+The returned buffer is backed by this buffer instance, meaning the view can change as the buffer is mutated. +Used only in tests to assert the buffer's content. +
+ + + +
+
+ Type +
+
+ +Buffer + + +
+
+ + + + + + + + + + + + + +
+ +
+ + + + +
+ + + +
+ +
+ Documentation generated by JSDoc 4.0.4 on Thu Aug 07 2025 12:45:55 GMT+0100 (British Summer Time) +
+ + + + + \ No newline at end of file diff --git a/docs/SenderBufferV1.html b/docs/SenderBufferV1.html new file mode 100644 index 0000000..db324f9 --- /dev/null +++ b/docs/SenderBufferV1.html @@ -0,0 +1,512 @@ + + + + + JSDoc: Class: SenderBufferV1 + + + + + + + + + + +
+ +

Class: SenderBufferV1

+ + + + + + +
+ +
+ +

SenderBufferV1(options)

+ +
Buffer implementation for protocol version 1. +Sends floating point numbers in their text form.
+ + +
+ +
+
+ + + + +

Constructor

+ + + +

new SenderBufferV1(options)

+ + + + + + +
+ Creates a new SenderBufferV1 instance. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
options + + +SenderOptions + + + + Sender configuration object.
+See SenderOptions documentation for detailed description of configuration options.
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + +

Methods

+ + + + + + + +

arrayColumn()

+ + + + + + +
+ Array columns are not supported in protocol v1. +
+ + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + +
Throws:
+ + + +
+ + Error indicating arrays are not supported in v1 + +
+ + + + + + + + + + + + + + + + +

floatColumn(name, value) → {Sender}

+ + + + + + +
+ Writes a 64-bit floating point value into the buffer using v1 serialization (text format).
+Use it to insert into DOUBLE or FLOAT database columns. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
name + + +string + + + + Column name.
value + + +number + + + + Column value, accepts only number values.
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + +
Returns:
+ + +
+ Returns with a reference to this sender. +
+ + + +
+
+ Type +
+
+ +Sender + + +
+
+ + + + + + + + + + + + + +
+ +
+ + + + +
+ + + +
+ +
+ Documentation generated by JSDoc 4.0.4 on Thu Aug 07 2025 12:45:55 GMT+0100 (British Summer Time) +
+ + + + + \ No newline at end of file diff --git a/docs/SenderBufferV2.html b/docs/SenderBufferV2.html new file mode 100644 index 0000000..796eed0 --- /dev/null +++ b/docs/SenderBufferV2.html @@ -0,0 +1,609 @@ + + + + + JSDoc: Class: SenderBufferV2 + + + + + + + + + + +
+ +

Class: SenderBufferV2

+ + + + + + +
+ +
+ +

SenderBufferV2(options)

+ +
Buffer implementation for protocol version 2. +Sends floating point numbers in binary form.
+ + +
+ +
+
+ + + + +

Constructor

+ + + +

new SenderBufferV2(options)

+ + + + + + +
+ Creates a new SenderBufferV2 instance. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
options + + +SenderOptions + + + + Sender configuration object.
+See SenderOptions documentation for detailed description of configuration options.
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + +

Methods

+ + + + + + + +

arrayColumn(name, value) → {Sender}

+ + + + + + +
+ Write an array column with its values into the buffer using v2 format. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
name + + +string + + + + Column name
value + + +Array.<unknown> + + + + Array values to write (currently supports double arrays)
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + +
Throws:
+ + + +
+ + Error if array validation fails: +- value is not an array +- or the shape of the array is irregular: the length of sub-arrays are different +- or the array is not homogeneous: its elements are not all the same type + +
+ + + + + +
Returns:
+ + +
+ Returns with a reference to this buffer. +
+ + + +
+
+ Type +
+
+ +Sender + + +
+
+ + + + + + + + + + + + + +

floatColumn(name, value) → {Sender}

+ + + + + + +
+ Writes a 64-bit floating point value into the buffer using v2 serialization (binary format).
+Use it to insert into DOUBLE or FLOAT database columns. +
+ + + + + + + + + +
Parameters:
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameTypeDescription
name + + +string + + + + Column name.
value + + +number + + + + Column value, accepts only number values.
+ + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
Source:
+
+ + + + + + + +
+ + + + + + + + + + + + + + + +
Returns:
+ + +
+ Returns with a reference to this buffer. +
+ + + +
+
+ Type +
+
+ +Sender + + +
+
+ + + + + + + + + + + + + +
+ +
+ + + + +
+ + + +
+ +
+ Documentation generated by JSDoc 4.0.4 on Thu Aug 07 2025 12:45:55 GMT+0100 (British Summer Time) +
+ + + + + \ No newline at end of file diff --git a/docs/SenderOptions.html b/docs/SenderOptions.html index 6bfe165..db5500e 100644 --- a/docs/SenderOptions.html +++ b/docs/SenderOptions.html @@ -35,13 +35,18 @@

SenderOptio Properties of the object are initialized through a configuration string.
The configuration string has the following format: <protocol>::<key>=<value><key>=<value>...;
The keys are case-sensitive, the trailing semicolon is optional.
-The values are validated, and an error is thrown if the format is invalid.
+The values are validated and an error is thrown if the format is invalid.

Connection and protocol options
  • protocol: enum, accepted values: http, https, tcp, tcps - The protocol used to communicate with the server.
    When https or tcps used, the connection is secured with TLS encryption.
  • +
  • protocol_version: enum, accepted values: auto, 1, 2 - The protocol version used for data serialization.
    +Version 1 uses text-based serialization for all data types. Version 2 uses binary encoding for doubles and arrays.
    +When set to 'auto' (default for HTTP/HTTPS), the client automatically negotiates the highest supported version with the server.
    +TCP/TCPS connections default to version 1. +
  • addr: string - Hostname and port, separated by colon. This key is mandatory, but the port part is optional.
    If no port is specified, a default will be used.
    When the protocol is HTTP/HTTPS, the port defaults to 9000. When the protocol is TCP/TCPS, the port defaults to 9009.
    @@ -67,8 +72,8 @@

    SenderOptio TLS options
    • tls_verify: enum, accepted values: on, unsafe_off - When the HTTPS or TCPS protocols are selected, TLS encryption is used.
      -By default, the Sender will verify the server's certificate, but this check can be disabled by setting this option to off. This is useful -non-production environments where self-signed certificates might be used, but should be avoided in production if possible. +By default, the Sender will verify the server's certificate, but this check can be disabled by setting this option to unsafe_off.
      +This is useful in non-production environments where self-signed certificates might be used, but should be avoided in production if possible.
    • tls_ca: string - Path to a file containing the root CA's certificate in PEM format.
      Can be useful when self-signed certificates are used, otherwise should not be set. @@ -80,9 +85,9 @@

      SenderOptio
    • auto_flush: enum, accepted values: on, off - The Sender automatically flushes the buffer by default. This can be switched off by setting this option to off.
      When disabled, the flush() method of the Sender has to be called explicitly to make sure data is sent to the server.
      -Manual buffer flushing can be useful, especially when we want to use transactions. When the HTTP protocol is used, each flush results in a single HTTP -request, which becomes a single transaction on the server side. The transaction either succeeds, and all rows sent in the request are -inserted; or it fails, and none of the rows make it into the database. +Manual buffer flushing can be useful, especially when we want to control transaction boundaries.
      +When the HTTP protocol is used, each flush results in a single HTTP request, which becomes a single transaction on the server side.
      +The transaction either succeeds, and all rows sent in the request are inserted; or it fails, and none of the rows make it into the database.
    • auto_flush_rows: integer - The number of rows that will trigger a flush. When set to 0, row-based flushing is disabled.
      The Sender will default this parameter to 75000 rows when HTTP protocol is used, and to 600 in case of TCP protocol. @@ -118,14 +123,12 @@

      SenderOptio
      Other options
        +
      • stdlib_http: enum, accepted values: on, off - With HTTP protocol the Undici library is used by default. By setting this option +to on the client switches to node's core http and https modules. +
      • max_name_len: integer - The maximum length of a table or column name, the Sender defaults this parameter to 127.
        Recommended to use the same setting as the server, which also uses 127 by default.
      • -
      • copy_buffer: enum, accepted values: on, off - By default, the Sender creates a new buffer for every flush() call, -and the data to be sent to the server is copied into this new buffer. -Setting the flag to off results in reusing the same buffer instance for each flush() call.
        -Use this flag only if calls to the client are serialised. -
      @@ -228,7 +231,7 @@

      Parameters:
      - 'log' is a logging function used by the Sender.
      Prototype: (level: 'error'|'warn'|'info'|'debug', message: string) => void.
      - 'agent' is a custom http/https agent used by the Sender when http/https transport is used.
      -A http.Agent or https.Agent object is expected. +Depends on which transport implementation and protocol used, one of the followings expected: undici.Agent, http.Agent or https.Agent. @@ -269,7 +272,7 @@
      Parameters:
      Source:
      @@ -325,7 +328,7 @@

      Methods

      -

      (static) fromConfig(configurationString, extraOptions) → {SenderOptions}

      +

      (async, static) fromConfig(configurationString, extraOptions) → {SenderOptions}

      @@ -412,7 +415,7 @@
      Parameters:
      - 'log' is a logging function used by the Sender.
      Prototype: (level: 'error'|'warn'|'info'|'debug', message: string) => void.
      - 'agent' is a custom http/https agent used by the Sender when http/https transport is used.
      -A http.Agent or https.Agent object is expected. +Depends on which transport implementation and protocol used, one of the followings expected: undici.Agent, http.Agent or https.Agent. @@ -453,7 +456,7 @@
      Parameters:
      Source:
      @@ -511,7 +514,7 @@
      Returns:
      -

      (static) fromEnv(extraOptions) → {SenderOptions}

      +

      (async, static) fromEnv(extraOptions) → {SenderOptions}

      @@ -573,9 +576,9 @@
      Parameters:
      Optional extra configuration.
      - 'log' is a logging function used by the Sender.
      - }in /**br> +Prototype: (level: 'error'|'warn'|'info'|'debug', message: string) => void.
      - 'agent' is a custom http/https agent used by the Sender when http/https transport is used.
      -A http.Agent or https.Agent object is expected. +Depends on which transport implementation and protocol used, one of the followings expected: undici.Agent, http.Agent or https.Agent. @@ -616,7 +619,7 @@
      Parameters:
      Source:
      @@ -668,6 +671,147 @@
      Returns:
      + + + + + + +

      (async, static) resolveAuto(options)

      + + + + + + +
      + Resolves the protocol version, if it is set to 'auto'.
      +If TCP transport is used, the protocol version will default to 1. +In case of HTTP transport the /settings endpoint of the database is used to find the protocol versions +supported by the server, and the highest will be selected. +When calling the /settings endpoint the timeout and TLs options are used from the options object. +
      + + + + + + + + + +
      Parameters:
      + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      NameTypeDescription
      options + + +SenderOptions + + + + SenderOptions instance needs resolving protocol version
      + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + + + + + + + + @@ -684,13 +828,13 @@
      Returns:

      - Documentation generated by JSDoc 4.0.4 on Mon Dec 09 2024 02:16:59 GMT+0100 (hora estándar de Europa central) + Documentation generated by JSDoc 4.0.4 on Thu Aug 07 2025 12:45:55 GMT+0100 (British Summer Time)
      diff --git a/docs/TcpTransport.html b/docs/TcpTransport.html new file mode 100644 index 0000000..1b49f18 --- /dev/null +++ b/docs/TcpTransport.html @@ -0,0 +1,695 @@ + + + + + JSDoc: Class: TcpTransport + + + + + + + + + + +
      + +

      Class: TcpTransport

      + + + + + + +
      + +
      + +

      TcpTransport(options)

      + +
      TCP transport implementation.
      +Supports both plain TCP or secure TLS-encrypted connections with configurable JWK token authentication.
      + + +
      + +
      +
      + + + + +

      Constructor

      + + + +

      new TcpTransport(options)

      + + + + + + +
      + Creates a new TcpTransport instance. +
      + + + + + + + + + +
      Parameters:
      + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      NameTypeDescription
      options + + +SenderOptions + + + + Sender configuration object containing connection and authentication details
      + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + +
      Throws:
      + + + +
      + + Error if required options are missing or protocol is not 'tcp' or 'tcps' + +
      + + + + + + + + + + + +
      + + + + + + + + + + + + + + + + +

      Methods

      + + + + + + + +

      (async) close()

      + + + + + + +
      + Closes the TCP connection to the database. +
      + + + + + + + + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +

      connect()

      + + + + + + +
      + Creates a TCP connection to the database. +
      + + + + + + + + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + +
      Throws:
      + + + +
      + + Error if connection fails or authentication is rejected + +
      + + + + + +
      Returns:
      + + +
      + Promise resolving to true if the connection is established successfully +
      + + + + + + + + + + + + + + + +

      getDefaultAutoFlushRows()

      + + + + + + +
      + Gets the default auto-flush row count for TCP transport. +
      + + + + + + + + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + + + +
      Returns:
      + + +
      + Default number of rows that trigger auto-flush +
      + + + + + + + + + + + + + + + +

      send(data)

      + + + + + + +
      + Sends data over the established TCP connection. +
      + + + + + + + + + +
      Parameters:
      + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      NameTypeDescription
      data + + +Buffer + + + + Buffer containing the data to send
      + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + +
      Throws:
      + + + +
      + + Error if the data could not be written to the socket + +
      + + + + + +
      Returns:
      + + +
      + Promise resolving to true if data was sent successfully +
      + + + + + + + + + + + + + + + +
      + +
      + + + + +
      + + + +
      + +
      + Documentation generated by JSDoc 4.0.4 on Thu Aug 07 2025 12:45:55 GMT+0100 (British Summer Time) +
      + + + + + \ No newline at end of file diff --git a/docs/UndiciTransport.html b/docs/UndiciTransport.html new file mode 100644 index 0000000..75def4d --- /dev/null +++ b/docs/UndiciTransport.html @@ -0,0 +1,395 @@ + + + + + JSDoc: Class: UndiciTransport + + + + + + + + + + +
      + +

      Class: UndiciTransport

      + + + + + + +
      + +
      + +

      UndiciTransport(options)

      + +
      HTTP transport implementation using the Undici library.
      +Provides high-performance HTTP requests with connection pooling and retry logic.
      +Supports both HTTP and HTTPS protocols with configurable authentication.
      + + +
      + +
      +
      + + + + +

      Constructor

      + + + +

      new UndiciTransport(options)

      + + + + + + +
      + Creates a new UndiciTransport instance. +
      + + + + + + + + + +
      Parameters:
      + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      NameTypeDescription
      options + + Sender configuration object containing connection and retry settings
      + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + +
      Throws:
      + + + +
      + + Error if the protocol is not 'http' or 'https' + +
      + + + + + + + + + + + +
      + + + + + + + + + + + + + + + + +

      Methods

      + + + + + + + +

      (async) send(data)

      + + + + + + +
      + Sends data to QuestDB using HTTP POST. +
      + + + + + + + + + +
      Parameters:
      + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      NameTypeDescription
      data + + +Buffer + + + + Buffer containing the data to send
      + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + +
      Throws:
      + + + +
      + + Error if request fails after all retries or times out + +
      + + + + + +
      Returns:
      + + +
      + Promise resolving to true if data was sent successfully +
      + + + + + + + + + + + + + + + +
      + +
      + + + + +
      + + + +
      + +
      + Documentation generated by JSDoc 4.0.4 on Thu Aug 07 2025 12:45:55 GMT+0100 (British Summer Time) +
      + + + + + \ No newline at end of file diff --git a/docs/global.html b/docs/global.html index 09db35e..b978d88 100644 --- a/docs/global.html +++ b/docs/global.html @@ -98,22 +98,1407 @@

      Members

      -

      (constant) DEFAULT_HTTP_OPTIONS :Agent.Options

      +

      (constant) DEFAULT_HTTP_AGENT_CONFIG

      +
      + Default configuration for HTTP agents. +- Persistent connections with 1 minute idle timeout +- Maximum of 256 open connections (matching server default) +
      + + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + +

      (constant) DEFAULT_HTTP_OPTIONS

      + + + + +
      + Default HTTP options for the Undici agent. +Configures keep-alive connections with 60-second timeout and single request pipelining. +
      + + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + +

      Methods

      + + + + + + + +

      createBuffer(options)

      + + + + + + +
      + Factory function to create a SenderBuffer instance based on the protocol version. +
      + + + + + + + + + +
      Parameters:
      + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      NameTypeDescription
      options + + +SenderOptions + + + + Sender configuration object.
      +See SenderOptions documentation for detailed description of configuration options.
      + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + +
      Throws:
      + + + +
      + + Error if protocol version is not specified or is unsupported + +
      + + + + + +
      Returns:
      + + +
      + A SenderBuffer instance appropriate for the specified protocol version +
      + + + + + + + + + + + + + + + +

      createTransport(options)

      + + + + + + +
      + Factory function to create appropriate transport instance based on configuration. +
      + + + + + + + + + +
      Parameters:
      + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      NameTypeDescription
      options + + +SenderOptions + + + + Sender configuration options including protocol and connection details
      + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + +
      Throws:
      + + + +
      + + Error if protocol or host options are missing or invalid + +
      + + + + + +
      Returns:
      + + +
      + Transport instance appropriate for the specified protocol +
      + + + + + + + + + + + + + + + +

      (async) fetchJson(url, agent, timeout)

      + + + + + + +
      + Fetches JSON data from a URL. +
      + + + + + + + + + +
      Parameters:
      + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      NameTypeDescription
      url + + +string + + + + The URL to fetch from
      agent + + +Agent + + + + HTTP agent to be used for the request
      timeout + + +number + + + + Request timeout, query will be aborted if not finished in time
      + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + +
      Throws:
      + + + +
      + + Error if the request fails or returns a non-OK status + +
      + + + + + +
      Returns:
      + + +
      + Promise resolving to the parsed JSON data +
      + + + + + + + + + + + + + + + +

      getDimensions(data)

      + + + + + + +
      + Analyzes the dimensions of a nested array structure. +
      + + + + + + + + + +
      Parameters:
      + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      NameTypeDescription
      data + + +unknown + + + + The array to analyze
      + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + +
      Throws:
      + + + +
      + + Error if any dimension has zero length + +
      + + + + + +
      Returns:
      + + +
      + Array of dimension sizes at each nesting level +
      + + + + + + + + + + + + + + + +

      isBoolean(value)

      + + + + + + +
      + Type guard to check if a value is a boolean. +
      + + + + + + + + + +
      Parameters:
      + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      NameTypeDescription
      value + + +unknown + + + + The value to check
      + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + + + +
      Returns:
      + + +
      + True if the value is a boolean, false otherwise +
      + + + + + + + + + + + + + + + +

      isInteger(value, lowerBound)

      + + + + + + +
      + Type guard to check if a value is an integer within specified bounds. +
      + + + + + + + + + +
      Parameters:
      + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      NameTypeDescription
      value + + +unknown + + + + The value to check
      lowerBound + + +number + + + + The minimum allowed value (inclusive)
      + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + + + +
      Returns:
      + + +
      + True if the value is an integer >= lowerBound, false otherwise +
      + + + + + + + + + + + + + + + +

      log(level, message)

      + + + + + + +
      + Simple logger to write log messages to the console.
      +Supported logging levels are `error`, `warn`, `info` and `debug`.
      +Throws an error if logging level is invalid. +
      + + + + + + + + + +
      Parameters:
      + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      NameTypeDescription
      level + + +'error' +| + +'warn' +| + +'info' +| + +'debug' + + + + The log level for the message
      message + + +string +| + +Error + + + + The message to log, either a string or Error object
      + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Source:
      +
      + + + + + + + +
      + + + + + + + + + + + + + + + + + + + + + + + + + + +

      timestampToMicros(timestamp, unit)

      + + + + + + +
      + Converts a timestamp from the specified unit to microseconds. +
      + + + + + + + + + +
      Parameters:
      + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      NameTypeDescription
      timestamp + + +bigint + + + + The timestamp value as a bigint
      unit + + +TimestampUnit -
      Type:
      -
        -
      • -Agent.Options +
      The source timestamp unit
      -
    • -

    @@ -148,7 +1533,7 @@

    Type:
    Source:
    @@ -164,11 +1549,41 @@
    Type:
    + + + + + + + +
    Throws:
    + + +
    + + Error if the timestamp unit is unknown +
    + -

    Methods

    + + +
    Returns:
    + + +
    + The timestamp converted to microseconds +
    + + + + + + + + @@ -176,7 +1591,7 @@

    Methods

    -

    log(level, message)

    +

    timestampToNanos(timestamp, unit)

    @@ -184,9 +1599,7 @@

    log - Simple logger to write log messages to the console.
    -Supported logging levels are `error`, `warn`, `info` and `debug`.
    -Throws an error if logging level is invalid. + Converts a timestamp from the specified unit to nanoseconds. @@ -222,22 +1635,199 @@
    Parameters:
    - level + timestamp -'error' -| +bigint -'warn' -| -'info' -| + + -'debug' + + + + + The timestamp value as a bigint + + + + + + + unit + + + + + +TimestampUnit + + + + + + + + + + The source timestamp unit + + + + + + + + + + + +
    + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Source:
    +
    + + + + + + + +
    + + + + + + + + + + + + + +
    Throws:
    + + + +
    + + Error if the timestamp unit is unknown + +
    + + + + + +
    Returns:
    + + +
    + The timestamp converted to nanoseconds +
    + + + + + + + + + + + + + + + +

    validateArray(data, dimensions)

    + + + + + + +
    + Validates an array structure.
    +Validation fails if: +- data is not an array +- the array is irregular: the length of its sub-arrays are different +- the array is not homogenous: the array contains mixed types +
    + + + + + + + + + +
    Parameters:
    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + @@ -311,7 +1901,7 @@
    Parameters:
    Source:
    @@ -334,9 +1924,31 @@
    Parameters:
    +
    Throws:
    + + + +
    + + Error if the validation fails + +
    + + + + + +
    Returns:
    + + +
    + The primitive type of the array's elements +
    + + @@ -355,8 +1967,7 @@

    val
    - Validates a column name.
    -Throws an error if column name is invalid. + Validates a column name.
    @@ -472,7 +2083,7 @@

    Parameters:
    Source:
    @@ -495,6 +2106,18 @@
    Parameters:
    +
    Throws:
    + + + +
    + + Error if column name is invalid. + +
    + + + @@ -516,8 +2139,7 @@

    vali
    - Validates a table name.
    -Throws an error if table name is invalid. + Validates a table name.
    @@ -633,7 +2255,7 @@

    Parameters:
    Source:
    @@ -656,6 +2278,18 @@
    Parameters:
    +
    Throws:
    + + + +
    + + Error if table name is invalid. + +
    + + + @@ -679,13 +2313,13 @@
    Parameters:

    - Documentation generated by JSDoc 4.0.4 on Mon Dec 09 2024 02:16:59 GMT+0100 (hora estándar de Europa central) + Documentation generated by JSDoc 4.0.4 on Thu Aug 07 2025 12:45:55 GMT+0100 (British Summer Time)
    diff --git a/docs/index.html b/docs/index.html index 0321e83..c09ab1e 100644 --- a/docs/index.html +++ b/docs/index.html @@ -49,8 +49,10 @@

    Requirements

    Installation

    # With npm
     npm i -s @questdb/nodejs-client
    +
     # With yarn
     yarn add @questdb/nodejs-client
    +
     # With pnpm
     pnpm add @questdb/nodejs-client
     
    @@ -58,21 +60,21 @@

    Compatibility table

    NameTypeDescription
    data + + +Array.<unknown> @@ -247,20 +1837,20 @@
    Parameters:
    -
    The log level of the message.The array to validate
    messagedimensions -string +Array.<number> @@ -270,7 +1860,7 @@
    Parameters:
    -
    The log message.The shape of the array
    - - + + - + - + - +
    QuestDB versionNode.js client versionQuestDB client versionNode.js version HTTP Agent
    ^5.0.0^4.0.0 >=v20.X.X Undici Agent
    ^4.0.0^3.0.0 <v20.X.XHttp.AgentHttp Agent
    @@ -86,36 +88,64 @@

    Basic API usage

    import { Sender } from "@questdb/nodejs-client";
     
     async function run() {
    -  // create a sender using HTTP protocol
    -  const sender = Sender.fromConfig("http::addr=127.0.0.1:9000");
    -
    -  // add rows to the buffer of the sender
    -  await sender
    -    .table("trades")
    -    .symbol("symbol", "ETH-USD")
    -    .symbol("side", "sell")
    -    .floatColumn("price", 2615.54)
    -    .floatColumn("amount", 0.00044)
    -    .at(Date.now(), "ms");
    -
    -  // flush the buffer of the sender, sending the data to QuestDB
    -  // the buffer is cleared after the data is sent, and the sender is ready to accept new data
    -  await sender.flush();
    -
    -  // close the connection after all rows ingested
    -  // unflushed data will be lost
    -  await sender.close();
    +    // create a sender
    +    const sender = await Sender.fromConfig('http::addr=localhost:9000');
    +
    +    // order book snapshots
    +    const orderBooks = [
    +        {
    +            symbol: 'BTC-USD',
    +            exchange: 'COINBASE',
    +            timestamp: Date.now(),
    +            bidPrices: [50100.25, 50100.20, 50100.15, 50100.10, 50100.05],
    +            bidSizes: [0.5, 1.2, 2.1, 0.8, 3.5],
    +            askPrices: [50100.30, 50100.35, 50100.40, 50100.45, 50100.50],
    +            askSizes: [0.6, 1.5, 1.8, 2.2, 4.0]
    +        },
    +        {
    +            symbol: 'ETH-USD',
    +            exchange: 'COINBASE',
    +            timestamp: Date.now(),
    +            bidPrices: [2850.50, 2850.45, 2850.40, 2850.35, 2850.30],
    +            bidSizes: [5.0, 8.2, 12.5, 6.8, 15.0],
    +            askPrices: [2850.55, 2850.60, 2850.65, 2850.70, 2850.75],
    +            askSizes: [4.5, 7.8, 10.2, 8.5, 20.0]
    +        }
    +    ];
    +
    +    try {
    +        // add rows to the buffer of the sender
    +        for (const orderBook of orderBooks) {
    +            await sender
    +                .table('order_book_l2')
    +                .symbol('symbol', orderBook.symbol)
    +                .symbol('exchange', orderBook.exchange)
    +                .arrayColumn('bid_prices', orderBook.bidPrices)
    +                .arrayColumn('bid_sizes', orderBook.bidSizes)
    +                .arrayColumn('ask_prices', orderBook.askPrices)
    +                .arrayColumn('ask_sizes', orderBook.askSizes)
    +                .at(orderBook.timestamp, 'ms');
    +        }
    +
    +        // flush the buffer of the sender, sending the data to QuestDB
    +        // the buffer is cleared after the data is sent, and the sender is ready to accept new data
    +        await sender.flush();
    +    } finally {
    +      // close the connection after all rows ingested
    +      await sender.close();
    +    }
     }
     
     run().then(console.log).catch(console.error);
     

    Authentication and secure connection

    +

    Username and password authentication

    import { Sender } from "@questdb/nodejs-client";
     
     async function run() {
       // create a sender using HTTPS protocol with username and password authentication
       const sender = Sender.fromConfig(
    -    "https::addr=127.0.0.1:9000;username=admin;password=quest;",
    +    "https::addr=127.0.0.1:9000;username=admin;password=quest",
       );
     
       // send the data over the authenticated and secure connection
    @@ -134,13 +164,13 @@ 

    Authentication and secure connection

    run().catch(console.error);
    -

    TypeScript example

    +

    Token authentication

    import { Sender } from "@questdb/nodejs-client";
     
     async function run(): Promise<void> {
       // create a sender using HTTPS protocol with bearer token authentication
       const sender: Sender = Sender.fromConfig(
    -    "https::addr=127.0.0.1:9000;token=Xyvd3er6GF87ysaHk;",
    +    "https::addr=127.0.0.1:9000;token=Xyvd3er6GF87ysaHk",
       );
     
       // send the data over the authenticated and secure connection
    @@ -244,7 +274,12 @@ 

    Worker threads example

    } run().then(console.log).catch(console.error); -
    + +

    Community

    +

    If you need help, have additional questions or want to provide feedback, you +may find us on our Community Forum.

    +

    You can also sign up to our mailing list +to get notified of new releases.

    @@ -255,13 +290,13 @@

    Worker threads example


    - Documentation generated by JSDoc 4.0.4 on Mon Dec 09 2024 02:16:59 GMT+0100 (hora estándar de Europa central) + Documentation generated by JSDoc 4.0.4 on Thu Aug 07 2025 12:45:55 GMT+0100 (British Summer Time)
    diff --git a/docs/index.js.html b/docs/index.js.html index 843f5c2..1f0c870 100644 --- a/docs/index.js.html +++ b/docs/index.js.html @@ -27,18 +27,24 @@

    Source: index.js

    var node_fs = require('node:fs');
    +var undici = require('undici');
    +var http = require('http');
    +var https = require('https');
     var node_buffer = require('node:buffer');
     var net = require('node:net');
     var tls = require('node:tls');
     var crypto = require('node:crypto');
    -var undici = require('undici');
     
     function _interopDefault (e) { return e && e.__esModule ? e : { default: e }; }
     
    +var http__default = /*#__PURE__*/_interopDefault(http);
    +var https__default = /*#__PURE__*/_interopDefault(https);
     var net__default = /*#__PURE__*/_interopDefault(net);
     var tls__default = /*#__PURE__*/_interopDefault(tls);
     var crypto__default = /*#__PURE__*/_interopDefault(crypto);
     
    +// Log level configuration with console methods and criticality levels. <br>
    +// Higher criticality values indicate more important messages.
     const LOG_LEVELS = {
         error: {
             log: console.error,
    @@ -57,14 +63,15 @@ 

    Source: index.js

    criticality: 0 } }; +// Default logging criticality level. Messages with criticality below this level are ignored. const DEFAULT_CRITICALITY = LOG_LEVELS.info.criticality; /** * Simple logger to write log messages to the console. <br> * Supported logging levels are `error`, `warn`, `info` and `debug`. <br> * Throws an error if logging level is invalid. * - * @param {'error'|'warn'|'info'|'debug'} level - The log level of the message. - * @param {string} message - The log message. + * @param {'error'|'warn'|'info'|'debug'} level - The log level for the message + * @param {string | Error} message - The message to log, either a string or Error object */ function log(level, message) { const logLevel = LOG_LEVELS[level]; if (!logLevel) { @@ -76,117 +83,225 @@

    Source: index.js

    } /** - * Validates a table name. <br> - * Throws an error if table name is invalid. - * - * @param {string} name - The table name to validate. - * @param {number} maxNameLength - The maximum length of table names. - */ function validateTableName(name, maxNameLength) { - const len = name.length; - if (len > maxNameLength) { - throw new Error(`Table name is too long, max length is ${maxNameLength}`); + * Type guard to check if a value is a boolean. + * @param {unknown} value - The value to check + * @returns True if the value is a boolean, false otherwise + */ function isBoolean(value) { + return typeof value === "boolean"; +} +/** + * Type guard to check if a value is an integer within specified bounds. + * @param {unknown} value - The value to check + * @param {number} lowerBound - The minimum allowed value (inclusive) + * @returns True if the value is an integer >= lowerBound, false otherwise + */ function isInteger(value, lowerBound) { + return typeof value === "number" && Number.isInteger(value) && value >= lowerBound; +} +/** + * Converts a timestamp from the specified unit to microseconds. + * @param {bigint} timestamp - The timestamp value as a bigint + * @param {TimestampUnit} unit - The source timestamp unit + * @returns The timestamp converted to microseconds + * @throws Error if the timestamp unit is unknown + */ function timestampToMicros(timestamp, unit) { + switch(unit){ + case "ns": + return timestamp / 1000n; + case "us": + return timestamp; + case "ms": + return timestamp * 1000n; + default: + throw new Error(`Unknown timestamp unit: ${unit}`); } - if (len === 0) { - throw new Error("Empty string is not allowed as table name"); +} +/** + * Converts a timestamp from the specified unit to nanoseconds. + * @param {bigint} timestamp - The timestamp value as a bigint + * @param {TimestampUnit} unit - The source timestamp unit + * @returns The timestamp converted to nanoseconds + * @throws Error if the timestamp unit is unknown + */ function timestampToNanos(timestamp, unit) { + switch(unit){ + case "ns": + return timestamp; + case "us": + return timestamp * 1000n; + case "ms": + return timestamp * 1000_000n; + default: + throw new Error(`Unknown timestamp unit: ${unit}`); } - for(let i = 0; i < len; i++){ - const ch = name[i]; - switch(ch){ - case ".": - if (i === 0 || i === len - 1 || name[i - 1] === ".") // single dot is allowed in the middle only - // starting with a dot hides directory in Linux - // ending with a dot can be trimmed by some Windows versions / file systems - // double or triple dot looks suspicious - // single dot allowed as compatibility, - // when someone uploads 'file_name.csv' the file name used as the table name - throw new Error("Table name cannot start or end with a dot, and only a single dot allowed"); - break; - case "?": - case ",": - case "'": - case '"': - case "\\": - case "/": - case ":": - case ")": - case "(": - case "+": - case "*": - case "%": - case "~": - case "\u0000": - case "\u0001": - case "\u0002": - case "\u0003": - case "\u0004": - case "\u0005": - case "\u0006": - case "\u0007": - case "\u0008": - case "\u0009": - case "\u000B": - case "\u000c": - case "\r": - case "\n": - case "\u000e": - case "\u000f": - case "\u007f": - case "\ufeff": - throw new Error(`Invalid character in table name: ${ch}`); +} +/** + * Analyzes the dimensions of a nested array structure. + * @param {unknown} data - The array to analyze + * @returns Array of dimension sizes at each nesting level + * @throws Error if any dimension has zero length + */ function getDimensions(data) { + const dimensions = []; + while(Array.isArray(data)){ + dimensions.push(data.length); + data = data[0]; + } + return dimensions; +} +/** + * Validates an array structure. <br> + * Validation fails if: + * - <i>data</i> is not an array + * - the array is irregular: the length of its sub-arrays are different + * - the array is not homogenous: the array contains mixed types + * @param {unknown[]} data - The array to validate + * @param {number[]} dimensions - The shape of the array + * @returns The primitive type of the array's elements + * @throws Error if the validation fails + */ function validateArray(data, dimensions) { + if (data === null || data === undefined) { + return null; + } + if (!Array.isArray(data)) { + throw new Error(`The value must be an array [value=${JSON.stringify(data)}, type=${typeof data}]`); + } + let expectedType = null; + function checkArray(array, depth = 0, path = "") { + const expectedLength = dimensions[depth]; + if (array.length !== expectedLength) { + throw new Error(`Lengths of sub-arrays do not match [expected=${expectedLength}, actual=${array.length}, dimensions=[${dimensions}], path=${path}]`); + } + if (depth < dimensions.length - 1) { + // intermediate level, expecting arrays + for(let i = 0; i < array.length; i++){ + if (!Array.isArray(array[i])) { + throw new Error(`Mixed types found [expected=array, current=${typeof array[i]}, path=${path}[${i}]]`); + } + checkArray(array[i], depth + 1, `${path}[${i}]`); + } + } else { + // leaf level, expecting primitives + if (expectedType === null && array[0] !== undefined) { + expectedType = typeof array[0]; + } + for(let i = 0; i < array.length; i++){ + const currentType = typeof array[i]; + if (currentType !== expectedType) { + throw new Error(expectedType !== null ? `Mixed types found [expected=${expectedType}, current=${currentType}, path=${path}[${i}]]` : `Unsupported array type [type=${currentType}]`); + } + } } } + checkArray(data); + return expectedType; } /** - * Validates a column name. <br> - * Throws an error if column name is invalid. - * - * @param {string} name - The column name to validate. - * @param {number} maxNameLength - The maximum length of column names. - */ function validateColumnName(name, maxNameLength) { - const len = name.length; - if (len > maxNameLength) { - throw new Error(`Column name is too long, max length is ${maxNameLength}`); + * Fetches JSON data from a URL. + * @template T - The expected type of the JSON response + * @param {string} url - The URL to fetch from + * @param {Agent} agent - HTTP agent to be used for the request + * @param {number} timeout - Request timeout, query will be aborted if not finished in time + * @returns Promise resolving to the parsed JSON data + * @throws Error if the request fails or returns a non-OK status + */ async function fetchJson(url, timeout, agent) { + const controller = new AbortController(); + const { signal } = controller; + const timeoutId = setTimeout(()=>controller.abort(), timeout); + let response; + try { + response = await fetch(url, { + dispatcher: agent, + signal + }); + } catch (error) { + throw new Error(`Failed to load ${url} [error=${error}]`); + } finally{ + clearTimeout(timeoutId); } - if (len === 0) { - throw new Error("Empty string is not allowed as column name"); + if (!response.ok) { + throw new Error(`Failed to load ${url} [statusCode=${response.status} (${response.statusText})]`); } - for (const ch of name){ - switch(ch){ - case "?": - case ".": - case ",": - case "'": - case '"': - case "\\": - case "/": - case ":": - case ")": - case "(": - case "+": - case "-": - case "*": - case "%": - case "~": - case "\u0000": - case "\u0001": - case "\u0002": - case "\u0003": - case "\u0004": - case "\u0005": - case "\u0006": - case "\u0007": - case "\u0008": - case "\u0009": - case "\u000B": - case "\u000c": - case "\r": - case "\n": - case "\u000e": - case "\u000f": - case "\u007f": - case "\ufeff": - throw new Error(`Invalid character in column name: ${ch}`); + return await response.json(); +} + +// @ts-check +// HTTP status code for successful request with no content. +const HTTP_NO_CONTENT = 204; +// Default number of rows that trigger auto-flush for HTTP transport. +const DEFAULT_HTTP_AUTO_FLUSH_ROWS = 75000; +// Default minimum throughput for HTTP requests (100 KB/sec). +const DEFAULT_REQUEST_MIN_THROUGHPUT = 102400; +// Default request timeout in milliseconds (10 seconds). +const DEFAULT_REQUEST_TIMEOUT = 10000; +// Default retry timeout in milliseconds (10 seconds). +const DEFAULT_RETRY_TIMEOUT = 10000; +// HTTP status codes that should trigger request retries. +// Includes server errors and gateway timeouts that may be transient. +const RETRIABLE_STATUS_CODES = [ + 500, + 503, + 504, + 507, + 509, + 523, + 524, + 529, + 599 +]; +/** + * Abstract base class for HTTP-based transport implementations. <br> + * Provides common configuration and functionality for HTTP and HTTPS protocols. + */ class HttpTransportBase { + /** + * Creates a new HttpTransportBase instance. + * + * @param {SenderOptions} options - Sender configuration options including connection and authentication details + * @throws Error if required protocol or host options are missing + */ constructor(options){ + if (!options || !options.protocol) { + throw new Error("The 'protocol' option is mandatory"); } + if (!options.host) { + throw new Error("The 'host' option is mandatory"); + } + this.log = typeof options.log === "function" ? options.log : log; + this.tlsVerify = isBoolean(options.tls_verify) ? options.tls_verify : true; + this.tlsCA = options.tls_ca ? node_fs.readFileSync(options.tls_ca) : undefined; + this.username = options.username; + this.password = options.password; + this.token = options.token; + if (!options.port) { + options.port = 9000; + } + this.host = options.host; + this.port = options.port; + this.requestMinThroughput = isInteger(options.request_min_throughput, 0) ? options.request_min_throughput : DEFAULT_REQUEST_MIN_THROUGHPUT; + this.requestTimeout = isInteger(options.request_timeout, 1) ? options.request_timeout : DEFAULT_REQUEST_TIMEOUT; + this.retryTimeout = isInteger(options.retry_timeout, 0) ? options.retry_timeout : DEFAULT_RETRY_TIMEOUT; + switch(options.protocol){ + case HTTP: + this.secure = false; + break; + case HTTPS: + this.secure = true; + break; + default: + throw new Error("The 'protocol' has to be 'http' or 'https' for the HTTP transport"); + } + } + /** + * HTTP transport does not require explicit connection establishment. + * @throws Error indicating connect is not required for HTTP transport + */ connect() { + throw new Error("'connect()' is not required for HTTP transport"); + } + /** + * HTTP transport does not require explicit connection closure. + * @returns Promise that resolves immediately + */ async close() {} + /** + * Gets the default auto-flush row count for HTTP transport. + * @returns {number} Default number of rows that trigger auto-flush + */ getDefaultAutoFlushRows() { + return DEFAULT_HTTP_AUTO_FLUSH_ROWS; } } @@ -199,19 +314,28 @@

    Source: index.js

    const ON = "on"; const OFF = "off"; const UNSAFE_OFF = "unsafe_off"; +const PROTOCOL_VERSION_AUTO = "auto"; +const PROTOCOL_VERSION_V1 = "1"; +const PROTOCOL_VERSION_V2 = "2"; +const LINE_PROTO_SUPPORT_VERSION = "line.proto.support.versions"; /** @classdesc * <a href="Sender.html">Sender</a> configuration options. <br> * <br> * Properties of the object are initialized through a configuration string. <br> * The configuration string has the following format: <i>&ltprotocol&gt::&ltkey&gt=&ltvalue&gt;&ltkey&gt=&ltvalue&gt;...;</i> <br> * The keys are case-sensitive, the trailing semicolon is optional. <br> - * The values are validated, and an error is thrown if the format is invalid. <br> + * The values are validated and an error is thrown if the format is invalid. <br> * <br> * Connection and protocol options * <ul> * <li> <b>protocol</b>: <i>enum, accepted values: http, https, tcp, tcps</i> - The protocol used to communicate with the server. <br> * When <i>https</i> or <i>tcps</i> used, the connection is secured with TLS encryption. * </li> + * <li> <b>protocol_version</b>: <i>enum, accepted values: auto, 1, 2</i> - The protocol version used for data serialization. <br> + * Version 1 uses text-based serialization for all data types. Version 2 uses binary encoding for doubles and arrays. <br> + * When set to 'auto' (default for HTTP/HTTPS), the client automatically negotiates the highest supported version with the server. <br> + * TCP/TCPS connections default to version 1. + * </li> * <li> addr: <i>string</i> - Hostname and port, separated by colon. This key is mandatory, but the port part is optional. <br> * If no port is specified, a default will be used. <br> * When the protocol is HTTP/HTTPS, the port defaults to 9000. When the protocol is TCP/TCPS, the port defaults to 9009. <br> @@ -237,8 +361,8 @@

    Source: index.js

    * TLS options * <ul> * <li> tls_verify: <i>enum, accepted values: on, unsafe_off</i> - When the HTTPS or TCPS protocols are selected, TLS encryption is used. <br> - * By default, the Sender will verify the server's certificate, but this check can be disabled by setting this option to <i>off</i>. This is useful - * non-production environments where self-signed certificates might be used, but should be avoided in production if possible. + * By default, the Sender will verify the server's certificate, but this check can be disabled by setting this option to <i>unsafe_off</i>. <br> + * This is useful in non-production environments where self-signed certificates might be used, but should be avoided in production if possible. * </li> * <li> tls_ca: <i>string</i> - Path to a file containing the root CA's certificate in PEM format. <br> * Can be useful when self-signed certificates are used, otherwise should not be set. @@ -250,9 +374,9 @@

    Source: index.js

    * <li> auto_flush: <i>enum, accepted values: on, off</i> - The Sender automatically flushes the buffer by default. This can be switched off * by setting this option to <i>off</i>. <br> * When disabled, the flush() method of the Sender has to be called explicitly to make sure data is sent to the server. <br> - * Manual buffer flushing can be useful, especially when we want to use transactions. When the HTTP protocol is used, each flush results in a single HTTP - * request, which becomes a single transaction on the server side. The transaction either succeeds, and all rows sent in the request are - * inserted; or it fails, and none of the rows make it into the database. + * Manual buffer flushing can be useful, especially when we want to control transaction boundaries. <br> + * When the HTTP protocol is used, each flush results in a single HTTP request, which becomes a single transaction on the server side. <br> + * The transaction either succeeds, and all rows sent in the request are inserted; or it fails, and none of the rows make it into the database. * </li> * <li> auto_flush_rows: <i>integer</i> - The number of rows that will trigger a flush. When set to 0, row-based flushing is disabled. <br> * The Sender will default this parameter to 75000 rows when HTTP protocol is used, and to 600 in case of TCP protocol. @@ -288,14 +412,12 @@

    Source: index.js

    * <br> * Other options * <ul> + * <li> stdlib_http: <i>enum, accepted values: on, off</i> - With HTTP protocol the Undici library is used by default. By setting this option + * to <i>on</i> the client switches to node's core http and https modules. + * </li> * <li> max_name_len: <i>integer</i> - The maximum length of a table or column name, the Sender defaults this parameter to 127. <br> * Recommended to use the same setting as the server, which also uses 127 by default. * </li> - * <li> copy_buffer: <i>enum, accepted values: on, off</i> - By default, the Sender creates a new buffer for every flush() call, - * and the data to be sent to the server is copied into this new buffer. - * Setting the flag to <i>off</i> results in reusing the same buffer instance for each flush() call. <br> - * Use this flag only if calls to the client are serialised. - * </li> * </ul> */ class SenderOptions { /** @@ -306,21 +428,72 @@

    Source: index.js

    * - 'log' is a logging function used by the <a href="Sender.html">Sender</a>. <br> * Prototype: <i>(level: 'error'|'warn'|'info'|'debug', message: string) => void</i>. <br> * - 'agent' is a custom http/https agent used by the <a href="Sender.html">Sender</a> when http/https transport is used. <br> - * A <i>http.Agent</i> or <i>https.Agent</i> object is expected. - */ constructor(configurationString, extraOptions = undefined){ + * Depends on which transport implementation and protocol used, one of the followings expected: <i>undici.Agent</i>, <i>http.Agent</i> or <i>https.Agent</i>. + */ constructor(configurationString, extraOptions){ parseConfigurationString(this, configurationString); if (extraOptions) { if (extraOptions.log && typeof extraOptions.log !== "function") { throw new Error("Invalid logging function"); } this.log = extraOptions.log; - if (extraOptions.agent && !(extraOptions.agent instanceof undici.Agent)) { - throw new Error("Invalid http/https agent"); + if (extraOptions.agent && !(extraOptions.agent instanceof undici.Agent) && !(extraOptions.agent instanceof http__default.default.Agent) && // @ts-expect-error - Not clear what the problem is, the two lines above have no issues + !(extraOptions.agent instanceof https__default.default.Agent)) { + throw new Error("Invalid HTTP agent"); } this.agent = extraOptions.agent; } } /** + * Resolves the protocol version, if it is set to 'auto'. <br> + * If TCP transport is used, the protocol version will default to 1. + * In case of HTTP transport the <i>/settings</i> endpoint of the database is used to find the protocol versions + * supported by the server, and the highest will be selected. + * When calling the <i>/settings</i> endpoint the timeout and TLs options are used from the <i>options</i> object. + * @param {SenderOptions} options SenderOptions instance needs resolving protocol version + */ static async resolveAuto(options) { + parseProtocolVersion(options); + if (options.protocol_version !== PROTOCOL_VERSION_AUTO) { + return options; + } + const url = `${options.protocol}://${options.host}:${options.port}/settings`; + const settings = await fetchJson(url, isInteger(options.request_timeout, 1) ? options.request_timeout : DEFAULT_REQUEST_TIMEOUT, new undici.Agent({ + connect: { + ca: options.tls_ca ? node_fs.readFileSync(options.tls_ca) : undefined, + rejectUnauthorized: isBoolean(options.tls_verify) ? options.tls_verify : true + } + })); + const supportedVersions = (settings.config[LINE_PROTO_SUPPORT_VERSION] ?? []).map((version)=>String(version)); + if (supportedVersions.length === 0) { + options.protocol_version = PROTOCOL_VERSION_V1; + } else if (supportedVersions.includes(PROTOCOL_VERSION_V2)) { + options.protocol_version = PROTOCOL_VERSION_V2; + } else if (supportedVersions.includes(PROTOCOL_VERSION_V1)) { + options.protocol_version = PROTOCOL_VERSION_V1; + } else { + throw new Error("Unsupported protocol versions received from server: " + supportedVersions); + } + return options; + } + static resolveDeprecated(options, log) { + if (!options) { + return; + } + // deal with deprecated options + if (options.copy_buffer !== undefined) { + log("warn", `Option 'copy_buffer' is not supported anymore, please, remove it`); + options.copy_buffer = undefined; + } + if (options.copyBuffer !== undefined) { + log("warn", `Option 'copyBuffer' is not supported anymore, please, remove it`); + options.copyBuffer = undefined; + } + if (options.bufferSize !== undefined) { + log("warn", `Option 'bufferSize' is not supported anymore, please, replace it with 'init_buf_size'`); + options.init_buf_size = options.bufferSize; + options.bufferSize = undefined; + } + } + /** * Creates a Sender options object by parsing the provided configuration string. * * @param {string} configurationString - Configuration string. <br> @@ -328,24 +501,26 @@

    Source: index.js

    * - 'log' is a logging function used by the <a href="Sender.html">Sender</a>. <br> * Prototype: <i>(level: 'error'|'warn'|'info'|'debug', message: string) => void</i>. <br> * - 'agent' is a custom http/https agent used by the <a href="Sender.html">Sender</a> when http/https transport is used. <br> - * A <i>http.Agent</i> or <i>https.Agent</i> object is expected. + * Depends on which transport implementation and protocol used, one of the followings expected: <i>undici.Agent</i>, <i>http.Agent</i> or <i>https.Agent</i>. * * @return {SenderOptions} A Sender configuration object initialized from the provided configuration string. - */ static fromConfig(configurationString, extraOptions = undefined) { - return new SenderOptions(configurationString, extraOptions); + */ static async fromConfig(configurationString, extraOptions) { + const options = new SenderOptions(configurationString, extraOptions); + await SenderOptions.resolveAuto(options); + return options; } /** * Creates a Sender options object by parsing the configuration string set in the <b>QDB_CLIENT_CONF</b> environment variable. * * @param {object} extraOptions - Optional extra configuration. <br> * - 'log' is a logging function used by the <a href="Sender.html">Sender</a>. <br> - }in /**br> + * Prototype: <i>(level: 'error'|'warn'|'info'|'debug', message: string) => void</i>. <br> * - 'agent' is a custom http/https agent used by the <a href="Sender.html">Sender</a> when http/https transport is used. <br> - * A <i>http.Agent</i> or <i>https.Agent</i> object is expected. + * Depends on which transport implementation and protocol used, one of the followings expected: <i>undici.Agent</i>, <i>http.Agent</i> or <i>https.Agent</i>. * * @return {SenderOptions} A Sender configuration object initialized from the <b>QDB_CLIENT_CONF</b> environment variable. - */ static fromEnv(extraOptions = undefined) { - return SenderOptions.fromConfig(process.env.QDB_CLIENT_CONF, extraOptions); + */ static async fromEnv(extraOptions) { + return await SenderOptions.fromConfig(process.env.QDB_CLIENT_CONF, extraOptions); } } function parseConfigurationString(options, configString) { @@ -354,13 +529,14 @@

    Source: index.js

    } const position = parseProtocol(options, configString); parseSettings(options, configString, position); + parseProtocolVersion(options); parseAddress(options); parseBufferSizes(options); parseAutoFlushOptions(options); parseTlsOptions(options); parseRequestTimeoutOptions(options); parseMaxNameLength(options); - parseCopyBuffer(options); + parseStdlibTransport(options); } function parseSettings(options, configString, position) { let index = configString.indexOf(";", position); @@ -390,6 +566,7 @@

    Source: index.js

    options[key] = value; } const ValidConfigKeys = [ + "protocol_version", "addr", "username", "password", @@ -399,13 +576,13 @@

    Source: index.js

    "auto_flush", "auto_flush_rows", "auto_flush_interval", - "copy_buffer", "request_min_throughput", "request_timeout", "retry_timeout", "init_buf_size", "max_buf_size", "max_name_len", + "stdlib_http", "tls_verify", "tls_ca", "tls_roots", @@ -444,6 +621,27 @@

    Source: index.js

    } return index + 2; } +function parseProtocolVersion(options) { + const protocol_version = options.protocol_version ?? PROTOCOL_VERSION_AUTO; + switch(protocol_version){ + case PROTOCOL_VERSION_AUTO: + switch(options.protocol){ + case HTTP: + case HTTPS: + options.protocol_version = PROTOCOL_VERSION_AUTO; + break; + default: + options.protocol_version = PROTOCOL_VERSION_V1; + } + break; + case PROTOCOL_VERSION_V1: + case PROTOCOL_VERSION_V2: + break; + default: + throw new Error(`Invalid protocol version: '${protocol_version}', accepted values: 'auto', '1', '2'`); + } + return; +} function parseAddress(options) { if (!options.addr) { throw new Error("Invalid configuration, 'addr' is required"); @@ -503,8 +701,8 @@

    Source: index.js

    function parseMaxNameLength(options) { parseInteger(options, "max_name_len", "max name length", 1); } -function parseCopyBuffer(options) { - parseBoolean(options, "copy_buffer", "copy buffer"); +function parseStdlibTransport(options) { + parseBoolean(options, "stdlib_http", "stdlib http"); } function parseBoolean(options, property, description, offValue = OFF) { if (options[property]) { @@ -535,259 +733,206 @@

    Source: index.js

    } // @ts-check -const HTTP_NO_CONTENT = 204; // success -const DEFAULT_HTTP_AUTO_FLUSH_ROWS = 75000; -const DEFAULT_TCP_AUTO_FLUSH_ROWS = 600; -const DEFAULT_AUTO_FLUSH_INTERVAL = 1000; // 1 sec -const DEFAULT_MAX_NAME_LENGTH = 127; -const DEFAULT_REQUEST_MIN_THROUGHPUT = 102400; // 100 KB/sec -const DEFAULT_REQUEST_TIMEOUT = 10000; // 10 sec -const DEFAULT_RETRY_TIMEOUT = 10000; // 10 sec -const DEFAULT_BUFFER_SIZE = 65536; // 64 KB -const DEFAULT_MAX_BUFFER_SIZE = 104857600; // 100 MB -/** @type {Agent.Options} */ const DEFAULT_HTTP_OPTIONS = { +/** + * Default HTTP options for the Undici agent. + * Configures keep-alive connections with 60-second timeout and single request pipelining. + */ const DEFAULT_HTTP_OPTIONS = { connect: { keepAlive: true }, pipelining: 1, keepAliveTimeout: 60000 }; -// an arbitrary public key, not used in authentication -// only used to construct a valid JWK token which is accepted by the crypto API +/** + * HTTP transport implementation using the Undici library. <br> + * Provides high-performance HTTP requests with connection pooling and retry logic. <br> + * Supports both HTTP and HTTPS protocols with configurable authentication. + */ class UndiciTransport extends HttpTransportBase { + /** + * Creates a new UndiciTransport instance. + * + * @param options - Sender configuration object containing connection and retry settings + * @throws Error if the protocol is not 'http' or 'https' + */ constructor(options){ + super(options); + switch(options.protocol){ + case HTTP: + this.agent = options.agent instanceof undici.Agent ? options.agent : UndiciTransport.getDefaultHttpAgent(); + break; + case HTTPS: + if (options.agent instanceof undici.Agent) { + this.agent = options.agent; + } else { + // Create a new agent with instance-specific TLS options + this.agent = new undici.Agent({ + ...DEFAULT_HTTP_OPTIONS, + connect: { + ...DEFAULT_HTTP_OPTIONS.connect, + requestCert: this.tlsVerify, + rejectUnauthorized: this.tlsVerify, + ca: this.tlsCA + } + }); + } + break; + default: + throw new Error("The 'protocol' has to be 'http' or 'https' for the Undici HTTP transport"); + } + this.dispatcher = new undici.RetryAgent(this.agent, { + maxRetries: Infinity, + minTimeout: 10, + maxTimeout: 1000, + timeoutFactor: 2, + retryAfter: true, + methods: [ + "GET", + "POST", + "PUT", + "DELETE", + "PATCH", + "OPTIONS", + "HEAD" + ], + statusCodes: RETRIABLE_STATUS_CODES, + errorCodes: [ + "ECONNRESET", + "EAI_AGAIN", + "ECONNREFUSED", + "ETIMEDOUT", + "EPIPE", + "UND_ERR_CONNECT_TIMEOUT", + "UND_ERR_HEADERS_TIMEOUT", + "UND_ERR_BODY_TIMEOUT" + ] + }); + } + /** + * Sends data to QuestDB using HTTP POST. + * + * @param {Buffer} data - Buffer containing the data to send + * @returns Promise resolving to true if data was sent successfully + * @throws Error if request fails after all retries or times out + */ async send(data) { + const headers = {}; + if (this.token) { + headers["Authorization"] = `Bearer ${this.token}`; + } else if (this.username && this.password) { + headers["Authorization"] = `Basic ${node_buffer.Buffer.from(`${this.username}:${this.password}`).toString("base64")}`; + } + const controller = new AbortController(); + const { signal } = controller; + setTimeout(()=>controller.abort(), this.retryTimeout); + let responseData; + try { + const timeoutMillis = data.length / this.requestMinThroughput * 1000 + this.requestTimeout; + responseData = await this.dispatcher.request({ + origin: `${this.secure ? "https" : "http"}://${this.host}:${this.port}`, + path: "/write?precision=n", + method: "POST", + headers, + body: data, + headersTimeout: this.requestTimeout, + bodyTimeout: timeoutMillis, + signal + }); + } catch (err) { + if (err.name === "AbortError") { + throw new Error("HTTP request timeout, no response from server in time"); + } else { + throw err; + } + } + const { statusCode } = responseData; + const body = await responseData.body.arrayBuffer(); + if (statusCode === HTTP_NO_CONTENT) { + if (body.byteLength > 0) { + const message = node_buffer.Buffer.from(body).toString(); + const logMessage = message.length < 256 ? message : `${message.substring(0, 256)}... (truncated, full length=${message.length})`; + this.log("warn", `Unexpected message from server: ${logMessage}`); + } + return true; + } else { + throw new Error(`HTTP request failed, statusCode=${statusCode}, error=${node_buffer.Buffer.from(body).toString()}`); + } + } + /** + * @ignore + * Gets or creates the default HTTP agent with standard configuration. + * Uses a singleton pattern to reuse the same agent across instances. + * @returns The default Undici agent instance + */ static getDefaultHttpAgent() { + if (!UndiciTransport.DEFAULT_HTTP_AGENT) { + UndiciTransport.DEFAULT_HTTP_AGENT = new undici.Agent(DEFAULT_HTTP_OPTIONS); + } + return UndiciTransport.DEFAULT_HTTP_AGENT; + } +} + +// @ts-check +// Default number of rows that trigger auto-flush for TCP transport. +const DEFAULT_TCP_AUTO_FLUSH_ROWS = 600; +// Arbitrary public key, used to construct valid JWK tokens. +// These are not used for actual authentication, only required for crypto API compatibility. const PUBLIC_KEY = { x: "aultdA0PjhD_cWViqKKyL5chm6H1n-BiZBo_48T-uqc", y: "__ptaol41JWSpTTL525yVEfzmY8A6Vi_QrW1FjKcHMg" }; -/* -We are retrying on the following response codes (copied from the Rust client): -500: Internal Server Error -503: Service Unavailable -504: Gateway Timeout - -// Unofficial extensions -507: Insufficient Storage -509: Bandwidth Limit Exceeded -523: Origin is Unreachable -524: A Timeout Occurred -529: Site is overloaded -599: Network Connect Timeout Error -*/ const RETRIABLE_STATUS_CODES = [ - 500, - 503, - 504, - 507, - 509, - 523, - 524, - 529, - 599 -]; -/** @classdesc - * The QuestDB client's API provides methods to connect to the database, ingest data, and close the connection. - * The supported protocols are HTTP and TCP. HTTP is preferred as it provides feedback in the HTTP response. <br> - * Based on benchmarks HTTP also provides higher throughput, if configured to ingest data in bigger batches. - * <p> - * The client supports authentication. <br> - * Authentication details can be passed to the Sender in its configuration options. <br> - * The client supports Basic username/password and Bearer token authentication methods when used with HTTP protocol, - * and JWK token authentication when ingesting data via TCP. <br> - * Please, note that authentication is enabled by default in QuestDB Enterprise only. <br> - * Details on how to configure authentication in the open source version of - * QuestDB: {@link https://questdb.io/docs/reference/api/ilp/authenticate} - * </p> - * <p> - * The client also supports TLS encryption for both, HTTP and TCP transports to provide a secure connection. <br> - * Please, note that the open source version of QuestDB does not support TLS, and requires an external reverse-proxy, - * such as Nginx to enable encryption. - * </p> - * <p> - * The client uses a buffer to store data. It automatically flushes the buffer by sending its content to the server. - * Auto flushing can be disabled via configuration options to gain control over transactions. Initial and maximum - * buffer sizes can also be set. - * </p> - * <p> - * It is recommended that the Sender is created by using one of the static factory methods, - * <i>Sender.fromConfig(configString, extraOptions)</i> or <i>Sender.fromEnv(extraOptions)</i>). - * If the Sender is created via its constructor, at least the SenderOptions configuration object should be - * initialized from a configuration string to make sure that the parameters are validated. <br> - * Detailed description of the Sender's configuration options can be found in - * the <a href="SenderOptions.html">SenderOptions</a> documentation. - * </p> - * <p> - * Extra options can be provided to the Sender in the <i>extraOptions</i> configuration object. <br> - * A custom logging function and a custom HTTP(S) agent can be passed to the Sender in this object. <br> - * The logger implementation provides the option to direct log messages to the same place where the host application's - * log is saved. The default logger writes to the console. <br> - * The custom HTTP(S) agent option becomes handy if there is a need to modify the default options set for the - * HTTP(S) connections. A popular setting would be disabling persistent connections, in this case an agent can be - * passed to the Sender with <i>keepAlive</i> set to <i>false</i>. <br> - * For example: <i>Sender.fromConfig(`http::addr=host:port`, { agent: new undici.Agent({ connect: { keepAlive: false } })})</i> <br> - * If no custom agent is configured, the Sender will use its own agent which overrides some default values - * of <i>undici.Agent</i>. The Sender's own agent uses persistent connections with 1 minute idle timeout, pipelines requests default to 1. - * </p> - */ class Sender { +/** + * TCP transport implementation. <br> + * Supports both plain TCP or secure TLS-encrypted connections with configurable JWK token authentication. + */ class TcpTransport { /** - * Creates an instance of Sender. + * Creates a new TcpTransport instance. * - * @param {SenderOptions} options - Sender configuration object. <br> - * See SenderOptions documentation for detailed description of configuration options. <br> + * @param {SenderOptions} options - Sender configuration object containing connection and authentication details + * @throws Error if required options are missing or protocol is not 'tcp' or 'tcps' */ constructor(options){ if (!options || !options.protocol) { throw new Error("The 'protocol' option is mandatory"); } - replaceDeprecatedOptions(options); + if (!options.host) { + throw new Error("The 'host' option is mandatory"); + } this.log = typeof options.log === "function" ? options.log : log; + this.tlsVerify = isBoolean(options.tls_verify) ? options.tls_verify : true; + this.tlsCA = options.tls_ca ? node_fs.readFileSync(options.tls_ca) : undefined; + this.host = options.host; + this.port = options.port; switch(options.protocol){ - case HTTP: - this.http = true; - this.secure = false; - this.agent = options.agent instanceof undici.Agent ? options.agent : this.getDefaultHttpAgent(); - break; - case HTTPS: - this.http = true; - this.secure = true; - this.agent = options.agent instanceof undici.Agent ? options.agent : this.getDefaultHttpAgent(); - break; case TCP: - this.http = false; this.secure = false; break; case TCPS: - this.http = false; this.secure = true; break; default: - throw new Error(`Invalid protocol: '${options.protocol}'`); - } - if (this.http) { - this.username = options.username; - this.password = options.password; - this.token = options.token; - if (!options.port) { - options.port = 9000; - } - } else { - if (!options.auth && !options.jwk) { - constructAuth(options); - } - this.jwk = constructJwk(options); - if (!options.port) { - options.port = 9009; - } + throw new Error("The 'protocol' has to be 'tcp' or 'tcps' for the TCP transport"); } - this.host = options.host; - this.port = options.port; - this.tlsVerify = isBoolean(options.tls_verify) ? options.tls_verify : true; - this.tlsCA = options.tls_ca ? node_fs.readFileSync(options.tls_ca) : undefined; - this.autoFlush = isBoolean(options.auto_flush) ? options.auto_flush : true; - this.autoFlushRows = isInteger(options.auto_flush_rows, 0) ? options.auto_flush_rows : this.http ? DEFAULT_HTTP_AUTO_FLUSH_ROWS : DEFAULT_TCP_AUTO_FLUSH_ROWS; - this.autoFlushInterval = isInteger(options.auto_flush_interval, 0) ? options.auto_flush_interval : DEFAULT_AUTO_FLUSH_INTERVAL; - this.maxNameLength = isInteger(options.max_name_len, 1) ? options.max_name_len : DEFAULT_MAX_NAME_LENGTH; - this.requestMinThroughput = isInteger(options.request_min_throughput, 0) ? options.request_min_throughput : DEFAULT_REQUEST_MIN_THROUGHPUT; - this.requestTimeout = isInteger(options.request_timeout, 1) ? options.request_timeout : DEFAULT_REQUEST_TIMEOUT; - this.retryTimeout = isInteger(options.retry_timeout, 0) ? options.retry_timeout : DEFAULT_RETRY_TIMEOUT; - const noCopy = isBoolean(options.copy_buffer) && !options.copy_buffer; - this.toBuffer = noCopy ? this.toBufferView : this.toBufferNew; - this.doResolve = noCopy ? (resolve)=>{ - compact(this); - resolve(true); - } : (resolve)=>{ - resolve(true); - }; - this.maxBufferSize = isInteger(options.max_buf_size, 1) ? options.max_buf_size : DEFAULT_MAX_BUFFER_SIZE; - this.resize(isInteger(options.init_buf_size, 1) ? options.init_buf_size : DEFAULT_BUFFER_SIZE); - this.reset(); - } - /** - * Creates a Sender options object by parsing the provided configuration string. - * - * @param {string} configurationString - Configuration string. <br> - * @param {object} extraOptions - Optional extra configuration. <br> - * - 'log' is a logging function used by the <a href="Sender.html">Sender</a>. <br> - * Prototype: <i>(level: 'error'|'warn'|'info'|'debug', message: string) => void</i>. <br> - * - 'agent' is a custom Undici agent used by the <a href="Sender.html">Sender</a> when http/https transport is used. <br> - * A <i>undici.Agent</i> object is expected. - * - * @return {Sender} A Sender object initialized from the provided configuration string. - */ static fromConfig(configurationString, extraOptions = undefined) { - return new Sender(SenderOptions.fromConfig(configurationString, extraOptions)); - } - /** - * Creates a Sender options object by parsing the configuration string set in the <b>QDB_CLIENT_CONF</b> environment variable. - * - * @param {object} extraOptions - Optional extra configuration. <br> - * - 'log' is a logging function used by the <a href="Sender.html">Sender</a>. <br> - * Prototype: <i>(level: 'error'|'warn'|'info'|'debug', message: string) => void</i>. <br> - * - 'agent' is a custom Undici agent used by the <a href="Sender.html">Sender</a> when http/https transport is used. <br> - * A <i>undici.Agent</i> object is expected. - * - * @return {Sender} A Sender object initialized from the <b>QDB_CLIENT_CONF</b> environment variable. - */ static fromEnv(extraOptions = undefined) { - return new Sender(SenderOptions.fromConfig(process.env.QDB_CLIENT_CONF, extraOptions)); - } - /** - * Extends the size of the sender's buffer. <br> - * Can be used to increase the size of buffer if overflown. - * The buffer's content is copied into the new buffer. - * - * @param {number} bufferSize - New size of the buffer used by the sender, provided in bytes. - */ resize(bufferSize) { - if (bufferSize > this.maxBufferSize) { - throw new Error(`Max buffer size is ${this.maxBufferSize} bytes, requested buffer size: ${bufferSize}`); + if (!options.auth && !options.jwk) { + constructAuth(options); } - this.bufferSize = bufferSize; - // Allocating an extra byte because Buffer.write() does not fail if the length of the data to be written is - // longer than the size of the buffer. It simply just writes whatever it can, and returns. - // If we can write into the extra byte, that indicates buffer overflow. - // See the check in our write() function. - const newBuffer = node_buffer.Buffer.alloc(this.bufferSize + 1, 0, "utf8"); - if (this.buffer) { - this.buffer.copy(newBuffer); + this.jwk = constructJwk(options); + if (!options.port) { + options.port = 9009; } - this.buffer = newBuffer; - } - /** - * Resets the buffer, data added to the buffer will be lost. <br> - * In other words it clears the buffer and sets the writing position to the beginning of the buffer. - * - * @return {Sender} Returns with a reference to this sender. - */ reset() { - this.position = 0; - this.lastFlushTime = Date.now(); - this.pendingRowCount = 0; - startNewRow(this); - return this; } /** * Creates a TCP connection to the database. - * - * @param {net.NetConnectOpts | tls.ConnectionOptions} connectOptions - Connection options, host and port are required. - * - * @return {Promise<boolean>} Resolves to true if the client is connected. - */ connect(connectOptions = undefined) { - if (this.http) { - throw new Error("'connect()' should be called only if the sender connects via TCP"); - } - if (!connectOptions) { - connectOptions = { - host: this.host, - port: this.port, - ca: this.tlsCA - }; - } - if (!connectOptions.host) { - throw new Error("Hostname is not set"); - } - if (!connectOptions.port) { - throw new Error("Port is not set"); - } + * @returns Promise resolving to true if the connection is established successfully + * @throws Error if connection fails or authentication is rejected + */ connect() { + const connOptions = { + host: this.host, + port: this.port, + ca: this.tlsCA + }; return new Promise((resolve, reject)=>{ if (this.socket) { throw new Error("Sender connected already"); } let authenticated = false; let data; - this.socket = !this.secure ? net__default.default.connect(connectOptions) : tls__default.default.connect(connectOptions, ()=>{ + this.socket = !this.secure ? net__default.default.connect(connOptions) : tls__default.default.connect(connOptions, ()=>{ if (authenticated) { resolve(true); } @@ -799,7 +944,7 @@

    Source: index.js

    raw ]); if (!authenticated) { - authenticated = await authenticate(this, data); + authenticated = await this.authenticate(data); if (authenticated) { resolve(true); } @@ -807,14 +952,10 @@

    Source: index.js

    this.log("warn", `Received unexpected data: ${data}`); } }).on("ready", async ()=>{ - this.log("info", `Successfully connected to ${connectOptions.host}:${connectOptions.port}`); + this.log("info", `Successfully connected to ${connOptions.host}:${connOptions.port}`); if (this.jwk) { - this.log("info", `Authenticating with ${connectOptions.host}:${connectOptions.port}`); - await this.socket.write(`${this.jwk.kid}\n`, (err)=>{ - if (err) { - reject(err); - } - }); + this.log("info", `Authenticating with ${connOptions.host}:${connOptions.port}`); + this.socket.write(`${this.jwk.kid}\n`, (err)=>err ? reject(err) : ()=>{}); } else { authenticated = true; if (!this.secure || !this.tlsVerify) { @@ -823,24 +964,33 @@

    Source: index.js

    } }).on("error", (err)=>{ this.log("error", err); - if (err.code !== "SELF_SIGNED_CERT_IN_CHAIN" || this.tlsVerify) { + if (this.tlsVerify || !err.code || err.code !== "SELF_SIGNED_CERT_IN_CHAIN") { reject(err); } }); }); } /** - * @ignore - * @return {Agent} Returns the default http agent. - */ getDefaultHttpAgent() { - if (!Sender.DEFAULT_HTTP_AGENT) { - Sender.DEFAULT_HTTP_AGENT = new undici.Agent(DEFAULT_HTTP_OPTIONS); + * Sends data over the established TCP connection. + * @param {Buffer} data - Buffer containing the data to send + * @returns Promise resolving to true if data was sent successfully + * @throws Error if the data could not be written to the socket + */ send(data) { + if (!this.socket || this.socket.destroyed) { + throw new Error("TCP transport is not connected"); } - return Sender.DEFAULT_HTTP_AGENT; + return new Promise((resolve, reject)=>{ + this.socket.write(data, (err)=>{ + if (err) { + reject(err); + } else { + resolve(true); + } + }); + }); } /** - * Closes the TCP connection to the database. <br> - * Data sitting in the Sender's buffer will be lost unless flush() is called before close(). + * Closes the TCP connection to the database. */ async close() { if (this.socket) { const address = this.socket.remoteAddress; @@ -851,186 +1001,587 @@

    Source: index.js

    } } /** - * Sends the buffer's content to the database and compacts the buffer. - * If the last row is not finished it stays in the sender's buffer. - * - * @return {Promise<boolean>} Resolves to true when there was data in the buffer to send. - */ async flush() { - const data = this.toBuffer(this.endOfLastRow); - if (!data) { - return false; - } - if (this.http) { - // const request = this.secure ? https.request : http.request; - const options = createRequestOptions(this, data); - return sendHttp(this, options, data, this.retryTimeout); - } else { - if (!this.socket) { - throw new Error("Sender is not connected"); - } - return sendTcp(this, data); - } - } - /** - * @ignore - * @return {Buffer} Returns a cropped buffer ready to send to the server or null if there is nothing to send. - * The returned buffer is backed by the sender's buffer. - */ toBufferView(pos = this.position) { - return pos > 0 ? this.buffer.subarray(0, pos) : null; + * Gets the default auto-flush row count for TCP transport. + * @returns Default number of rows that trigger auto-flush + */ getDefaultAutoFlushRows() { + return DEFAULT_TCP_AUTO_FLUSH_ROWS; } /** * @ignore - * @return {Buffer|null} Returns a cropped buffer ready to send to the server or null if there is nothing to send. - * The returned buffer is a copy of the sender's buffer. - */ toBufferNew(pos = this.position) { - if (pos > 0) { - const data = node_buffer.Buffer.allocUnsafe(pos); - this.buffer.copy(data, 0, 0, pos); - compact(this); - return data; + * Handles the JWK token authentication challenge-response flow. + * @param {Buffer} challenge - Challenge buffer received from the server + * @returns Promise resolving to true if authentication is successful + */ async authenticate(challenge) { + // Check for trailing \n which ends the challenge + if (challenge.subarray(-1).readInt8() === 10) { + const keyObject = crypto__default.default.createPrivateKey({ + key: this.jwk, + format: "jwk" + }); + const signature = crypto__default.default.sign("RSA-SHA256", challenge.subarray(0, challenge.length - 1), keyObject); + return new Promise((resolve, reject)=>{ + this.socket.write(`${node_buffer.Buffer.from(signature).toString("base64")}\n`, (err)=>{ + if (err) { + reject(err); + } else { + resolve(true); + } + }); + }); } - return null; + return false; } - /** - * Write the table name into the buffer of the sender. - * - * @param {string} table - Table name. - * @return {Sender} Returns with a reference to this sender. - */ table(table) { - if (typeof table !== "string") { - throw new Error(`Table name must be a string, received ${typeof table}`); +} +/** + * @ignore + * Constructs authentication configuration from username/token options. + * @param {SenderOptions} options - Sender options that may contain authentication details + * @throws Error if username or token is missing when authentication is intended + */ function constructAuth(options) { + if (!options.username && !options.token && !options.password) { + // no intention to authenticate + return; + } + if (!options.username || !options.token) { + throw new Error("TCP transport requires a username and a private key for authentication, " + "please, specify the 'username' and 'token' config options"); + } + options.auth = { + keyId: options.username, + token: options.token + }; +} +/** + * @ignore + * Constructs a JWK (JSON Web Key) object for cryptographic authentication. + * @param {SenderOptions} options - Sender options containing authentication configuration + * @returns JWK object with key ID, private key, and public key coordinates + * @throws Error if required authentication properties are missing or invalid + */ function constructJwk(options) { + if (options.auth) { + if (!options.auth.keyId) { + throw new Error("Missing username, please, specify the 'keyId' property of the 'auth' config option. " + "For example: new Sender({protocol: 'tcp', host: 'host', auth: {keyId: 'username', token: 'private key'}})"); } - if (this.hasTable) { - throw new Error("Table name has already been set"); + if (typeof options.auth.keyId !== "string") { + throw new Error("Please, specify the 'keyId' property of the 'auth' config option as a string. " + "For example: new Sender({protocol: 'tcp', host: 'host', auth: {keyId: 'username', token: 'private key'}})"); } - validateTableName(table, this.maxNameLength); - checkCapacity(this, [ - table - ]); - writeEscaped(this, table); - this.hasTable = true; - return this; + if (!options.auth.token) { + throw new Error("Missing private key, please, specify the 'token' property of the 'auth' config option. " + "For example: new Sender({protocol: 'tcp', host: 'host', auth: {keyId: 'username', token: 'private key'}})"); + } + if (typeof options.auth.token !== "string") { + throw new Error("Please, specify the 'token' property of the 'auth' config option as a string. " + "For example: new Sender({protocol: 'tcp', host: 'host', auth: {keyId: 'username', token: 'private key'}})"); + } + return { + kid: options.auth.keyId, + d: options.auth.token, + ...PUBLIC_KEY, + kty: "EC", + crv: "P-256" + }; + } else { + return options.jwk; } +} + +// @ts-check +/** + * Default configuration for HTTP agents. + * - Persistent connections with 1 minute idle timeout + * - Maximum of 256 open connections (matching server default) + */ const DEFAULT_HTTP_AGENT_CONFIG = { + maxSockets: 256, + keepAlive: true, + timeout: 60000 +}; +/** + * HTTP transport implementation using Node.js built-in http/https modules. <br> + * Supports both HTTP and HTTPS protocols with configurable authentication. + */ class HttpTransport extends HttpTransportBase { /** - * Write a symbol name and value into the buffer of the sender. + * Creates a new HttpTransport instance using Node.js HTTP modules. * - * @param {string} name - Symbol name. - * @param {any} value - Symbol value, toString() will be called to extract the actual symbol value from the parameter. - * @return {Sender} Returns with a reference to this sender. - */ symbol(name, value) { - if (typeof name !== "string") { - throw new Error(`Symbol name must be a string, received ${typeof name}`); - } - if (!this.hasTable || this.hasColumns) { - throw new Error("Symbol can be added only after table name is set and before any column added"); + * @param {SenderOptions} options - Sender configuration object containing connection details + * @throws Error if the protocol is not 'http' or 'https' + */ constructor(options){ + super(options); + switch(options.protocol){ + case HTTP: + this.agent = options.agent instanceof http__default.default.Agent ? options.agent : HttpTransport.getDefaultHttpAgent(); + break; + case HTTPS: + this.agent = options.agent instanceof https__default.default.Agent ? options.agent : HttpTransport.getDefaultHttpsAgent(); + break; + default: + throw new Error("The 'protocol' has to be 'http' or 'https' for the HTTP transport"); } - const valueStr = value.toString(); - checkCapacity(this, [ - name, - valueStr - ], 2 + name.length + valueStr.length); - write(this, ","); - validateColumnName(name, this.maxNameLength); - writeEscaped(this, name); - write(this, "="); - writeEscaped(this, valueStr); - this.hasSymbols = true; - return this; } /** - * Write a string column with its value into the buffer of the sender. + * Sends data to QuestDB using HTTP POST. * - * @param {string} name - Column name. - * @param {string} value - Column value, accepts only string values. - * @return {Sender} Returns with a reference to this sender. - */ stringColumn(name, value) { - writeColumn(this, name, value, ()=>{ - checkCapacity(this, [ - value - ], 2 + value.length); - write(this, '"'); - writeEscaped(this, value, true); - write(this, '"'); - }, "string"); - return this; + * @param {Buffer} data - Buffer containing the data to send + * @param {number} retryBegin - Internal parameter for tracking retry start time + * @param {number} retryInterval - Internal parameter for tracking retry intervals + * @returns Promise resolving to true if data was sent successfully + * @throws Error if request fails after all retries or times out + */ send(data, retryBegin = -1, retryInterval = -1) { + const request = this.secure ? https__default.default.request : http__default.default.request; + const timeoutMillis = data.length / this.requestMinThroughput * 1000 + this.requestTimeout; + const options = this.createRequestOptions(timeoutMillis); + return new Promise((resolve, reject)=>{ + let statusCode = -1; + const req = request(options, (response)=>{ + statusCode = response.statusCode; + const body = []; + response.on("data", (chunk)=>{ + body.push(chunk); + }).on("error", (err)=>{ + this.log("error", `resp err=${err}`); + }); + if (statusCode === HTTP_NO_CONTENT) { + response.on("end", ()=>{ + if (body.length > 0) { + const message = node_buffer.Buffer.concat(body).toString(); + const logMessage = message.length < 256 ? message : `${message.substring(0, 256)}... (truncated, full length=${message.length})`; + this.log("warn", `Unexpected message from server: ${logMessage}`); + } + resolve(true); + }); + } else { + req.destroy(new Error(`HTTP request failed, statusCode=${statusCode}, error=${node_buffer.Buffer.concat(body)}`)); + } + }); + if (this.token) { + req.setHeader("Authorization", `Bearer ${this.token}`); + } else if (this.username && this.password) { + req.setHeader("Authorization", `Basic ${node_buffer.Buffer.from(`${this.username}:${this.password}`).toString("base64")}`); + } + req.on("timeout", ()=>{ + // set a retryable error code + statusCode = 524; + req.destroy(new Error("HTTP request timeout, no response from server in time")); + }); + req.on("error", (err)=>{ + // if the error is thrown while the request is sent, statusCode is -1 => no retry + // request timeout comes through with statusCode 524 => retry + // if the error is thrown while the response is processed, the statusCode is taken from the response => retry depends on statusCode + if (isRetryable(statusCode) && this.retryTimeout > 0) { + if (retryBegin < 0) { + retryBegin = Date.now(); + retryInterval = 10; + } else { + const elapsed = Date.now() - retryBegin; + if (elapsed > this.retryTimeout) { + reject(err); + return; + } + } + const jitter = Math.floor(Math.random() * 10) - 5; + setTimeout(()=>{ + retryInterval = Math.min(retryInterval * 2, 1000); + this.send(data, retryBegin, retryInterval).then(()=>resolve(true)).catch((e)=>reject(e)); + }, retryInterval + jitter); + } else { + reject(err); + } + }); + req.write(data, (err)=>err ? reject(err) : ()=>{}); + req.end(); + }); } /** - * Write a boolean column with its value into the buffer of the sender. + * @ignore + * Creates HTTP request options based on configuration. * - * @param {string} name - Column name. - * @param {boolean} value - Column value, accepts only boolean values. - * @return {Sender} Returns with a reference to this sender. - */ booleanColumn(name, value) { - writeColumn(this, name, value, ()=>{ - checkCapacity(this, [], 1); - write(this, value ? "t" : "f"); - }, "boolean"); + * @param {number} timeoutMillis - Request timeout in milliseconds + * @returns HTTP or HTTPS request options object + */ createRequestOptions(timeoutMillis) { + return { + hostname: this.host, + port: this.port, + agent: this.agent, + path: "/write?precision=n", + method: "POST", + timeout: timeoutMillis, + rejectUnauthorized: this.secure && this.tlsVerify, + ca: this.secure ? this.tlsCA : undefined + }; + } + /** + * @ignore + * Gets or creates the default HTTP agent with standard configuration. + * Uses a singleton pattern to reuse the same agent across instances. + * @returns The default HTTP agent instance + */ static getDefaultHttpAgent() { + if (!HttpTransport.DEFAULT_HTTP_AGENT) { + HttpTransport.DEFAULT_HTTP_AGENT = new http__default.default.Agent(DEFAULT_HTTP_AGENT_CONFIG); + } + return HttpTransport.DEFAULT_HTTP_AGENT; + } + /** + * @ignore + * Gets or creates the default HTTPS agent with standard configuration. + * Uses a singleton pattern to reuse the same agent across instances. + * @returns The default HTTPS agent instance + */ static getDefaultHttpsAgent() { + if (!HttpTransport.DEFAULT_HTTPS_AGENT) { + HttpTransport.DEFAULT_HTTPS_AGENT = new https__default.default.Agent(DEFAULT_HTTP_AGENT_CONFIG); + } + return HttpTransport.DEFAULT_HTTPS_AGENT; + } +} +/** + * @ignore + * Determines if an HTTP status code should trigger a retry. + * @param {number} statusCode - HTTP status code to check + * @returns True if the status code indicates a retryable error + */ function isRetryable(statusCode) { + return RETRIABLE_STATUS_CODES.includes(statusCode); +} + +// @ts-check +/** + * Factory function to create appropriate transport instance based on configuration. + * @param {SenderOptions} options - Sender configuration options including protocol and connection details + * @returns Transport instance appropriate for the specified protocol + * @throws Error if protocol or host options are missing or invalid + */ function createTransport(options) { + if (!options || !options.protocol) { + throw new Error("The 'protocol' option is mandatory"); + } + if (!options.host) { + throw new Error("The 'host' option is mandatory"); + } + switch(options.protocol){ + case HTTP: + case HTTPS: + return options.stdlib_http ? new HttpTransport(options) : new UndiciTransport(options); + case TCP: + case TCPS: + return new TcpTransport(options); + default: + throw new Error(`Invalid protocol: '${options.protocol}'`); + } +} + +/** + * Validates a table name. + * + * @param {string} name - The table name to validate. + * @param {number} maxNameLength - The maximum length of table names. + * @throws Error if table name is invalid. + */ function validateTableName(name, maxNameLength) { + const len = name.length; + if (len > maxNameLength) { + throw new Error(`Table name is too long, max length is ${maxNameLength}`); + } + if (len === 0) { + throw new Error("Empty string is not allowed as table name"); + } + for(let i = 0; i < len; i++){ + const ch = name[i]; + switch(ch){ + case ".": + if (i === 0 || i === len - 1 || name[i - 1] === ".") // single dot is allowed in the middle only + // starting with a dot hides directory in Linux + // ending with a dot can be trimmed by some Windows versions / file systems + // double or triple dot looks suspicious + // single dot allowed as compatibility, + // when someone uploads 'file_name.csv' the file name used as the table name + throw new Error("Table name cannot start or end with a dot, and only a single dot allowed"); + break; + case "?": + case ",": + case "'": + case '"': + case "\\": + case "/": + case ":": + case ")": + case "(": + case "+": + case "*": + case "%": + case "~": + case "\u0000": + case "\u0001": + case "\u0002": + case "\u0003": + case "\u0004": + case "\u0005": + case "\u0006": + case "\u0007": + case "\u0008": + case "\u0009": + case "\u000B": + case "\u000c": + case "\r": + case "\n": + case "\u000e": + case "\u000f": + case "\u007f": + case "\ufeff": + throw new Error(`Invalid character in table name: ${ch}`); + } + } +} +/** + * Validates a column name. + * + * @param {string} name - The column name to validate. + * @param {number} maxNameLength - The maximum length of column names. + * @throws Error if column name is invalid. + */ function validateColumnName(name, maxNameLength) { + const len = name.length; + if (len > maxNameLength) { + throw new Error(`Column name is too long, max length is ${maxNameLength}`); + } + if (len === 0) { + throw new Error("Empty string is not allowed as column name"); + } + for (const ch of name){ + switch(ch){ + case "?": + case ".": + case ",": + case "'": + case '"': + case "\\": + case "/": + case ":": + case ")": + case "(": + case "+": + case "-": + case "*": + case "%": + case "~": + case "\u0000": + case "\u0001": + case "\u0002": + case "\u0003": + case "\u0004": + case "\u0005": + case "\u0006": + case "\u0007": + case "\u0008": + case "\u0009": + case "\u000B": + case "\u000c": + case "\r": + case "\n": + case "\u000e": + case "\u000f": + case "\u007f": + case "\ufeff": + throw new Error(`Invalid character in column name: ${ch}`); + } + } +} + +// @ts-check +// Default maximum length for table and column names. +const DEFAULT_MAX_NAME_LENGTH = 127; +/** + * Abstract base class for SenderBuffer implementations. <br> + * Provides common functionality for writing data into the buffer. + */ class SenderBufferBase { + /** + * Creates an instance of SenderBufferBase. + * + * @param {SenderOptions} options - Sender configuration object. <br> + * See SenderOptions documentation for detailed description of configuration options. + */ constructor(options){ + this.log = options && typeof options.log === "function" ? options.log : log; + SenderOptions.resolveDeprecated(options, this.log); + this.maxNameLength = options && isInteger(options.max_name_len, 1) ? options.max_name_len : DEFAULT_MAX_NAME_LENGTH; + this.maxBufferSize = options && isInteger(options.max_buf_size, 1) ? options.max_buf_size : DEFAULT_MAX_BUFFER_SIZE; + this.resize(options && isInteger(options.init_buf_size, 1) ? options.init_buf_size : DEFAULT_BUFFER_SIZE); + this.reset(); + } + /** + * @ignore + * Resizes the buffer. <br> + * Can be used to increase the size of the buffer if data to be written would not fit. + * Creates a new buffer, and copies the content of the old buffer into the new one. + * + * @param {number} bufferSize - New size of the buffer used by the sender, provided in bytes + * @throws Error if the requested buffer size exceeds the maximum allowed size + */ resize(bufferSize) { + if (bufferSize > this.maxBufferSize) { + throw new Error(`Max buffer size is ${this.maxBufferSize} bytes, requested buffer size: ${bufferSize}`); + } + this.bufferSize = bufferSize; + const newBuffer = node_buffer.Buffer.alloc(this.bufferSize, 0); + if (this.buffer) { + this.buffer.copy(newBuffer); + } + this.buffer = newBuffer; + } + /** + * Resets the buffer, data sitting in the buffer will be lost. <br> + * In other words it clears the buffer, and sets the writing position to the beginning of the buffer. + * + * @return {SenderBuffer} Returns with a reference to this buffer. + */ reset() { + this.position = 0; + this.startNewRow(); + return this; + } + startNewRow() { + this.endOfLastRow = this.position; + this.hasTable = false; + this.hasSymbols = false; + this.hasColumns = false; + } + /** + * @return {Buffer} Returns a cropped buffer, or null if there is nothing to send. <br> + * The returned buffer is backed by this buffer instance, meaning the view can change as the buffer is mutated. + * Used only in tests to assert the buffer's content. + */ toBufferView(pos = this.endOfLastRow) { + return pos > 0 ? this.buffer.subarray(0, pos) : null; + } + /** + * @return {Buffer} Returns a cropped buffer ready to send to the server, or null if there is nothing to send. <br> + * The returned buffer is a copy of this buffer. + * It also compacts the buffer. + */ toBufferNew(pos = this.endOfLastRow) { + if (pos > 0) { + const data = node_buffer.Buffer.allocUnsafe(pos); + this.buffer.copy(data, 0, 0, pos); + this.compact(); + return data; + } + return null; + } + /** + * Writes the table name into the buffer. + * + * @param {string} table - Table name. + * @return {SenderBuffer} Returns with a reference to this buffer. + */ table(table) { + if (typeof table !== "string") { + throw new Error(`Table name must be a string, received ${typeof table}`); + } + if (this.hasTable) { + throw new Error("Table name has already been set"); + } + validateTableName(table, this.maxNameLength); + this.checkCapacity([ + table + ], table.length); + this.writeEscaped(table); + this.hasTable = true; + return this; + } + /** + * Writes a symbol name and value into the buffer. <br> + * Use it to insert into SYMBOL columns. + * + * @param {string} name - Symbol name. + * @param {unknown} value - Symbol value, toString() is called to extract the actual symbol value from the parameter. + * @return {SenderBuffer} Returns with a reference to this buffer. + */ symbol(name, value) { + if (typeof name !== "string") { + throw new Error(`Symbol name must be a string, received ${typeof name}`); + } + if (!this.hasTable || this.hasColumns) { + throw new Error("Symbol can be added only after table name is set and before any column added"); + } + const valueStr = value.toString(); + this.checkCapacity([ + name, + valueStr + ], 2 + name.length + valueStr.length); + this.write(","); + validateColumnName(name, this.maxNameLength); + this.writeEscaped(name); + this.write("="); + this.writeEscaped(valueStr); + this.hasSymbols = true; return this; } /** - * Write a float column with its value into the buffer of the sender. + * Writes a string column with its value into the buffer. <br> + * Use it to insert into VARCHAR and STRING columns. * * @param {string} name - Column name. - * @param {number} value - Column value, accepts only number values. - * @return {Sender} Returns with a reference to this sender. - */ floatColumn(name, value) { - writeColumn(this, name, value, ()=>{ - const valueStr = value.toString(); - checkCapacity(this, [ - valueStr - ], valueStr.length); - write(this, valueStr); - }, "number"); + * @param {string} value - Column value, accepts only string values. + * @return {SenderBuffer} Returns with a reference to this buffer. + */ stringColumn(name, value) { + this.writeColumn(name, value, ()=>{ + this.checkCapacity([ + value + ], 2 + value.length); + this.write('"'); + this.writeEscaped(value, true); + this.write('"'); + }, "string"); + return this; + } + /** + * Writes a boolean column with its value into the buffer. <br> + * Use it to insert into BOOLEAN columns. + * + * @param {string} name - Column name. + * @param {boolean} value - Column value, accepts only boolean values. + * @return {SenderBuffer} Returns with a reference to this buffer. + */ booleanColumn(name, value) { + this.writeColumn(name, value, ()=>{ + this.checkCapacity([], 1); + this.write(value ? "t" : "f"); + }, "boolean"); return this; } /** - * Write an integer column with its value into the buffer of the sender. + * Writes a 64-bit signed integer into the buffer. <br> + * Use it to insert into LONG, INT, SHORT and BYTE columns. * * @param {string} name - Column name. * @param {number} value - Column value, accepts only number values. - * @return {Sender} Returns with a reference to this sender. + * @return {SenderBuffer} Returns with a reference to this buffer. + * @throws Error if the value is not an integer */ intColumn(name, value) { if (!Number.isInteger(value)) { throw new Error(`Value must be an integer, received ${value}`); } - writeColumn(this, name, value, ()=>{ + this.writeColumn(name, value, ()=>{ const valueStr = value.toString(); - checkCapacity(this, [ + this.checkCapacity([ valueStr - ], 1 + valueStr.length); - write(this, valueStr); - write(this, "i"); + ], 1); + this.write(valueStr); + this.write("i"); }); return this; } /** - * Write a timestamp column with its value into the buffer of the sender. + * Writes a timestamp column with its value into the buffer. <br> + * Use it to insert into TIMESTAMP columns. * * @param {string} name - Column name. * @param {number | bigint} value - Epoch timestamp, accepts numbers or BigInts. * @param {string} [unit=us] - Timestamp unit. Supported values: 'ns' - nanoseconds, 'us' - microseconds, 'ms' - milliseconds. Defaults to 'us'. - * @return {Sender} Returns with a reference to this sender. + * @return {SenderBuffer} Returns with a reference to this buffer. */ timestampColumn(name, value, unit = "us") { if (typeof value !== "bigint" && !Number.isInteger(value)) { throw new Error(`Value must be an integer or BigInt, received ${value}`); } - writeColumn(this, name, value, ()=>{ + this.writeColumn(name, value, ()=>{ const valueMicros = timestampToMicros(BigInt(value), unit); const valueStr = valueMicros.toString(); - checkCapacity(this, [ + this.checkCapacity([ valueStr - ], 1 + valueStr.length); - write(this, valueStr); - write(this, "t"); + ], 1); + this.write(valueStr); + this.write("t"); }); return this; } /** - * Closing the row after writing the designated timestamp into the buffer of the sender. + * Closes the row after writing the designated timestamp into the buffer. * * @param {number | bigint} timestamp - Designated epoch timestamp, accepts numbers or BigInts. * @param {string} [unit=us] - Timestamp unit. Supported values: 'ns' - nanoseconds, 'us' - microseconds, 'ms' - milliseconds. Defaults to 'us'. - */ async at(timestamp, unit = "us") { + */ at(timestamp, unit = "us") { if (!this.hasSymbols && !this.hasColumns) { throw new Error("The row must have a symbol or column set before it is closed"); } @@ -1039,325 +1590,616 @@

    Source: index.js

    } const timestampNanos = timestampToNanos(BigInt(timestamp), unit); const timestampStr = timestampNanos.toString(); - checkCapacity(this, [], 2 + timestampStr.length); - write(this, " "); - write(this, timestampStr); - write(this, "\n"); - this.pendingRowCount++; - startNewRow(this); - await autoFlush(this); + this.checkCapacity([ + timestampStr + ], 2); + this.write(" "); + this.write(timestampStr); + this.write("\n"); + this.startNewRow(); } /** - * Closing the row without writing designated timestamp into the buffer of the sender. <br> + * Closes the row without writing designated timestamp into the buffer. <br> * Designated timestamp will be populated by the server on this record. - */ async atNow() { + */ atNow() { if (!this.hasSymbols && !this.hasColumns) { throw new Error("The row must have a symbol or column set before it is closed"); } - checkCapacity(this, [], 1); - write(this, "\n"); - this.pendingRowCount++; - startNewRow(this); - await autoFlush(this); + this.checkCapacity([], 1); + this.write("\n"); + this.startNewRow(); } -} -function isBoolean(value) { - return typeof value === "boolean"; -} -function isInteger(value, lowerBound) { - return typeof value === "number" && Number.isInteger(value) && value >= lowerBound; -} -async function authenticate(sender, challenge) { - // Check for trailing \n which ends the challenge - if (challenge.subarray(-1).readInt8() === 10) { - const keyObject = crypto__default.default.createPrivateKey({ - key: sender.jwk, - format: "jwk" - }); - const signature = crypto__default.default.sign("RSA-SHA256", challenge.subarray(0, challenge.length - 1), keyObject); - return new Promise((resolve, reject)=>{ - sender.socket.write(`${node_buffer.Buffer.from(signature).toString("base64")}\n`, (err)=>{ - if (err) { - reject(err); - } else { - resolve(true); - } - }); - }); + /** + * Returns the current position of the buffer. <br> + * New data will be written into the buffer starting from this position. + */ currentPosition() { + return this.position; } - return false; -} -function startNewRow(sender) { - sender.endOfLastRow = sender.position; - sender.hasTable = false; - sender.hasSymbols = false; - sender.hasColumns = false; -} -function createRequestOptions(sender, data) { - const timeoutMillis = data.length / sender.requestMinThroughput * 1000 + sender.requestTimeout; - const options = { - hostname: sender.host, - port: sender.port, - agent: sender.agent, - protocol: sender.secure ? "https" : "http", - path: "/write?precision=n", - method: "POST", - timeout: timeoutMillis - }; - return options; -} -async function sendHttp(sender, options, data, retryTimeout) { - const retryBegin = Date.now(); - const headers = {}; - if (sender.secure) { - sender.agent = new undici.Agent({ - ...DEFAULT_HTTP_OPTIONS, - connect: { - ...DEFAULT_HTTP_OPTIONS.connect, - requestCert: sender.tlsVerify, - rejectUnauthorized: sender.tlsVerify, - ca: sender.tlsCA - } - }); + /** + * Checks if the buffer has sufficient capacity for additional data and resizes if needed. + * @param data - Array of strings to calculate the required capacity for + * @param base - Base number of bytes to add to the calculation + */ checkCapacity(data, base = 0) { + let length = base; + for (const str of data){ + length += node_buffer.Buffer.byteLength(str, "utf8"); + } + if (this.position + length > this.bufferSize) { + let newSize = this.bufferSize; + do { + newSize += this.bufferSize; + }while (this.position + length > newSize) + this.resize(newSize); + } } - const dispatcher = new undici.RetryAgent(sender.agent, { - maxRetries: Infinity, - minTimeout: 10, - maxTimeout: 1000, - timeoutFactor: 2, - retryAfter: true, - methods: [ - "GET", - "POST", - "PUT", - "DELETE", - "PATCH", - "OPTIONS", - "HEAD" - ], - statusCodes: RETRIABLE_STATUS_CODES, - errorCodes: [ - "ECONNRESET", - "EAI_AGAIN", - "ECONNREFUSED", - "ETIMEDOUT", - "EPIPE", - "UND_ERR_CONNECT_TIMEOUT", - "UND_ERR_HEADERS_TIMEOUT", - "UND_ERR_BODY_TIMEOUT" - ], - retry (err, context, callback) { - const elapsed = Date.now() - retryBegin; - if (elapsed > retryTimeout) { - // Stop retrying if the total retry timeout is exceeded - return callback(err); - } - return callback(null); + /** + * @ignore + * Compacts the buffer by removing completed rows. + * Moves any remaining data to the beginning of the buffer. + */ compact() { + if (this.endOfLastRow > 0) { + this.buffer.copy(this.buffer, 0, this.endOfLastRow, this.position); + this.position = this.position - this.endOfLastRow; + this.endOfLastRow = 0; } - }); - if (sender.token) { - headers["Authorization"] = "Bearer " + sender.token; - } else if (sender.username && sender.password) { - headers["Authorization"] = "Basic " + node_buffer.Buffer.from(sender.username + ":" + sender.password).toString("base64"); } - try { - const { statusCode, body } = await dispatcher.request({ - origin: `${options.protocol}://${options.hostname}:${options.port}`, - path: options.path, - method: options.method, - headers, - body: data, - headersTimeout: sender.requestTimeout - }); - const responseBody = await body.arrayBuffer(); - if (statusCode === HTTP_NO_CONTENT) { - if (responseBody.byteLength > 0) { - sender.log("warn", `Unexpected message from server: ${responseBody.toString()}`); - } - return true; - } else { - const error = new Error(`HTTP request failed, statusCode=${statusCode}, error=${responseBody.toString()}`); - throw error; + /** + * @ignore + * Common logic for writing column data to the buffer. + * @param name - Column name + * @param value - Column value + * @param writeValue - Function to write the value portion to the buffer + * @param valueType - Optional expected type for validation + */ writeColumn(name, value, writeValue, valueType) { + if (typeof name !== "string") { + throw new Error(`Column name must be a string, received ${typeof name}`); } - } catch (err) { - if (err.code === "UND_ERR_HEADERS_TIMEOUT") { - sender.log("error", `HTTP request timeout, no response from server in time`); - throw new Error(`HTTP request timeout, no response from server in time`); + if (valueType && typeof value !== valueType) { + throw new Error(`Column value must be of type ${valueType}, received ${typeof value}`); + } + if (!this.hasTable) { + throw new Error("Column can be set only after table name is set"); + } + this.checkCapacity([ + name + ], 2 + name.length); + this.write(this.hasColumns ? "," : " "); + validateColumnName(name, this.maxNameLength); + this.writeEscaped(name); + this.write("="); + writeValue(); + this.hasColumns = true; + } + /** + * @ignore + * Writes string data to the buffer at the current position. + * @param data - String data to write + */ write(data) { + this.position += this.buffer.write(data, this.position); + } + /** + * @ignore + * Writes a single byte to the buffer at the current position. + * @param data - Byte value to write + */ writeByte(data) { + this.position = this.buffer.writeInt8(data, this.position); + } + /** + * @ignore + * Writes a 32-bit integer to the buffer in little-endian format. + * @param data - Integer value to write + */ writeInt(data) { + this.position = this.buffer.writeInt32LE(data, this.position); + } + /** + * @ignore + * Writes a double-precision float to the buffer in little-endian format. + * @param data - Double value to write + */ writeDouble(data) { + this.position = this.buffer.writeDoubleLE(data, this.position); + } + writeEscaped(data, quoted = false) { + for (const ch of data){ + if (ch > "\\") { + this.write(ch); + continue; + } + switch(ch){ + case " ": + case ",": + case "=": + if (!quoted) { + this.write("\\"); + } + this.write(ch); + break; + case "\n": + case "\r": + this.write("\\"); + this.write(ch); + break; + case '"': + if (quoted) { + this.write("\\"); + } + this.write(ch); + break; + case "\\": + this.write("\\\\"); + break; + default: + this.write(ch); + break; + } } - sender.log("error", `HTTP request failed, statusCode=500, error=`); - throw new Error(`HTTP request failed, statusCode=500, error=${err.message}`); } } -async function autoFlush(sender) { - if (sender.autoFlush && sender.pendingRowCount > 0 && (sender.autoFlushRows > 0 && sender.pendingRowCount >= sender.autoFlushRows || sender.autoFlushInterval > 0 && Date.now() - sender.lastFlushTime >= sender.autoFlushInterval)) { - await sender.flush(); + +// @ts-check +/** + * Buffer implementation for protocol version 1. + * Sends floating point numbers in their text form. + */ class SenderBufferV1 extends SenderBufferBase { + /** + * Creates a new SenderBufferV1 instance. + * + * @param {SenderOptions} options - Sender configuration object. <br> + * See SenderOptions documentation for detailed description of configuration options. */ constructor(options){ + super(options); + } + /** + * Writes a 64-bit floating point value into the buffer using v1 serialization (text format). <br> + * Use it to insert into DOUBLE or FLOAT database columns. + * + * @param {string} name - Column name. + * @param {number} value - Column value, accepts only number values. + * @return {Sender} Returns with a reference to this sender. + */ floatColumn(name, value) { + this.writeColumn(name, value, ()=>{ + const valueStr = value.toString(); + this.checkCapacity([ + valueStr + ]); + this.write(valueStr); + }, "number"); + return this; + } + /** + * Array columns are not supported in protocol v1. + * + * @throws Error indicating arrays are not supported in v1 + */ arrayColumn() { + throw new Error("Arrays are not supported in protocol v1"); } } -function sendTcp(sender, data) { - return new Promise((resolve, reject)=>{ - sender.socket.write(data, (err)=>{ - if (err) { - reject(err); + +// @ts-check +// Column type constants for protocol v2. +const COLUMN_TYPE_DOUBLE = 10; +const COLUMN_TYPE_NULL = 33; +// Entity type constants for protocol v2. +const ENTITY_TYPE_ARRAY = 14; +const ENTITY_TYPE_DOUBLE = 16; +// ASCII code for equals sign used in binary protocol. +const EQUALS_SIGN = "=".charCodeAt(0); +/** + * Buffer implementation for protocol version 2. + * Sends floating point numbers in binary form. + */ class SenderBufferV2 extends SenderBufferBase { + /** + * Creates a new SenderBufferV2 instance. + * + * @param {SenderOptions} options - Sender configuration object. <br> + * See SenderOptions documentation for detailed description of configuration options. + */ constructor(options){ + super(options); + } + /** + * Writes a 64-bit floating point value into the buffer using v2 serialization (binary format). <br> + * Use it to insert into DOUBLE or FLOAT database columns. + * + * @param {string} name - Column name. + * @param {number} value - Column value, accepts only number values. + * @returns {Sender} Returns with a reference to this buffer. + */ floatColumn(name, value) { + this.writeColumn(name, value, ()=>{ + this.checkCapacity([], 10); + this.writeByte(EQUALS_SIGN); + this.writeByte(ENTITY_TYPE_DOUBLE); + this.writeDouble(value); + }, "number"); + return this; + } + /** + * Write an array column with its values into the buffer using v2 format. + * + * @param {string} name - Column name + * @param {unknown[]} value - Array values to write (currently supports double arrays) + * @returns {Sender} Returns with a reference to this buffer. + * @throws Error if array validation fails: + * - value is not an array + * - or the shape of the array is irregular: the length of sub-arrays are different + * - or the array is not homogeneous: its elements are not all the same type + */ arrayColumn(name, value) { + const dimensions = getDimensions(value); + const type = validateArray(value, dimensions); + // only number arrays and NULL supported for now + if (type !== "number" && type !== null) { + throw new Error(`Unsupported array type [type=${type}]`); + } + this.writeColumn(name, value, ()=>{ + this.checkCapacity([], 3); + this.writeByte(EQUALS_SIGN); + this.writeByte(ENTITY_TYPE_ARRAY); + if (!value) { + this.writeByte(COLUMN_TYPE_NULL); } else { - sender.doResolve(resolve); + this.writeByte(COLUMN_TYPE_DOUBLE); + this.writeArray(value, dimensions, type); } }); - }); -} -function checkCapacity(sender, data, base = 0) { - let length = base; - for (const str of data){ - length += node_buffer.Buffer.byteLength(str, "utf8"); - } - if (sender.position + length > sender.bufferSize) { - let newSize = sender.bufferSize; - do { - newSize += sender.bufferSize; - }while (sender.position + length > newSize) - sender.resize(newSize); + return this; } -} -function compact(sender) { - if (sender.endOfLastRow > 0) { - sender.buffer.copy(sender.buffer, 0, sender.endOfLastRow, sender.position); - sender.position = sender.position - sender.endOfLastRow; - sender.endOfLastRow = 0; - sender.lastFlushTime = Date.now(); - sender.pendingRowCount = 0; + writeArray(arr, dimensions, type) { + this.checkCapacity([], 1 + dimensions.length * 4); + this.writeByte(dimensions.length); + for(let i = 0; i < dimensions.length; i++){ + this.writeInt(dimensions[i]); + } + this.checkCapacity([], SenderBufferV2.arraySize(dimensions, type)); + this.writeArrayValues(arr, dimensions); } -} -function writeColumn(sender, name, value, writeValue, valueType) { - if (typeof name !== "string") { - throw new Error(`Column name must be a string, received ${typeof name}`); - } - if (valueType != null && typeof value !== valueType) { - throw new Error(`Column value must be of type ${valueType}, received ${typeof value}`); - } - if (!sender.hasTable) { - throw new Error("Column can be set only after table name is set"); - } - checkCapacity(sender, [ - name - ], 2 + name.length); - write(sender, sender.hasColumns ? "," : " "); - validateColumnName(name, sender.maxNameLength); - writeEscaped(sender, name); - write(sender, "="); - writeValue(); - sender.hasColumns = true; -} -function write(sender, data) { - sender.position += sender.buffer.write(data, sender.position); - if (sender.position > sender.bufferSize) { - throw new Error(`Buffer overflow [position=${sender.position}, bufferSize=${sender.bufferSize}]`); + writeArrayValues(arr, dimensions) { + if (Array.isArray(arr[0])) { + for(let i = 0; i < arr.length; i++){ + this.writeArrayValues(arr[i], dimensions); + } + } else { + const type = arr[0] !== undefined ? typeof arr[0] : null; + switch(type){ + case "number": + for(let i = 0; i < arr.length; i++){ + this.position = this.buffer.writeDoubleLE(arr[i], this.position); + } + break; + case null: + break; + default: + throw new Error(`Unsupported array type [type=${type}]`); + } + } } -} -function writeEscaped(sender, data, quoted = false) { - for (const ch of data){ - if (ch > "\\") { - write(sender, ch); - continue; + static arraySize(dimensions, type) { + let numOfElements = 1; + for(let i = 0; i < dimensions.length; i++){ + numOfElements *= dimensions[i]; } - switch(ch){ - case " ": - case ",": - case "=": - if (!quoted) { - write(sender, "\\"); - } - write(sender, ch); - break; - case "\n": - case "\r": - write(sender, "\\"); - write(sender, ch); - break; - case '"': - if (quoted) { - write(sender, "\\"); - } - write(sender, ch); - break; - case "\\": - write(sender, "\\\\"); - break; + switch(type){ + case "number": + return numOfElements * 8; + case "boolean": + return numOfElements; + case "string": + // in case of string[] capacity check is done separately for each array element + return 0; + case null: + // empty array + return 0; default: - write(sender, ch); - break; + throw new Error(`Unsupported array type [type=${type}]`); } } } -function timestampToMicros(timestamp, unit) { - switch(unit){ - case "ns": - return timestamp / 1000n; - case "us": - return timestamp; - case "ms": - return timestamp * 1000n; + +// @ts-check +// Default initial buffer size in bytes (64 KB). +const DEFAULT_BUFFER_SIZE = 65536; // 64 KB +// Default maximum buffer size in bytes (100 MB). +const DEFAULT_MAX_BUFFER_SIZE = 104857600; // 100 MB +/** + * Factory function to create a SenderBuffer instance based on the protocol version. + * + * @param {SenderOptions} options - Sender configuration object. <br> + * See SenderOptions documentation for detailed description of configuration options. + * + * @returns A SenderBuffer instance appropriate for the specified protocol version + * @throws Error if protocol version is not specified or is unsupported + */ function createBuffer(options) { + switch(options.protocol_version){ + case PROTOCOL_VERSION_V2: + return new SenderBufferV2(options); + case PROTOCOL_VERSION_V1: + return new SenderBufferV1(options); + case PROTOCOL_VERSION_AUTO: + case undefined: + case null: + case "": + throw new Error("Provide the 'protocol_version' option, or call 'await SenderOptions.resolveAuto(options)' first"); default: - throw new Error("Unknown timestamp unit: " + unit); + throw new Error("Unsupported protocol version: " + options.protocol_version); } } -function timestampToNanos(timestamp, unit) { - switch(unit){ - case "ns": - return timestamp; - case "us": - return timestamp * 1000n; - case "ms": - return timestamp * 1000_000n; - default: - throw new Error("Unknown timestamp unit: " + unit); + +// @ts-check +const DEFAULT_AUTO_FLUSH_INTERVAL = 1000; // 1 sec +/** @classdesc + * The QuestDB client's API provides methods to connect to the database, ingest data, and close the connection. <br> + * The client supports multiple transport protocols. + * <p> + * <b>Transport Options:</b> + * <ul> + * <li><b>HTTP</b>: Uses standard HTTP requests for data ingestion. Provides immediate feedback via HTTP response codes. + * Recommended for most use cases due to superior error handling and debugging capabilities. Uses Undici library by default for high performance.</li> + * <li><b>HTTPS</b>: Secure HTTP transport with TLS encryption. Same benefits as HTTP but with encrypted communication. + * Supports certificate validation and custom CA certificates.</li> + * <li><b>TCP</b>: Direct TCP connection, provides persistent connections. Uses JWK token-based authentication.</li> + * <li><b>TCPS</b>: Secure TCP transport with TLS encryption.</li> + * </ul> + * </p> + * <p> + * The client supports authentication. <br> + * Authentication details can be passed to the Sender in its configuration options. <br> + * The client supports Basic username/password and Bearer token authentication methods when used with HTTP protocol, + * and JWK token authentication when ingesting data via TCP. <br> + * Please, note that authentication is enabled by default in QuestDB Enterprise only. <br> + * Details on how to configure authentication in the open source version of + * QuestDB: {@link https://questdb.io/docs/reference/api/ilp/authenticate} + * </p> + * <p> + * The client also supports TLS encryption for both, HTTP and TCP transports to provide a secure connection. <br> + * Please, note that the open source version of QuestDB does not support TLS, and requires an external reverse-proxy, + * such as Nginx to enable encryption. + * </p> + * <p> + * The client supports multiple protocol versions for data serialization. Protocol version 1 uses text-based + * serialization, while version 2 uses binary encoding for doubles and supports array columns for improved + * performance. The client can automatically negotiate the protocol version with the server when using HTTP/HTTPS + * by setting the protocol_version to 'auto' (default behavior). + * </p> + * <p> + * The client uses a buffer to store data. It automatically flushes the buffer by sending its content to the server. + * Auto flushing can be disabled via configuration options to gain control over transactions. Initial and maximum + * buffer sizes can also be set. + * </p> + * <p> + * It is recommended that the Sender is created by using one of the static factory methods, + * <i>Sender.fromConfig(configString, extraOptions)</i> or <i>Sender.fromEnv(extraOptions)</i>. + * If the Sender is created via its constructor, at least the SenderOptions configuration object should be + * initialized from a configuration string to make sure that the parameters are validated. <br> + * Detailed description of the Sender's configuration options can be found in + * the <a href="SenderOptions.html">SenderOptions</a> documentation. + * </p> + * <p> + * <b>Transport Configuration Examples:</b> + * <ul> + * <li>HTTP: <i>Sender.fromConfig("http::addr=localhost:9000")</i></li> + * <li>HTTPS with authentication: <i>Sender.fromConfig("https::addr=localhost:9000;username=admin;password=secret")</i></li> + * <li>TCP: <i>Sender.fromConfig("tcp::addr=localhost:9009")</i></li> + * <li>TCPS with authentication: <i>Sender.fromConfig("tcps::addr=localhost:9009;username=user;token=private_key")</i></li> + * </ul> + * </p> + * <p> + * <b>HTTP Transport Implementation:</b><br> + * By default, HTTP/HTTPS transport uses the high-performance Undici library for connection management and request handling. + * For compatibility or specific requirements, you can enable the standard HTTP transport using Node.js built-in modules + * by setting <i>stdlib_http=on</i> in the configuration string. The standard HTTP transport provides the same functionality + * but uses Node.js http/https modules instead of Undici. + * </p> + * <p> + * Extra options can be provided to the Sender in the <i>extraOptions</i> configuration object. <br> + * A custom logging function and a custom HTTP(S) agent can be passed to the Sender in this object. <br> + * The logger implementation provides the option to direct log messages to the same place where the host application's + * log is saved. The default logger writes to the console. <br> + * The custom HTTP(S) agent option becomes handy if there is a need to modify the default options set for the + * HTTP(S) connections. A popular setting would be disabling persistent connections, in this case an agent can be + * passed to the Sender with <i>keepAlive</i> set to <i>false</i>. <br> + * For example: <i>Sender.fromConfig(`http::addr=host:port`, { agent: new undici.Agent({ connect: { keepAlive: false } })})</i> <br> + * If no custom agent is configured, the Sender will use its own agent which overrides some default values + * of <i>undici.Agent</i>. The Sender's own agent uses persistent connections with 1 minute idle timeout, pipelines requests default to 1. + * </p> + */ class Sender { + /** + * Creates an instance of Sender. + * + * @param {SenderOptions} options - Sender configuration object. <br> + * See SenderOptions documentation for detailed description of configuration options. + */ constructor(options){ + this.transport = createTransport(options); + this.buffer = createBuffer(options); + this.log = typeof options.log === "function" ? options.log : log; + this.autoFlush = isBoolean(options.auto_flush) ? options.auto_flush : true; + this.autoFlushRows = isInteger(options.auto_flush_rows, 0) ? options.auto_flush_rows : this.transport.getDefaultAutoFlushRows(); + this.autoFlushInterval = isInteger(options.auto_flush_interval, 0) ? options.auto_flush_interval : DEFAULT_AUTO_FLUSH_INTERVAL; + this.reset(); } -} -function replaceDeprecatedOptions(options) { - // deal with deprecated options - if (options.copyBuffer) { - options.copy_buffer = options.copyBuffer; - options.copyBuffer = undefined; + /** + * Creates a Sender object by parsing the provided configuration string. + * + * @param {string} configurationString - Configuration string. <br> + * @param {object} extraOptions - Optional extra configuration. <br> + * - 'log' is a logging function used by the <a href="Sender.html">Sender</a>. <br> + * Prototype: <i>(level: 'error'|'warn'|'info'|'debug', message: string) => void</i>. <br> + * - 'agent' is a custom http/https agent used by the <a href="Sender.html">Sender</a> when http/https transport is used. <br> + * Depends on which transport implementation and protocol used, one of the followings expected: <i>undici.Agent</i>, <i>http.Agent</i> or <i>https.Agent</i>. + * + * @return {Sender} A Sender object initialized from the provided configuration string. + */ static async fromConfig(configurationString, extraOptions) { + return new Sender(await SenderOptions.fromConfig(configurationString, extraOptions)); } - if (options.bufferSize) { - options.init_buf_size = options.bufferSize; - options.bufferSize = undefined; + /** + * Creates a Sender object by parsing the configuration string set in the <b>QDB_CLIENT_CONF</b> environment variable. + * + * @param {object} extraOptions - Optional extra configuration. <br> + * - 'log' is a logging function used by the <a href="Sender.html">Sender</a>. <br> + * Prototype: <i>(level: 'error'|'warn'|'info'|'debug', message: string) => void</i>. <br> + * - 'agent' is a custom http/https agent used by the <a href="Sender.html">Sender</a> when http/https transport is used. <br> + * Depends on which transport implementation and protocol used, one of the followings expected: <i>undici.Agent</i>, <i>http.Agent</i> or <i>https.Agent</i>. + * + * @return {Sender} A Sender object initialized from the <b>QDB_CLIENT_CONF</b> environment variable. + */ static async fromEnv(extraOptions) { + return new Sender(await SenderOptions.fromConfig(process.env.QDB_CLIENT_CONF, extraOptions)); } -} -function constructAuth(options) { - if (!options.username && !options.token && !options.password) { - // no intention to authenticate - return; + /** + * Resets the sender's buffer, data sitting in the buffer will be lost. <br> + * In other words it clears the buffer, and sets the writing position to the beginning of the buffer. + * + * @return {Sender} Returns with a reference to this sender. + */ reset() { + this.buffer.reset(); + this.resetAutoFlush(); + return this; } - if (!options.username || !options.token) { - throw new Error("TCP transport requires a username and a private key for authentication, " + "please, specify the 'username' and 'token' config options"); + /** + * Creates a TCP connection to the database. + * + * @return {Promise<boolean>} Resolves to true if the client is connected. + */ connect() { + return this.transport.connect(); } - options.auth = { - keyId: options.username, - token: options.token - }; -} -function constructJwk(options) { - if (options.auth) { - if (!options.auth.keyId) { - throw new Error("Missing username, please, specify the 'keyId' property of the 'auth' config option. " + "For example: new Sender({protocol: 'tcp', host: 'host', auth: {keyId: 'username', token: 'private key'}})"); - } - if (typeof options.auth.keyId !== "string") { - throw new Error("Please, specify the 'keyId' property of the 'auth' config option as a string. " + "For example: new Sender({protocol: 'tcp', host: 'host', auth: {keyId: 'username', token: 'private key'}})"); + /** + * Sends the content of the sender's buffer to the database and compacts the buffer. + * If the last row is not finished it stays in the sender's buffer. + * + * @return {Promise<boolean>} Resolves to true when there was data in the buffer to send, and it was sent successfully. + */ async flush() { + const dataToSend = this.buffer.toBufferNew(); + if (!dataToSend) { + return false; // Nothing to send } - if (!options.auth.token) { - throw new Error("Missing private key, please, specify the 'token' property of the 'auth' config option. " + "For example: new Sender({protocol: 'tcp', host: 'host', auth: {keyId: 'username', token: 'private key'}})"); + this.log("debug", `Flushing, number of flushed rows: ${this.pendingRowCount}`); + this.resetAutoFlush(); + await this.transport.send(dataToSend); + } + /** + * Closes the connection to the database. <br> + * Data sitting in the Sender's buffer will be lost unless flush() is called before close(). + */ async close() { + const pos = this.buffer.currentPosition(); + if (pos > 0) { + this.log("warn", `Buffer contains data which has not been flushed before closing the sender, and it will be lost [position=${pos}]`); } - if (typeof options.auth.token !== "string") { - throw new Error("Please, specify the 'token' property of the 'auth' config option as a string. " + "For example: new Sender({protocol: 'tcp', host: 'host', auth: {keyId: 'username', token: 'private key'}})"); + return this.transport.close(); + } + /** + * Writes the table name into the buffer of the sender of the sender. + * + * @param {string} table - Table name. + * @return {Sender} Returns with a reference to this sender. + */ table(table) { + this.buffer.table(table); + return this; + } + /** + * Writes a symbol name and value into the buffer of the sender. <br> + * Use it to insert into SYMBOL columns. + * + * @param {string} name - Symbol name. + * @param {unknown} value - Symbol value, toString() is called to extract the actual symbol value from the parameter. + * @return {Sender} Returns with a reference to this sender. + */ symbol(name, value) { + this.buffer.symbol(name, value); + return this; + } + /** + * Writes a string column with its value into the buffer of the sender. <br> + * Use it to insert into VARCHAR and STRING columns. + * + * @param {string} name - Column name. + * @param {string} value - Column value, accepts only string values. + * @return {Sender} Returns with a reference to this sender. + */ stringColumn(name, value) { + this.buffer.stringColumn(name, value); + return this; + } + /** + * Writes a boolean column with its value into the buffer of the sender. <br> + * Use it to insert into BOOLEAN columns. + * + * @param {string} name - Column name. + * @param {boolean} value - Column value, accepts only boolean values. + * @return {Sender} Returns with a reference to this sender. + */ booleanColumn(name, value) { + this.buffer.booleanColumn(name, value); + return this; + } + /** + * Writes a 64-bit floating point value into the buffer of the sender. <br> + * Use it to insert into DOUBLE or FLOAT database columns. + * + * @param {string} name - Column name. + * @param {number} value - Column value, accepts only number values. + * @return {Sender} Returns with a reference to this sender. + */ floatColumn(name, value) { + this.buffer.floatColumn(name, value); + return this; + } + /** + * Writes an array column with its values into the buffer of the sender. + * + * @param {string} name - Column name + * @param {unknown[]} value - Array values to write (currently supports double arrays) + * @returns {Sender} Returns with a reference to this sender. + * @throws Error if arrays are not supported by the buffer implementation, or array validation fails: + * - value is not an array + * - or the shape of the array is irregular: the length of sub-arrays are different + * - or the array is not homogeneous: its elements are not all the same type + */ arrayColumn(name, value) { + this.buffer.arrayColumn(name, value); + return this; + } + /** + * Writes a 64-bit signed integer into the buffer of the sender. <br> + * Use it to insert into LONG, INT, SHORT and BYTE columns. + * + * @param {string} name - Column name. + * @param {number} value - Column value, accepts only number values. + * @return {Sender} Returns with a reference to this sender. + * @throws Error if the value is not an integer + */ intColumn(name, value) { + this.buffer.intColumn(name, value); + return this; + } + /** + * Writes a timestamp column with its value into the buffer of the sender. <br> + * Use it to insert into TIMESTAMP columns. + * + * @param {string} name - Column name. + * @param {number | bigint} value - Epoch timestamp, accepts numbers or BigInts. + * @param {string} [unit=us] - Timestamp unit. Supported values: 'ns' - nanoseconds, 'us' - microseconds, 'ms' - milliseconds. Defaults to 'us'. + * @return {Sender} Returns with a reference to this sender. + */ timestampColumn(name, value, unit = "us") { + this.buffer.timestampColumn(name, value, unit); + return this; + } + /** + * Closes the row after writing the designated timestamp into the buffer of the sender. + * + * @param {number | bigint} timestamp - Designated epoch timestamp, accepts numbers or BigInts. + * @param {string} [unit=us] - Timestamp unit. Supported values: 'ns' - nanoseconds, 'us' - microseconds, 'ms' - milliseconds. Defaults to 'us'. + */ async at(timestamp, unit = "us") { + this.buffer.at(timestamp, unit); + this.pendingRowCount++; + this.log("debug", `Pending row count: ${this.pendingRowCount}`); + await this.tryFlush(); + } + /** + * Closes the row without writing designated timestamp into the buffer of the sender. <br> + * Designated timestamp will be populated by the server on this record. + */ async atNow() { + this.buffer.atNow(); + this.pendingRowCount++; + this.log("debug", `Pending row count: ${this.pendingRowCount}`); + await this.tryFlush(); + } + resetAutoFlush() { + this.lastFlushTime = Date.now(); + this.pendingRowCount = 0; + this.log("debug", `Pending row count: ${this.pendingRowCount}`); + } + async tryFlush() { + if (this.autoFlush && this.pendingRowCount > 0 && (this.autoFlushRows > 0 && this.pendingRowCount >= this.autoFlushRows || this.autoFlushInterval > 0 && Date.now() - this.lastFlushTime >= this.autoFlushInterval)) { + await this.flush(); } - return { - kid: options.auth.keyId, - d: options.auth.token, - ...PUBLIC_KEY, - kty: "EC", - crv: "P-256" - }; - } else { - return options.jwk; } } @@ -1372,13 +2214,13 @@

    Source: index.js


    - Documentation generated by JSDoc 4.0.4 on Mon Dec 09 2024 02:16:59 GMT+0100 (hora estándar de Europa central) + Documentation generated by JSDoc 4.0.4 on Thu Aug 07 2025 12:45:55 GMT+0100 (British Summer Time)
    diff --git a/docs/scripts/prettify/Apache-License-2.0.txt b/docs/scripts/prettify/Apache-License-2.0.txt old mode 100755 new mode 100644 diff --git a/src/buffer/base.ts b/src/buffer/base.ts index 14e514d..6182a19 100644 --- a/src/buffer/base.ts +++ b/src/buffer/base.ts @@ -42,7 +42,7 @@ abstract class SenderBufferBase implements SenderBuffer { * Creates an instance of SenderBufferBase. * * @param {SenderOptions} options - Sender configuration object.
    - * See SenderOptions documentation for detailed description of configuration options.
    + * See SenderOptions documentation for detailed description of configuration options. */ protected constructor(options: SenderOptions) { this.log = options && typeof options.log === "function" ? options.log : log; @@ -67,11 +67,13 @@ abstract class SenderBufferBase implements SenderBuffer { } /** - * Extends the size of the buffer.
    - * Can be used to increase the size of buffer if overflown. - * The buffer's content is copied into the new buffer. + * @ignore + * Resizes the buffer.
    + * Can be used to increase the size of the buffer if data to be written would not fit. + * Creates a new buffer, and copies the content of the old buffer into the new one. * - * @param {number} bufferSize - New size of the buffer used by the sender, provided in bytes. + * @param {number} bufferSize - New size of the buffer used by the sender, provided in bytes + * @throws Error if the requested buffer size exceeds the maximum allowed size */ private resize(bufferSize: number) { if (bufferSize > this.maxBufferSize) { @@ -88,10 +90,10 @@ abstract class SenderBufferBase implements SenderBuffer { } /** - * Resets the buffer, data added to the buffer will be lost.
    - * In other words it clears the buffer and sets the writing position to the beginning of the buffer. + * Resets the buffer, data sitting in the buffer will be lost.
    + * In other words it clears the buffer, and sets the writing position to the beginning of the buffer. * - * @return {Sender} Returns with a reference to this sender. + * @return {SenderBuffer} Returns with a reference to this buffer. */ reset(): SenderBuffer { this.position = 0; @@ -134,7 +136,7 @@ abstract class SenderBufferBase implements SenderBuffer { * Writes the table name into the buffer. * * @param {string} table - Table name. - * @return {Sender} Returns with a reference to this sender. + * @return {SenderBuffer} Returns with a reference to this buffer. */ table(table: string): SenderBuffer { if (typeof table !== "string") { @@ -156,7 +158,7 @@ abstract class SenderBufferBase implements SenderBuffer { * * @param {string} name - Symbol name. * @param {unknown} value - Symbol value, toString() is called to extract the actual symbol value from the parameter. - * @return {Sender} Returns with a reference to this sender. + * @return {SenderBuffer} Returns with a reference to this buffer. */ symbol(name: string, value: unknown): SenderBuffer { if (typeof name !== "string") { @@ -184,7 +186,7 @@ abstract class SenderBufferBase implements SenderBuffer { * * @param {string} name - Column name. * @param {string} value - Column value, accepts only string values. - * @return {Sender} Returns with a reference to this sender. + * @return {SenderBuffer} Returns with a reference to this buffer. */ stringColumn(name: string, value: string): SenderBuffer { this.writeColumn( @@ -207,7 +209,7 @@ abstract class SenderBufferBase implements SenderBuffer { * * @param {string} name - Column name. * @param {boolean} value - Column value, accepts only boolean values. - * @return {Sender} Returns with a reference to this sender. + * @return {SenderBuffer} Returns with a reference to this buffer. */ booleanColumn(name: string, value: boolean): SenderBuffer { this.writeColumn( @@ -228,16 +230,20 @@ abstract class SenderBufferBase implements SenderBuffer { * * @param {string} name - Column name. * @param {number} value - Column value, accepts only number values. - * @return {Sender} Returns with a reference to this sender. + * @return {SenderBuffer} Returns with a reference to this buffer. */ abstract floatColumn(name: string, value: number): SenderBuffer; /** * Writes an array column with its values into the buffer. * - * @param {string} name - Column name. - * @param {unknown[]} value - Column value, accepts only arrays. - * @return {Sender} Returns with a reference to this sender. + * @param {string} name - Column name + * @param {unknown[]} value - Array values to write (currently supports double arrays) + * @returns {SenderBuffer} Returns with a reference to this buffer. + * @throws Error if arrays are not supported by the buffer implementation, or array validation fails: + * - value is not an array + * - or the shape of the array is irregular: the length of sub-arrays are different + * - or the array is not homogeneous: its elements are not all the same type */ abstract arrayColumn(name: string, value: unknown[]): SenderBuffer; @@ -247,7 +253,7 @@ abstract class SenderBufferBase implements SenderBuffer { * * @param {string} name - Column name. * @param {number} value - Column value, accepts only number values. - * @return {Sender} Returns with a reference to this sender. + * @return {SenderBuffer} Returns with a reference to this buffer. * @throws Error if the value is not an integer */ intColumn(name: string, value: number): SenderBuffer { @@ -270,7 +276,7 @@ abstract class SenderBufferBase implements SenderBuffer { * @param {string} name - Column name. * @param {number | bigint} value - Epoch timestamp, accepts numbers or BigInts. * @param {string} [unit=us] - Timestamp unit. Supported values: 'ns' - nanoseconds, 'us' - microseconds, 'ms' - milliseconds. Defaults to 'us'. - * @return {Sender} Returns with a reference to this sender. + * @return {SenderBuffer} Returns with a reference to this buffer. */ timestampColumn( name: string, @@ -339,6 +345,11 @@ abstract class SenderBufferBase implements SenderBuffer { return this.position; } + /** + * Checks if the buffer has sufficient capacity for additional data and resizes if needed. + * @param data - Array of strings to calculate the required capacity for + * @param base - Base number of bytes to add to the calculation + */ protected checkCapacity(data: string[], base = 0) { let length = base; for (const str of data) { @@ -353,6 +364,11 @@ abstract class SenderBufferBase implements SenderBuffer { } } + /** + * @ignore + * Compacts the buffer by removing completed rows. + * Moves any remaining data to the beginning of the buffer. + */ private compact() { if (this.endOfLastRow > 0) { this.buffer.copy(this.buffer, 0, this.endOfLastRow, this.position); @@ -361,6 +377,14 @@ abstract class SenderBufferBase implements SenderBuffer { } } + /** + * @ignore + * Common logic for writing column data to the buffer. + * @param name - Column name + * @param value - Column value + * @param writeValue - Function to write the value portion to the buffer + * @param valueType - Optional expected type for validation + */ protected writeColumn( name: string, value: unknown, @@ -387,18 +411,38 @@ abstract class SenderBufferBase implements SenderBuffer { this.hasColumns = true; } + /** + * @ignore + * Writes string data to the buffer at the current position. + * @param data - String data to write + */ protected write(data: string) { this.position += this.buffer.write(data, this.position); } + /** + * @ignore + * Writes a single byte to the buffer at the current position. + * @param data - Byte value to write + */ protected writeByte(data: number) { this.position = this.buffer.writeInt8(data, this.position); } + /** + * @ignore + * Writes a 32-bit integer to the buffer in little-endian format. + * @param data - Integer value to write + */ protected writeInt(data: number) { this.position = this.buffer.writeInt32LE(data, this.position); } + /** + * @ignore + * Writes a double-precision float to the buffer in little-endian format. + * @param data - Double value to write + */ protected writeDouble(data: number) { this.position = this.buffer.writeDoubleLE(data, this.position); } diff --git a/src/buffer/bufferv1.ts b/src/buffer/bufferv1.ts index 239d704..16dfecf 100644 --- a/src/buffer/bufferv1.ts +++ b/src/buffer/bufferv1.ts @@ -8,6 +8,11 @@ import { SenderBufferBase } from "./base"; * Sends floating point numbers in their text form. */ class SenderBufferV1 extends SenderBufferBase { + /** + * Creates a new SenderBufferV1 instance. + * + * @param {SenderOptions} options - Sender configuration object.
    + * See SenderOptions documentation for detailed description of configuration options. */ constructor(options: SenderOptions) { super(options); } @@ -34,6 +39,11 @@ class SenderBufferV1 extends SenderBufferBase { return this; } + /** + * Array columns are not supported in protocol v1. + * + * @throws Error indicating arrays are not supported in v1 + */ arrayColumn(): SenderBuffer { throw new Error("Arrays are not supported in protocol v1"); } diff --git a/src/buffer/bufferv2.ts b/src/buffer/bufferv2.ts index 2ad167e..9f471cb 100644 --- a/src/buffer/bufferv2.ts +++ b/src/buffer/bufferv2.ts @@ -4,12 +4,15 @@ import { SenderBuffer } from "./index"; import { SenderBufferBase } from "./base"; import { ArrayPrimitive, getDimensions, validateArray } from "../utils"; +// Column type constants for protocol v2. const COLUMN_TYPE_DOUBLE: number = 10; const COLUMN_TYPE_NULL: number = 33; +// Entity type constants for protocol v2. const ENTITY_TYPE_ARRAY: number = 14; const ENTITY_TYPE_DOUBLE: number = 16; +// ASCII code for equals sign used in binary protocol. const EQUALS_SIGN: number = "=".charCodeAt(0); /** @@ -17,6 +20,12 @@ const EQUALS_SIGN: number = "=".charCodeAt(0); * Sends floating point numbers in binary form. */ class SenderBufferV2 extends SenderBufferBase { + /** + * Creates a new SenderBufferV2 instance. + * + * @param {SenderOptions} options - Sender configuration object.
    + * See SenderOptions documentation for detailed description of configuration options. + */ constructor(options: SenderOptions) { super(options); } @@ -27,7 +36,7 @@ class SenderBufferV2 extends SenderBufferBase { * * @param {string} name - Column name. * @param {number} value - Column value, accepts only number values. - * @return {Sender} Returns with a reference to this sender. + * @returns {Sender} Returns with a reference to this buffer. */ floatColumn(name: string, value: number): SenderBuffer { this.writeColumn( @@ -44,6 +53,17 @@ class SenderBufferV2 extends SenderBufferBase { return this; } + /** + * Write an array column with its values into the buffer using v2 format. + * + * @param {string} name - Column name + * @param {unknown[]} value - Array values to write (currently supports double arrays) + * @returns {Sender} Returns with a reference to this buffer. + * @throws Error if array validation fails: + * - value is not an array + * - or the shape of the array is irregular: the length of sub-arrays are different + * - or the array is not homogeneous: its elements are not all the same type + */ arrayColumn(name: string, value: unknown[]): SenderBuffer { const dimensions = getDimensions(value); const type = validateArray(value, dimensions); diff --git a/src/buffer/index.ts b/src/buffer/index.ts index 0cb1297..f35c78a 100644 --- a/src/buffer/index.ts +++ b/src/buffer/index.ts @@ -11,9 +11,21 @@ import { TimestampUnit } from "../utils"; import { SenderBufferV1 } from "./bufferv1"; import { SenderBufferV2 } from "./bufferv2"; +// Default initial buffer size in bytes (64 KB). const DEFAULT_BUFFER_SIZE = 65536; // 64 KB + +// Default maximum buffer size in bytes (100 MB). const DEFAULT_MAX_BUFFER_SIZE = 104857600; // 100 MB +/** + * Factory function to create a SenderBuffer instance based on the protocol version. + * + * @param {SenderOptions} options - Sender configuration object.
    + * See SenderOptions documentation for detailed description of configuration options. + * + * @returns A SenderBuffer instance appropriate for the specified protocol version + * @throws Error if protocol version is not specified or is unsupported + */ function createBuffer(options: SenderOptions): SenderBuffer { switch (options.protocol_version) { case PROTOCOL_VERSION_V2: @@ -39,8 +51,8 @@ function createBuffer(options: SenderOptions): SenderBuffer { */ interface SenderBuffer { /** - * Resets the buffer, data added to the buffer will be lost.
    - * In other words it clears the buffer and sets the writing position to the beginning of the buffer. + * Resets the buffer, data sitting in the buffer will be lost.
    + * In other words it clears the buffer, and sets the writing position to the beginning of the buffer. * * @return {Sender} Returns with a reference to this sender. */ @@ -69,7 +81,8 @@ interface SenderBuffer { table(table: string): SenderBuffer; /** - * Writes a symbol name and value into the buffer. + * Writes a symbol name and value into the buffer.
    + * Use it to insert into SYMBOL columns. * * @param {string} name - Symbol name. * @param {unknown} value - Symbol value, toString() is called to extract the actual symbol value from the parameter. @@ -78,7 +91,8 @@ interface SenderBuffer { symbol(name: string, value: unknown): SenderBuffer; /** - * Writes a string column with its value into the buffer. + * Writes a string column with its value into the buffer.
    + * Use it to insert into VARCHAR and STRING columns. * * @param {string} name - Column name. * @param {string} value - Column value, accepts only string values. @@ -87,7 +101,8 @@ interface SenderBuffer { stringColumn(name: string, value: string): SenderBuffer; /** - * Writes a boolean column with its value into the buffer. + * Writes a boolean column with its value into the buffer.
    + * Use it to insert into BOOLEAN columns. * * @param {string} name - Column name. * @param {boolean} value - Column value, accepts only boolean values. @@ -96,7 +111,8 @@ interface SenderBuffer { booleanColumn(name: string, value: boolean): SenderBuffer; /** - * Writes a float column with its value into the buffer. + * Writes a 64-bit floating point value into the buffer.
    + * Use it to insert into DOUBLE or FLOAT database columns. * * @param {string} name - Column name. * @param {number} value - Column value, accepts only number values. @@ -104,10 +120,22 @@ interface SenderBuffer { */ floatColumn(name: string, value: number): SenderBuffer; + /** + * Writes an array column with its values into the buffer. + * + * @param {string} name - Column name + * @param {unknown[]} value - Array values to write (currently supports double arrays) + * @returns {Sender} Returns with a reference to this buffer. + * @throws Error if arrays are not supported by the buffer implementation, or array validation fails: + * - value is not an array + * - or the shape of the array is irregular: the length of sub-arrays are different + * - or the array is not homogeneous: its elements are not all the same type + */ arrayColumn(name: string, value: unknown[]): SenderBuffer; /** - * Writes an integer column with its value into the buffer. + * Writes a 64-bit signed integer into the buffer.
    + * Use it to insert into LONG, INT, SHORT and BYTE columns. * * @param {string} name - Column name. * @param {number} value - Column value, accepts only number values. @@ -117,7 +145,8 @@ interface SenderBuffer { intColumn(name: string, value: number): SenderBuffer; /** - * Writes a timestamp column with its value into the buffer. + * Writes a timestamp column with its value into the buffer.
    + * Use it to insert into TIMESTAMP columns. * * @param {string} name - Column name. * @param {number | bigint} value - Epoch timestamp, accepts numbers or BigInts. diff --git a/src/logging.ts b/src/logging.ts index 751629e..e9762fc 100644 --- a/src/logging.ts +++ b/src/logging.ts @@ -1,3 +1,5 @@ +// Log level configuration with console methods and criticality levels.
    +// Higher criticality values indicate more important messages. const LOG_LEVELS = { error: { log: console.error, criticality: 3 }, warn: { log: console.warn, criticality: 2 }, @@ -5,8 +7,15 @@ const LOG_LEVELS = { debug: { log: console.debug, criticality: 0 }, }; +// Default logging criticality level. Messages with criticality below this level are ignored. const DEFAULT_CRITICALITY = LOG_LEVELS.info.criticality; +/** + * Logger function type definition. + * + * @param {'error'|'warn'|'info'|'debug'} level - The log level for the message + * @param {string | Error} message - The message to log, either a string or Error object + */ type Logger = ( level: "error" | "warn" | "info" | "debug", message: string | Error, @@ -17,8 +26,8 @@ type Logger = ( * Supported logging levels are `error`, `warn`, `info` and `debug`.
    * Throws an error if logging level is invalid. * - * @param {'error'|'warn'|'info'|'debug'} level - The log level of the message. - * @param {string | Error} message - The log message. + * @param {'error'|'warn'|'info'|'debug'} level - The log level for the message + * @param {string | Error} message - The message to log, either a string or Error object */ function log( level: "error" | "warn" | "info" | "debug", diff --git a/src/options.ts b/src/options.ts index f1b588c..8d60555 100644 --- a/src/options.ts +++ b/src/options.ts @@ -46,7 +46,7 @@ type DeprecatedOptions = { * Properties of the object are initialized through a configuration string.
    * The configuration string has the following format: <protocol>::<key>=<value><key>=<value>...;
    * The keys are case-sensitive, the trailing semicolon is optional.
    - * The values are validated, and an error is thrown if the format is invalid.
    + * The values are validated and an error is thrown if the format is invalid.
    *
    * Connection and protocol options *
      @@ -54,7 +54,7 @@ type DeprecatedOptions = { * When https or tcps used, the connection is secured with TLS encryption. * *
    • protocol_version: enum, accepted values: auto, 1, 2 - The protocol version used for data serialization.
      - * Version 1 uses text-based serialization for all data types. Version 2 uses binary encoding for doubles.
      + * Version 1 uses text-based serialization for all data types. Version 2 uses binary encoding for doubles and arrays.
      * When set to 'auto' (default for HTTP/HTTPS), the client automatically negotiates the highest supported version with the server.
      * TCP/TCPS connections default to version 1. *
    • @@ -83,8 +83,8 @@ type DeprecatedOptions = { * TLS options *
        *
      • tls_verify: enum, accepted values: on, unsafe_off - When the HTTPS or TCPS protocols are selected, TLS encryption is used.
        - * By default, the Sender will verify the server's certificate, but this check can be disabled by setting this option to off. This is useful - * non-production environments where self-signed certificates might be used, but should be avoided in production if possible. + * By default, the Sender will verify the server's certificate, but this check can be disabled by setting this option to unsafe_off.
        + * This is useful in non-production environments where self-signed certificates might be used, but should be avoided in production if possible. *
      • *
      • tls_ca: string - Path to a file containing the root CA's certificate in PEM format.
        * Can be useful when self-signed certificates are used, otherwise should not be set. @@ -96,9 +96,9 @@ type DeprecatedOptions = { *
      • auto_flush: enum, accepted values: on, off - The Sender automatically flushes the buffer by default. This can be switched off * by setting this option to off.
        * When disabled, the flush() method of the Sender has to be called explicitly to make sure data is sent to the server.
        - * Manual buffer flushing can be useful, especially when we want to use transactions. When the HTTP protocol is used, each flush results in a single HTTP - * request, which becomes a single transaction on the server side. The transaction either succeeds, and all rows sent in the request are - * inserted; or it fails, and none of the rows make it into the database. + * Manual buffer flushing can be useful, especially when we want to control transaction boundaries.
        + * When the HTTP protocol is used, each flush results in a single HTTP request, which becomes a single transaction on the server side.
        + * The transaction either succeeds, and all rows sent in the request are inserted; or it fails, and none of the rows make it into the database. *
      • *
      • auto_flush_rows: integer - The number of rows that will trigger a flush. When set to 0, row-based flushing is disabled.
        * The Sender will default this parameter to 75000 rows when HTTP protocol is used, and to 600 in case of TCP protocol. @@ -197,7 +197,7 @@ class SenderOptions { * - 'log' is a logging function used by the Sender.
        * Prototype: (level: 'error'|'warn'|'info'|'debug', message: string) => void.
        * - 'agent' is a custom http/https agent used by the Sender when http/https transport is used.
        - * A http.Agent or https.Agent object is expected. + * Depends on which transport implementation and protocol used, one of the followings expected: undici.Agent, http.Agent or https.Agent. */ constructor(configurationString: string, extraOptions?: ExtraOptions) { parseConfigurationString(this, configurationString); @@ -224,9 +224,10 @@ class SenderOptions { /** * Resolves the protocol version, if it is set to 'auto'.
        * If TCP transport is used, the protocol version will default to 1. - * In case of HTTP transport the /settings endpoint of the database is used to find the protocol versions + * In case of HTTP transport the /settings endpoint of the database is used to find the protocol versions * supported by the server, and the highest will be selected. - * @param options SenderOptions instance needs resolving protocol version + * When calling the /settings endpoint the timeout and TLs options are used from the options object. + * @param {SenderOptions} options SenderOptions instance needs resolving protocol version */ static async resolveAuto(options: SenderOptions) { parseProtocolVersion(options); @@ -311,7 +312,7 @@ class SenderOptions { * - 'log' is a logging function used by the Sender.
        * Prototype: (level: 'error'|'warn'|'info'|'debug', message: string) => void.
        * - 'agent' is a custom http/https agent used by the Sender when http/https transport is used.
        - * A http.Agent or https.Agent object is expected. + * Depends on which transport implementation and protocol used, one of the followings expected: undici.Agent, http.Agent or https.Agent. * * @return {SenderOptions} A Sender configuration object initialized from the provided configuration string. */ @@ -329,9 +330,9 @@ class SenderOptions { * * @param {object} extraOptions - Optional extra configuration.
        * - 'log' is a logging function used by the Sender.
        - }in /**br> + * Prototype: (level: 'error'|'warn'|'info'|'debug', message: string) => void.
        * - 'agent' is a custom http/https agent used by the Sender when http/https transport is used.
        - * A http.Agent or https.Agent object is expected. + * Depends on which transport implementation and protocol used, one of the followings expected: undici.Agent, http.Agent or https.Agent. * * @return {SenderOptions} A Sender configuration object initialized from the QDB_CLIENT_CONF environment variable. */ diff --git a/src/sender.ts b/src/sender.ts index ae92875..58db7b4 100644 --- a/src/sender.ts +++ b/src/sender.ts @@ -8,9 +8,19 @@ import { isBoolean, isInteger, TimestampUnit } from "./utils"; const DEFAULT_AUTO_FLUSH_INTERVAL = 1000; // 1 sec /** @classdesc - * The QuestDB client's API provides methods to connect to the database, ingest data, and close the connection. - * The supported protocols are HTTP and TCP. HTTP is preferred as it provides feedback in the HTTP response.
        - * Based on benchmarks HTTP also provides higher throughput, if configured to ingest data in bigger batches. + * The QuestDB client's API provides methods to connect to the database, ingest data, and close the connection.
        + * The client supports multiple transport protocols. + *

        + * Transport Options: + *

          + *
        • HTTP: Uses standard HTTP requests for data ingestion. Provides immediate feedback via HTTP response codes. + * Recommended for most use cases due to superior error handling and debugging capabilities. Uses Undici library by default for high performance.
        • + *
        • HTTPS: Secure HTTP transport with TLS encryption. Same benefits as HTTP but with encrypted communication. + * Supports certificate validation and custom CA certificates.
        • + *
        • TCP: Direct TCP connection, provides persistent connections. Uses JWK token-based authentication.
        • + *
        • TCPS: Secure TCP transport with TLS encryption.
        • + *
        + *

        *

        * The client supports authentication.
        * Authentication details can be passed to the Sender in its configuration options.
        @@ -26,6 +36,12 @@ const DEFAULT_AUTO_FLUSH_INTERVAL = 1000; // 1 sec * such as Nginx to enable encryption. *

        *

        + * The client supports multiple protocol versions for data serialization. Protocol version 1 uses text-based + * serialization, while version 2 uses binary encoding for doubles and supports array columns for improved + * performance. The client can automatically negotiate the protocol version with the server when using HTTP/HTTPS + * by setting the protocol_version to 'auto' (default behavior). + *

        + *

        * The client uses a buffer to store data. It automatically flushes the buffer by sending its content to the server. * Auto flushing can be disabled via configuration options to gain control over transactions. Initial and maximum * buffer sizes can also be set. @@ -39,6 +55,22 @@ const DEFAULT_AUTO_FLUSH_INTERVAL = 1000; // 1 sec * the SenderOptions documentation. *

        *

        + * Transport Configuration Examples: + *

          + *
        • HTTP: Sender.fromConfig("http::addr=localhost:9000")
        • + *
        • HTTPS with authentication: Sender.fromConfig("https::addr=localhost:9000;username=admin;password=secret")
        • + *
        • TCP: Sender.fromConfig("tcp::addr=localhost:9009")
        • + *
        • TCPS with authentication: Sender.fromConfig("tcps::addr=localhost:9009;username=user;token=private_key")
        • + *
        + *

        + *

        + * HTTP Transport Implementation:
        + * By default, HTTP/HTTPS transport uses the high-performance Undici library for connection management and request handling. + * For compatibility or specific requirements, you can enable the standard HTTP transport using Node.js built-in modules + * by setting stdlib_http=on in the configuration string. The standard HTTP transport provides the same functionality + * but uses Node.js http/https modules instead of Undici. + *

        + *

        * Extra options can be provided to the Sender in the extraOptions configuration object.
        * A custom logging function and a custom HTTP(S) agent can be passed to the Sender in this object.
        * The logger implementation provides the option to direct log messages to the same place where the host application's @@ -68,7 +100,7 @@ class Sender { * Creates an instance of Sender. * * @param {SenderOptions} options - Sender configuration object.
        - * See SenderOptions documentation for detailed description of configuration options.
        + * See SenderOptions documentation for detailed description of configuration options. */ constructor(options: SenderOptions) { this.transport = createTransport(options); @@ -88,14 +120,14 @@ class Sender { } /** - * Creates a Sender options object by parsing the provided configuration string. + * Creates a Sender object by parsing the provided configuration string. * * @param {string} configurationString - Configuration string.
        * @param {object} extraOptions - Optional extra configuration.
        * - 'log' is a logging function used by the Sender.
        * Prototype: (level: 'error'|'warn'|'info'|'debug', message: string) => void.
        - * - 'agent' is a custom Undici agent used by the Sender when http/https transport is used.
        - * A undici.Agent object is expected. + * - 'agent' is a custom http/https agent used by the Sender when http/https transport is used.
        + * Depends on which transport implementation and protocol used, one of the followings expected: undici.Agent, http.Agent or https.Agent. * * @return {Sender} A Sender object initialized from the provided configuration string. */ @@ -109,13 +141,13 @@ class Sender { } /** - * Creates a Sender options object by parsing the configuration string set in the QDB_CLIENT_CONF environment variable. + * Creates a Sender object by parsing the configuration string set in the QDB_CLIENT_CONF environment variable. * * @param {object} extraOptions - Optional extra configuration.
        * - 'log' is a logging function used by the Sender.
        * Prototype: (level: 'error'|'warn'|'info'|'debug', message: string) => void.
        - * - 'agent' is a custom Undici agent used by the Sender when http/https transport is used.
        - * A undici.Agent object is expected. + * - 'agent' is a custom http/https agent used by the Sender when http/https transport is used.
        + * Depends on which transport implementation and protocol used, one of the followings expected: undici.Agent, http.Agent or https.Agent. * * @return {Sender} A Sender object initialized from the QDB_CLIENT_CONF environment variable. */ @@ -126,8 +158,8 @@ class Sender { } /** - * Resets the buffer, data added to the buffer will be lost.
        - * In other words it clears the buffer and sets the writing position to the beginning of the buffer. + * Resets the sender's buffer, data sitting in the buffer will be lost.
        + * In other words it clears the buffer, and sets the writing position to the beginning of the buffer. * * @return {Sender} Returns with a reference to this sender. */ @@ -147,7 +179,7 @@ class Sender { } /** - * Sends the buffer's content to the database and compacts the buffer. + * Sends the content of the sender's buffer to the database and compacts the buffer. * If the last row is not finished it stays in the sender's buffer. * * @return {Promise} Resolves to true when there was data in the buffer to send, and it was sent successfully. @@ -183,7 +215,7 @@ class Sender { } /** - * Writes the table name into the buffer of the sender. + * Writes the table name into the buffer of the sender of the sender. * * @param {string} table - Table name. * @return {Sender} Returns with a reference to this sender. @@ -194,7 +226,8 @@ class Sender { } /** - * Writes a symbol name and value into the buffer of the sender. + * Writes a symbol name and value into the buffer of the sender.
        + * Use it to insert into SYMBOL columns. * * @param {string} name - Symbol name. * @param {unknown} value - Symbol value, toString() is called to extract the actual symbol value from the parameter. @@ -206,7 +239,8 @@ class Sender { } /** - * Writes a string column with its value into the buffer of the sender. + * Writes a string column with its value into the buffer of the sender.
        + * Use it to insert into VARCHAR and STRING columns. * * @param {string} name - Column name. * @param {string} value - Column value, accepts only string values. @@ -218,7 +252,8 @@ class Sender { } /** - * Writes a boolean column with its value into the buffer of the sender. + * Writes a boolean column with its value into the buffer of the sender.
        + * Use it to insert into BOOLEAN columns. * * @param {string} name - Column name. * @param {boolean} value - Column value, accepts only boolean values. @@ -230,7 +265,8 @@ class Sender { } /** - * Writes a float column with its value into the buffer of the sender. + * Writes a 64-bit floating point value into the buffer of the sender.
        + * Use it to insert into DOUBLE or FLOAT database columns. * * @param {string} name - Column name. * @param {number} value - Column value, accepts only number values. @@ -241,17 +277,30 @@ class Sender { return this; } + /** + * Writes an array column with its values into the buffer of the sender. + * + * @param {string} name - Column name + * @param {unknown[]} value - Array values to write (currently supports double arrays) + * @returns {Sender} Returns with a reference to this sender. + * @throws Error if arrays are not supported by the buffer implementation, or array validation fails: + * - value is not an array + * - or the shape of the array is irregular: the length of sub-arrays are different + * - or the array is not homogeneous: its elements are not all the same type + */ arrayColumn(name: string, value: unknown[]): Sender { this.buffer.arrayColumn(name, value); return this; } /** - * Writes an integer column with its value into the buffer of the sender. + * Writes a 64-bit signed integer into the buffer of the sender.
        + * Use it to insert into LONG, INT, SHORT and BYTE columns. * * @param {string} name - Column name. * @param {number} value - Column value, accepts only number values. * @return {Sender} Returns with a reference to this sender. + * @throws Error if the value is not an integer */ intColumn(name: string, value: number): Sender { this.buffer.intColumn(name, value); @@ -259,7 +308,8 @@ class Sender { } /** - * Writes a timestamp column with its value into the buffer of the sender. + * Writes a timestamp column with its value into the buffer of the sender.
        + * Use it to insert into TIMESTAMP columns. * * @param {string} name - Column name. * @param {number | bigint} value - Epoch timestamp, accepts numbers or BigInts. diff --git a/src/transport/http/base.ts b/src/transport/http/base.ts index f4f8dd7..c772899 100644 --- a/src/transport/http/base.ts +++ b/src/transport/http/base.ts @@ -7,28 +7,23 @@ import { SenderOptions, HTTP, HTTPS } from "../../options"; import { SenderTransport } from "../index"; import { isBoolean, isInteger } from "../../utils"; -const HTTP_NO_CONTENT = 204; // success +// HTTP status code for successful request with no content. +const HTTP_NO_CONTENT = 204; +// Default number of rows that trigger auto-flush for HTTP transport. const DEFAULT_HTTP_AUTO_FLUSH_ROWS = 75000; -const DEFAULT_REQUEST_MIN_THROUGHPUT = 102400; // 100 KB/sec -const DEFAULT_REQUEST_TIMEOUT = 10000; // 10 sec -const DEFAULT_RETRY_TIMEOUT = 10000; // 10 sec - -/* -We are retrying on the following response codes (copied from the Rust client): -500: Internal Server Error -503: Service Unavailable -504: Gateway Timeout - -// Unofficial extensions -507: Insufficient Storage -509: Bandwidth Limit Exceeded -523: Origin is Unreachable -524: A Timeout Occurred -529: Site is overloaded -599: Network Connect Timeout Error -*/ +// Default minimum throughput for HTTP requests (100 KB/sec). +const DEFAULT_REQUEST_MIN_THROUGHPUT = 102400; + +// Default request timeout in milliseconds (10 seconds). +const DEFAULT_REQUEST_TIMEOUT = 10000; + +// Default retry timeout in milliseconds (10 seconds). +const DEFAULT_RETRY_TIMEOUT = 10000; + +// HTTP status codes that should trigger request retries. +// Includes server errors and gateway timeouts that may be transient. const RETRIABLE_STATUS_CODES = [500, 503, 504, 507, 509, 523, 524, 529, 599]; /** @@ -55,7 +50,8 @@ abstract class HttpTransportBase implements SenderTransport { /** * Creates a new HttpTransportBase instance. - * @param options - Sender configuration options including connection and authentication details + * + * @param {SenderOptions} options - Sender configuration options including connection and authentication details * @throws Error if required protocol or host options are missing */ protected constructor(options: SenderOptions) { @@ -120,7 +116,7 @@ abstract class HttpTransportBase implements SenderTransport { /** * Gets the default auto-flush row count for HTTP transport. - * @returns Default number of rows that trigger auto-flush + * @returns {number} Default number of rows that trigger auto-flush */ getDefaultAutoFlushRows(): number { return DEFAULT_HTTP_AUTO_FLUSH_ROWS; @@ -129,7 +125,7 @@ abstract class HttpTransportBase implements SenderTransport { /** * Sends data to the QuestDB server via HTTP. * Must be implemented by concrete HTTP transport classes. - * @param data - Buffer containing the data to send + * @param {Buffer} data - Buffer containing the data to send * @returns Promise resolving to true if data was sent successfully */ abstract send(data: Buffer): Promise; diff --git a/src/transport/http/stdlib.ts b/src/transport/http/stdlib.ts index 8717c41..af840e6 100644 --- a/src/transport/http/stdlib.ts +++ b/src/transport/http/stdlib.ts @@ -34,7 +34,7 @@ class HttpTransport extends HttpTransportBase { /** * Creates a new HttpTransport instance using Node.js HTTP modules. * - * @param options - Sender configuration object containing connection details + * @param {SenderOptions} options - Sender configuration object containing connection details * @throws Error if the protocol is not 'http' or 'https' */ constructor(options: SenderOptions) { @@ -62,13 +62,18 @@ class HttpTransport extends HttpTransportBase { /** * Sends data to QuestDB using HTTP POST. - * @param data - Buffer containing data to send - * @param retryBegin - Internal parameter for tracking retry start time - * @param retryInterval - Internal parameter for tracking retry intervals + * + * @param {Buffer} data - Buffer containing the data to send + * @param {number} retryBegin - Internal parameter for tracking retry start time + * @param {number} retryInterval - Internal parameter for tracking retry intervals * @returns Promise resolving to true if data was sent successfully * @throws Error if request fails after all retries or times out */ - send(data: Buffer, retryBegin = -1, retryInterval = -1): Promise { + send( + data: Buffer, + retryBegin: number = -1, + retryInterval: number = -1, + ): Promise { const request = this.secure ? https.request : http.request; const timeoutMillis = @@ -157,6 +162,13 @@ class HttpTransport extends HttpTransportBase { }); } + /** + * @ignore + * Creates HTTP request options based on configuration. + * + * @param {number} timeoutMillis - Request timeout in milliseconds + * @returns HTTP or HTTPS request options object + */ private createRequestOptions( timeoutMillis: number, ): http.RequestOptions | https.RequestOptions { @@ -174,7 +186,9 @@ class HttpTransport extends HttpTransportBase { /** * @ignore - * @return {http.Agent} Returns the default http agent. + * Gets or creates the default HTTP agent with standard configuration. + * Uses a singleton pattern to reuse the same agent across instances. + * @returns The default HTTP agent instance */ private static getDefaultHttpAgent(): http.Agent { if (!HttpTransport.DEFAULT_HTTP_AGENT) { @@ -187,7 +201,9 @@ class HttpTransport extends HttpTransportBase { /** * @ignore - * @return {https.Agent} Returns the default https agent. + * Gets or creates the default HTTPS agent with standard configuration. + * Uses a singleton pattern to reuse the same agent across instances. + * @returns The default HTTPS agent instance */ private static getDefaultHttpsAgent(): https.Agent { if (!HttpTransport.DEFAULT_HTTPS_AGENT) { @@ -199,6 +215,12 @@ class HttpTransport extends HttpTransportBase { } } +/** + * @ignore + * Determines if an HTTP status code should trigger a retry. + * @param {number} statusCode - HTTP status code to check + * @returns True if the status code indicates a retryable error + */ function isRetryable(statusCode: number) { return RETRIABLE_STATUS_CODES.includes(statusCode); } diff --git a/src/transport/http/undici.ts b/src/transport/http/undici.ts index 9d22349..a24dadb 100644 --- a/src/transport/http/undici.ts +++ b/src/transport/http/undici.ts @@ -94,7 +94,8 @@ class UndiciTransport extends HttpTransportBase { /** * Sends data to QuestDB using HTTP POST. - * @param data - Buffer containing data to send + * + * @param {Buffer} data - Buffer containing the data to send * @returns Promise resolving to true if data was sent successfully * @throws Error if request fails after all retries or times out */ @@ -156,7 +157,9 @@ class UndiciTransport extends HttpTransportBase { /** * @ignore - * @return {Agent} Returns the default http agent. + * Gets or creates the default HTTP agent with standard configuration. + * Uses a singleton pattern to reuse the same agent across instances. + * @returns The default Undici agent instance */ private static getDefaultHttpAgent(): Agent { if (!UndiciTransport.DEFAULT_HTTP_AGENT) { diff --git a/src/transport/index.ts b/src/transport/index.ts index 380040a..e81a7db 100644 --- a/src/transport/index.ts +++ b/src/transport/index.ts @@ -19,8 +19,8 @@ interface SenderTransport { connect(): Promise; /** - * Sends buffered data to the database server. - * @param data - Buffer containing the data to send + * Sends the data to the database server. + * @param {Buffer} data - Buffer containing the data to send * @returns Promise resolving to true if data was sent successfully */ send(data: Buffer): Promise; @@ -41,7 +41,7 @@ interface SenderTransport { /** * Factory function to create appropriate transport instance based on configuration. - * @param options - Sender configuration options including protocol and connection details + * @param {SenderOptions} options - Sender configuration options including protocol and connection details * @returns Transport instance appropriate for the specified protocol * @throws Error if protocol or host options are missing or invalid */ diff --git a/src/transport/tcp.ts b/src/transport/tcp.ts index bc84464..57b5026 100644 --- a/src/transport/tcp.ts +++ b/src/transport/tcp.ts @@ -10,6 +10,7 @@ import { SenderOptions, TCP, TCPS } from "../options"; import { SenderTransport } from "./index"; import { isBoolean } from "../utils"; +// Default number of rows that trigger auto-flush for TCP transport. const DEFAULT_TCP_AUTO_FLUSH_ROWS = 600; // Arbitrary public key, used to construct valid JWK tokens. @@ -39,7 +40,7 @@ class TcpTransport implements SenderTransport { /** * Creates a new TcpTransport instance. * - * @param options - Sender configuration object containing connection and authentication details + * @param {SenderOptions} options - Sender configuration object containing connection and authentication details * @throws Error if required options are missing or protocol is not 'tcp' or 'tcps' */ constructor(options: SenderOptions) { @@ -155,7 +156,7 @@ class TcpTransport implements SenderTransport { /** * Sends data over the established TCP connection. - * @param data - Buffer containing the data to send + * @param {Buffer} data - Buffer containing the data to send * @returns Promise resolving to true if data was sent successfully * @throws Error if the data could not be written to the socket */ @@ -175,7 +176,7 @@ class TcpTransport implements SenderTransport { } /** - * Closes the TCP connection to the database.
        + * Closes the TCP connection to the database. */ async close(): Promise { if (this.socket) { @@ -187,10 +188,20 @@ class TcpTransport implements SenderTransport { } } + /** + * Gets the default auto-flush row count for TCP transport. + * @returns Default number of rows that trigger auto-flush + */ getDefaultAutoFlushRows(): number { return DEFAULT_TCP_AUTO_FLUSH_ROWS; } + /** + * @ignore + * Handles the JWK token authentication challenge-response flow. + * @param {Buffer} challenge - Challenge buffer received from the server + * @returns Promise resolving to true if authentication is successful + */ private async authenticate(challenge: Buffer): Promise { // Check for trailing \n which ends the challenge if (challenge.subarray(-1).readInt8() === 10) { @@ -221,6 +232,12 @@ class TcpTransport implements SenderTransport { } } +/** + * @ignore + * Constructs authentication configuration from username/token options. + * @param {SenderOptions} options - Sender options that may contain authentication details + * @throws Error if username or token is missing when authentication is intended + */ function constructAuth(options: SenderOptions) { if (!options.username && !options.token && !options.password) { // no intention to authenticate @@ -239,6 +256,13 @@ function constructAuth(options: SenderOptions) { }; } +/** + * @ignore + * Constructs a JWK (JSON Web Key) object for cryptographic authentication. + * @param {SenderOptions} options - Sender options containing authentication configuration + * @returns JWK object with key ID, private key, and public key coordinates + * @throws Error if required authentication properties are missing or invalid + */ function constructJwk(options: SenderOptions) { if (options.auth) { if (!options.auth.keyId) { diff --git a/src/utils.ts b/src/utils.ts index 4141f20..d7f580c 100644 --- a/src/utils.ts +++ b/src/utils.ts @@ -1,19 +1,44 @@ import { Agent } from "undici"; +/** + * Primitive types for QuestDB arrays.
        + * Currently only number arrays are supported by the server. + */ type ArrayPrimitive = "number" | "boolean" | "string" | null; +/** + * Supported timestamp units for QuestDB operations. + */ type TimestampUnit = "ns" | "us" | "ms"; +/** + * Type guard to check if a value is a boolean. + * @param {unknown} value - The value to check + * @returns True if the value is a boolean, false otherwise + */ function isBoolean(value: unknown): value is boolean { return typeof value === "boolean"; } +/** + * Type guard to check if a value is an integer within specified bounds. + * @param {unknown} value - The value to check + * @param {number} lowerBound - The minimum allowed value (inclusive) + * @returns True if the value is an integer >= lowerBound, false otherwise + */ function isInteger(value: unknown, lowerBound: number): value is number { return ( typeof value === "number" && Number.isInteger(value) && value >= lowerBound ); } +/** + * Converts a timestamp from the specified unit to microseconds. + * @param {bigint} timestamp - The timestamp value as a bigint + * @param {TimestampUnit} unit - The source timestamp unit + * @returns The timestamp converted to microseconds + * @throws Error if the timestamp unit is unknown + */ function timestampToMicros(timestamp: bigint, unit: TimestampUnit) { switch (unit) { case "ns": @@ -27,6 +52,13 @@ function timestampToMicros(timestamp: bigint, unit: TimestampUnit) { } } +/** + * Converts a timestamp from the specified unit to nanoseconds. + * @param {bigint} timestamp - The timestamp value as a bigint + * @param {TimestampUnit} unit - The source timestamp unit + * @returns The timestamp converted to nanoseconds + * @throws Error if the timestamp unit is unknown + */ function timestampToNanos(timestamp: bigint, unit: TimestampUnit) { switch (unit) { case "ns": @@ -40,6 +72,12 @@ function timestampToNanos(timestamp: bigint, unit: TimestampUnit) { } } +/** + * Analyzes the dimensions of a nested array structure. + * @param {unknown} data - The array to analyze + * @returns Array of dimension sizes at each nesting level + * @throws Error if any dimension has zero length + */ function getDimensions(data: unknown) { const dimensions: number[] = []; while (Array.isArray(data)) { @@ -49,6 +87,17 @@ function getDimensions(data: unknown) { return dimensions; } +/** + * Validates an array structure.
        + * Validation fails if: + * - data is not an array + * - the array is irregular: the length of its sub-arrays are different + * - the array is not homogenous: the array contains mixed types + * @param {unknown[]} data - The array to validate + * @param {number[]} dimensions - The shape of the array + * @returns The primitive type of the array's elements + * @throws Error if the validation fails + */ function validateArray(data: unknown[], dimensions: number[]): ArrayPrimitive { if (data === null || data === undefined) { return null; @@ -109,9 +158,9 @@ function validateArray(data: unknown[], dimensions: number[]): ArrayPrimitive { /** * Fetches JSON data from a URL. * @template T - The expected type of the JSON response - * @param url - The URL to fetch from - * @param agent - HTTP agent to be used for the request - * @param timeout - Request timeout, query will be aborted if not finished in time + * @param {string} url - The URL to fetch from + * @param {Agent} agent - HTTP agent to be used for the request + * @param {number} timeout - Request timeout, query will be aborted if not finished in time * @returns Promise resolving to the parsed JSON data * @throws Error if the request fails or returns a non-OK status */ diff --git a/src/validation.ts b/src/validation.ts index 7ebea0b..e358d35 100644 --- a/src/validation.ts +++ b/src/validation.ts @@ -1,9 +1,9 @@ /** - * Validates a table name.
        - * Throws an error if table name is invalid. + * Validates a table name. * * @param {string} name - The table name to validate. * @param {number} maxNameLength - The maximum length of table names. + * @throws Error if table name is invalid. */ function validateTableName(name: string, maxNameLength: number) { const len = name.length; @@ -65,11 +65,11 @@ function validateTableName(name: string, maxNameLength: number) { } /** - * Validates a column name.
        - * Throws an error if column name is invalid. + * Validates a column name. * * @param {string} name - The column name to validate. * @param {number} maxNameLength - The maximum length of column names. + * @throws Error if column name is invalid. */ function validateColumnName(name: string, maxNameLength: number) { const len = name.length;