Merge pull request #9 from Edujugon/dev

V2 - Worker parallelizer

Author: Edujugon

README.md

@@ -1,21 +1,30 @@
 # Node Parallelizer
-A NodeJS package for running code in parallel. Initially created to provide multiprocessing in an AWS Lambda function, but it can be used in any NodeJS environment.
+A NodeJS package for running code in parallel. Initially created to provide multiprocessing in an **AWS Lambda function**, but it can be used in any NodeJS environment.
 
 ## Supported parallelizers
 - Child Process
-- Worker threads [Coming soon]
+- Worker threads
 
 ### Child Process Parallelizer
-This parallelizer is specifically designed for processing hundreds or thousands of records in a single invocation when your code performs both CPU-intensive and I/O-intensive operations. It uses the NodeJS [child process module](https://nodejs.org/api/child_process.html) behind the scenes.
+This parallelizer is specifically designed for processing hundreds or thousands of records in a single invocation when your code performs both CPU-intensive and **I/O-intensive operations**.
 
 When you call the `runBatch(records)` method in this parallelizer, the package will split the list of records you provide into smaller subsets, and your code will be used to execute each subset in parallel.
 
-## AWS Lambda & Child Process Parallelizer
-The package can detect the number of vCPU cores allocated to your Lambda function and maximize their utilization. By default, it generates one child process per vCPU core, but this setting can be customized to meet your specific requirements. Alternatively, you can manually specify the number of child processes the library creates, regardless of the number of vCPU cores available.
+It uses the NodeJS [child process module](https://nodejs.org/api/child_process.html) behind the scenes.
+
+### Worker Threads Parallelizer
+This parallelizer is specifically designed for processing hundreds or thousands of records in a single invocation when your code performs **CPU-intensive operations**.
+
+When you call the `runBatch(records)` method in this parallelizer, the package will split the list of records you provide into smaller subsets, and your code will be used to execute each subset in parallel.
+
+It uses the NodeJS [worker threads module](https://nodejs.org/api/worker_threads.html) behind the scenes.
+
+## AWS Lambda & Node Parallelizer
+This package can detect the number of vCPU cores allocated to your Lambda function and maximize their utilization. By default, it generates one child process/thread per vCPU core, but this setting can be customized to meet your specific requirements. Alternatively, you can manually specify the number of child processes/threads the library creates, regardless of the number of vCPU cores available.
 
 It uses the Lambda function environment `/tmp` folder to create the required module that runs in the child.
 
-When you call the `parallelizerFunction` method outside of the Lambda handler function, it will reuse the child processes across the different invocations within a Lambda instance, improving performance. Furthermore, if the package detects a disconnection of any of the child processes, it will recreate it automatically without affecting the execution.
+On the Child Process Parallelizer, when you call the `parallelizerFunction` method outside of the Lambda handler function, it will reuse the child processes across the different invocations within a Lambda instance, minimazing the impact of creating child process on every invocation. Furthermore, if the package detects a disconnection of any of the child processes, it will recreate it automatically without affecting the execution.
 
 ## Installation
 To add this package to your dependency list, run:
@@ -24,34 +33,37 @@ To add this package to your dependency list, run:
 npm i node-parallelizer --save
 ```
 ## Usage
-### Child Process Parallelizer
-#### Class instantiation
-`ChildProcess({ tmpPath = '/tmp', maxProcesses = false, processesPerCPU = 1, debug = false })`
+
+<details>
+  <summary>Child Process Parallelizer (<b>I/O-intensive operations or CPU-intensive operations && I/O-intensive operations</b>)</summary>
+
+  #### Class instantiation
+`ChildProcess({ tmpPath = '/tmp', maxParallelization = false, parallelizationPerCPU = 1, debug = false })`
 
 **Parameters**
 - `tmpPath` (String) (Default value: '/tmp'): The path where the module that runs in the child will be created.
-- `maxProcesses` (Number|false) (Default value: false): The maximum number of child processes that will be created. If false, it is based on the CPU cores available.
-- `processesPerCPU` (Number) (Default value: 1): If the `maxProcesses` is set to `false`, this parameter defines the amount of processes per CPU.
+- `maxParallelization` (Number|false) (Default value: false): The maximum number of child processes that will be created. If false, it is based on the CPU cores available.
+- `parallelizationPerCPU` (Number) (Default value: 1): If the `maxParallelization` is set to `false`, this parameter defines the amount of processes per CPU.
 - `debug` (Boolean) (Default value: false): Enables the internal logs for debuggin purposes.
 #### Main methods
 `parallelizerFunction({ filePath, processBatchFunctionName })`
 
 **Parameters**
-- `filePath` (String): The absolute path to the file that contains the function that will be executed with the subset.
-- `processBatchFunctionName` (String): The name of the function that will be executed with the subset.
+- `filePath` (String): The absolute path to the file that contains the function that will be executed in parallel.
+- `processBatchFunctionName` (String): The name of the function that will be executed in parallel.
 
 `runBatch(batch)`
 
 **Parameters**
 - `batch` (Array): The records you want to process in parallel.
 
-**Returns** (Array): The responses of the child processes.
+**Returns** (Array): The child processes' responses.
 #### Using child process parallizer in AWS Lambda.
 In this example, the repository structure looks like this
 ```
 src/
   handler.js
-  my-child-code.js
+  parallel.js
 serverless.yml
 package.json
 ```
@@ -65,7 +77,7 @@ const { ChildProcess } = require("node-parallelizer");
 // Creates a new child process instance.
 const childProcess = new ChildProcess();
 // Creates child processes based on your code.
-childProcess.parallelizerFunction({ filePath: "/var/task/src/my-child-code.js", processBatchFunctionName: 'batchProcessor' });
+childProcess.parallelizerFunction({ filePath: "/var/task/src/parallel.js", processBatchFunctionName: 'batchProcessor' });
 
 module.exports.handler = async(event) => {
   // Run batch in parallel
@@ -75,11 +87,11 @@ module.exports.handler = async(event) => {
 };
 
 ```
-> Make sure to provide the filePath parameter as an absolute path. In this example, we've included '/var/task/' in the path for the child code, as Lambda deploys your code within that folder.
+> Make sure to provide the filePath parameter as an absolute path. In this example, we've included '/var/task/' prefix in the path because Lambda deploys your code within that folder.
 
 The below snippet represents the code you want to run in parallel
 ```javascript
-// my-child-code.js
+// parallel.js
 
 const batchProcessor = ({ batch }) => {
 
@@ -95,6 +107,83 @@ module.exports = { batchProcessor }
 
 ```
 > Verify that the input signature of your function (in this case, batchProcessor) includes batch as a parameter, as it contains the subset of records that a child process will handle.
+  
+</details>
+<details>
+  <summary>Worker Threads Parallelizer (<b>CPU-intensive operations</b>)</summary>
+
+  #### Class instantiation
+`WorkerThreads({ tmpPath = '/tmp', maxParallelization = false, parallelizationPerCPU = 1, debug = false })`
+
+**Parameters**
+- `tmpPath` (String) (Default value: '/tmp'): The path where the module that runs in the thread will be created.
+- `maxParallelization` (Number|false) (Default value: false): The maximum number of threads that will be created. If false, it is based on the CPU cores available.
+- `parallelizationPerCPU` (Number) (Default value: 1): If the `maxParallelization` is set to `false`, this parameter defines the amount of threads per CPU.
+- `debug` (Boolean) (Default value: false): Enables the internal logs for debuggin purposes.
+#### Main methods
+`parallelizerFunction({ filePath, processBatchFunctionName })`
+
+**Parameters**
+- `filePath` (String): The absolute path to the file that contains the function that will be executed in parallel.
+- `processBatchFunctionName` (String): The name of the function that will be executed in parallel.
+
+`runBatch(batch)`
+
+**Parameters**
+- `batch` (Array): The records you want to process in parallel.
+
+**Returns** (Array): The thread's responses.
+#### Using worker threads parallizer in AWS Lambda.
+In this example, the repository structure looks like this
+```
+src/
+  handler.js
+  parallel.js
+serverless.yml
+package.json
+```
+
+The below snippet represents your Lambda handler
+```javascript
+// handler.js
+
+const { WorkerThreads } = require("node-parallelizer");
+
+// Creates a new child process instance.
+const threads = new WorkerThreads();
+// Creates child processes based on your code.
+threads.parallelizerFunction({ filePath: "/var/task/src/parallel.js", processBatchFunctionName: 'batchProcessor' });
+
+module.exports.handler = async(event) => {
+  // Run batch in parallel
+  const responses = await threads.runBatch(event.Records);
+  
+  console.log(responses);
+};
+
+```
+> Make sure to provide the filePath parameter as an absolute path. In this example, we've included '/var/task/' prefix in the path because Lambda deploys your code within that folder.
+
+The below snippet represents the code you want to run in parallel
+```javascript
+// parallel.js
+
+const batchProcessor = ({ batch }) => {
+  
+  //
+  // HERE YOUR CODE
+  //
+
+  return { success: true, count: batch.length }
+}
+
+
+module.exports = { batchProcessor }
+
+```
+> Verify that the input signature of your function (in this case, batchProcessor) includes batch as a parameter, as it contains the subset of records that a child process will handle.
+  
+</details>
 
 ## Contribution
-We welcome contributions to this project. If you are interested in contributing, please feel free to submit a pull request.
+We welcome contributions to this project. If you are interested in contributing, please feel free to submit a pull request.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "node-parallelizer",
-  "version": "1.2.0",
+  "version": "2.0.0",
   "description": "A NodeJS package for running code in parallel. Initially created to provide multiprocessing in an AWS Lambda function, but it can be used in any NodeJS environment.",
   "main": "src/index.js",
   "scripts": {

src/child-process.js

@@ -5,17 +5,17 @@ const os = require("os");
 const fs = require('fs');
 const crypto = require('crypto');
 
-const childFileName = "child-process-child-file.js";
+const childFileName = "child-process-file";
 
 class ChildProcess {
-  constructor({ tmpPath = '/tmp', maxProcesses = false, processesPerCPU = 1, debug = false, generateStats = false, generateChildStats = false } = {}) {
+  constructor({ tmpPath = '/tmp', maxParallelization = false, parallelizationPerCPU = 1, debug = false, generateStats = false, generateChildStats = false } = {}) {
     const uniqueId = crypto.randomBytes(16).toString('hex');
 
     this.tmpPath = `${tmpPath}/${childFileName}-${uniqueId}.js`;
     this.childFile = null;
     this.childProcesses = [];
-    this.maxProcesses = maxProcesses;
-    this.processesPerCPU = processesPerCPU;
+    this.maxParallelization = maxParallelization;
+    this.parallelizationPerCPU = parallelizationPerCPU;
 
     this.processesCount = 1;
     this.debug = debug;
@@ -38,7 +38,7 @@ class ChildProcess {
   }
 
   _createChildProcesses() {
-    this.processesCount = (typeof this.maxProcesses === 'number') ? this.maxProcesses : this._getProcessesCount();
+    this.processesCount = (typeof this.maxParallelization === 'number') ? this.maxParallelization : this._getProcessesCount();
 
     for (let id = 0; id < this.processesCount; id++) {
       this.childProcesses.push(this._createFork());
@@ -164,7 +164,7 @@ class ChildProcess {
 
   _getProcessesCount() {
     const cpuData = os.cpus();
-    return cpuData.length * this.processesPerCPU;
+    return cpuData.length * this.parallelizationPerCPU;
   }
 
   _createFork() {

src/index.js

@@ -1,3 +1,17 @@
 const ChildProcess = require("./child-process");
+const WorkerThreads = require("./worker-thread");
 
-module.exports = { ChildProcess };
+const PARALLELIZER_CHILD = 'child-process';
+const PARALLELIZER_THREADS = 'worker-threads';
+class Parallelizer {
+  constructor(params) {
+    const parallelizer = params.type;
+
+    if(parallelizer === PARALLELIZER_CHILD) {
+      return new ChildProcess(params);
+    }else if(parallelizer === PARALLELIZER_THREADS) {
+      return new WorkerThreads(params);
+    }
+  }
+}
+module.exports = { ChildProcess, WorkerThreads, Parallelizer };