Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
2f0fd4f
Add EditorConfig, make whitespace consistent
jeffbyrnes Jul 9, 2021
bb2731d
Drop separate “dev” Docker Compose config
jeffbyrnes Jul 9, 2021
c503114
Drop unnecessary directives from Dockerfile
jeffbyrnes Jul 9, 2021
a69c5e7
Remove unnecessary port number
jeffbyrnes Jul 9, 2021
e271b1f
Clean up & add to script commands
jeffbyrnes Jul 9, 2021
f686658
Use official Mongo image instead of Bitnami
jeffbyrnes Jul 9, 2021
519efe1
Add example stanza for mongo-express
jeffbyrnes Jul 9, 2021
f1dc733
Run the test every hour by default
jeffbyrnes Jul 9, 2021
50eca8d
Upgrade to forever v4.x & vis-charts v3.x
jeffbyrnes Jul 9, 2021
7300423
npm update
jeffbyrnes Sep 13, 2021
79d2e9b
Upgrade to nodemon v2.x
jeffbyrnes Jan 22, 2022
06e75dd
Upgrade to mongodb v4.x
jeffbyrnes Jan 22, 2022
a623725
Upgrade to eslint v8.x
jeffbyrnes Jan 22, 2022
eb1ba30
Reduce Docker build layers
jeffbyrnes Jan 22, 2022
00ee577
Use ENTRYPOINT w/ npm scripts
jeffbyrnes Jan 22, 2022
4ca897d
Update README (syntax, docs on using forever)
jeffbyrnes Jul 9, 2021
cd43475
Update README (syntax, docs on using forever)
jeffbyrnes Jan 22, 2022
9d0dad2
Consistently name services & NPM package
jeffbyrnes Jan 22, 2022
7d0b0dc
Rename image to make it my own
jeffbyrnes Jan 22, 2022
8dca1c5
Ignore local speedtest binary
jeffbyrnes Jan 22, 2022
d6b0648
Bump to v1.0.0
jeffbyrnes Jan 22, 2022
10a94e0
Add GitHub Action to build & push Docker image
jeffbyrnes Jan 22, 2022
409f2f6
Update to speedtest v1.2.0
jeffbyrnes Sep 5, 2022
b3bee23
Upgrade to building with Node.js v18
jeffbyrnes Sep 5, 2022
7793d46
Switch docker-compose to build images
jeffbyrnes Sep 5, 2022
1ba13f5
Ensure internet-speed-logger-runner has prereqs
jeffbyrnes Sep 5, 2022
b31a5a9
Remove unused var from build script
jeffbyrnes Sep 5, 2022
92754a8
npm update
jeffbyrnes Sep 5, 2022
68a5435
Bump to v2.0.0
jeffbyrnes Sep 5, 2022
962f30b
Update to latest version of GitHub Actions
jeffbyrnes Sep 5, 2022
4fb8239
Convert MongoCLient connection to async/await
jeffbyrnes Sep 7, 2022
653f500
Fix typo referring to config file
jeffbyrnes Sep 7, 2022
0895c1e
Update example docs for running via Docker Compose
jeffbyrnes Sep 7, 2022
fad9bc1
Monitor GitHub Actions to stay up-to-date
jeffbyrnes Sep 7, 2022
d4a3093
Add build & push for latest tag
jeffbyrnes Sep 7, 2022
d554b8d
Add label to Dockerfile
jeffbyrnes Sep 7, 2022
8bb5a5b
Only build versioned images if tagged
jeffbyrnes Sep 7, 2022
2632925
Only build on pushes to main branch
jeffbyrnes Sep 7, 2022
72281b2
Merge pull request #1 from jeffbyrnes/rename-for-myself
jeffbyrnes Sep 7, 2022
a8133bb
Remove inline caching for builds
jeffbyrnes Sep 8, 2022
a6f0697
Bump to v2.0.1
jeffbyrnes Sep 8, 2022
c492f11
Add further documentation around configuration
jeffbyrnes Sep 8, 2022
d9b03bc
Set example docker-compose.yml to use latest image
jeffbyrnes Sep 8, 2022
69884a8
Bump docker/build-push-action from 3 to 4
dependabot[bot] Jan 31, 2023
9be73c5
Merge pull request #2 from jeffbyrnes/dependabot/github_actions/docke…
jeffbyrnes Jun 28, 2023
5c0ac84
npm update
jeffbyrnes Jun 28, 2023
4138f7c
Simplify/standardize image build
jeffbyrnes Jun 28, 2023
c63b0a9
Upgrade to Node.js v20.x
jeffbyrnes Jun 28, 2023
b659be9
Upgrade to ESLint v8.43
jeffbyrnes Jun 28, 2023
13dab43
Satisfy ESLint
jeffbyrnes Jun 28, 2023
48d623f
Bump to v3.0.0
jeffbyrnes Jun 28, 2023
0f71a2a
Bump actions/checkout from 3 to 4
dependabot[bot] Sep 5, 2023
81b867f
Bump docker/login-action from 2 to 3
dependabot[bot] Sep 12, 2023
e522b9a
Bump docker/metadata-action from 4 to 5
dependabot[bot] Sep 12, 2023
75f92c3
Bump docker/build-push-action from 4 to 5
dependabot[bot] Sep 12, 2023
74128ca
Merge pull request #3 from jeffbyrnes/dependabot/github_actions/actio…
jeffbyrnes Jun 3, 2024
0416ee9
Merge remote-tracking branch 'origin/dependabot/github_actions/docker…
jeffbyrnes Jun 3, 2024
389f9c6
Merge remote-tracking branch 'origin/dependabot/github_actions/docker…
jeffbyrnes Jun 3, 2024
75a4d93
Merge remote-tracking branch 'origin/dependabot/github_actions/docker…
jeffbyrnes Jun 3, 2024
f79ab89
npm update
jeffbyrnes Jun 3, 2024
be19765
Bump to v3.0.1
jeffbyrnes Jun 3, 2024
3708cd7
Drop obsolete docker-compose version directive
jeffbyrnes Jun 3, 2024
c0909b8
Upgrade to node:22 base image
jeffbyrnes Jun 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
node_modules
bin
mongo-persistence
mongo-persistence
27 changes: 27 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
; EditorConfig is awesome: http://EditorConfig.org

root = true

; UTF-8 charset
; Unix-style newlines with a newline ending every file
; 2 space indent
; Trim trailing whitespace
[*]
indent_style = space
indent_size = 2
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true

; Markdown
; 4 space indent
; Trailing whitespace is potentially meaningful, leave it
[*.md]
indent_size = 4
trim_trailing_whitespace = false

; Shell scripts, Python, HTML
; 4 space indent
[*.{html,py,sh}]
indent_size = 4
12 changes: 11 additions & 1 deletion .eslintrc.json
Original file line number Diff line number Diff line change
@@ -1 +1,11 @@
{ "extends": "airbnb-base" }
{
"env": {
"commonjs": true,
"es2021": true
},
"extends": "airbnb-base",
"parserOptions": {
"ecmaVersion": "latest"
},
"rules": {}
}
7 changes: 7 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
version: 2
updates:
# Maintain dependencies for GitHub Actions
- package-ecosystem: github-actions
directory: /
schedule:
interval: daily
51 changes: 51 additions & 0 deletions .github/workflows/build-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: Create and publish a Docker image

"on":
push:
branches:
- main

env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}

jobs:
build-and-push-image:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Log in to the Container registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

- name: Build and push versioned Docker image
uses: docker/build-push-action@v5
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
if: startsWith(github.ref, 'refs/tags/')

- name: Build and push latest Docker image
uses: docker/build-push-action@v5
with:
context: .
push: true
tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
labels: ${{ steps.meta.outputs.labels }}
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -61,4 +61,5 @@ typings/
.next

# Persistent volume for mongodb
mongo-persistence
mongo-persistence
/bin/speedtest
20 changes: 7 additions & 13 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,19 +1,13 @@
FROM node:16 as build
WORKDIR /data/
FROM node:22
WORKDIR /home/node/app
ENV NODE_ENV=production
RUN export SPEEDTESTVERSION="1.1.1" && \
LABEL org.opencontainers.image.source=https://github.com/jeffbyrnes/internet-speed-logger
RUN export SPEEDTESTVERSION="1.2.0" && \
export SPEEDTESTARCH="x86_64" && \
export SPEEDTESTPLATFORM="linux" && \
mkdir -p bin && \
curl -Ss -L https://install.speedtest.net/app/cli/ookla-speedtest-$SPEEDTESTVERSION-$SPEEDTESTPLATFORM-$SPEEDTESTARCH.tgz | tar -zx -C /data/bin && \
curl -Ss -L https://install.speedtest.net/app/cli/ookla-speedtest-$SPEEDTESTVERSION-$SPEEDTESTPLATFORM-$SPEEDTESTARCH.tgz | tar -zx -C ./bin && \
chmod +x bin/speedtest
COPY package.json package-lock.json* ./
RUN npm ci
COPY . .

FROM node:16 as app
WORKDIR /data/
COPY --from=build --chown=node:node /data/ .
USER node

CMD ["node", "index.js"]
RUN npm ci
CMD ["npm", "start"]
120 changes: 59 additions & 61 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,93 +8,91 @@ This is a time series based application which continuously monitors your interne

An early version of this service has been running for many years (~2016) and it has been instrumental for tracking internet performance issues.

To bring it online, simply run:
```
git clone https://github.com/brennentsmith/internet-speed-logger.git
cd internet-speed-logger
docker-compose up
```
Wait a couple minutes for MongoDB to initialize, and then go to `http://localhost:3000` in your browser, and away you go!

In case you see errors from mongodb with:<br />
`mongodb | mkdir: cannot create directory '/bitnami/mongodb': Permission denied`<br />
`mongodb exited with code 1`

Try to set the persistent data directory rights:
```
docker-compose down
sudo chown -R 1001 mongo-persistence/
docker-compose up
```

## Updating
To get the latest Docker image, run:
```
docker-compose stop
docker-compose pull
docker-compose up
```

## Components

The requrements for Internet Speed Logger are:
- NodeJS 12-LTS or newer

- Node.js >= v12.0.0
- MongoDB

There are three core components to running Internet Speed Logger:
- Webserver (`/index.js`) - Webserver which delivers static assets and provides API.
- Speedrunner (`/run-speedtest.js`) - Daemon or One Shot process which performs the internet speed test.

- Webserver (`index.js`) - Webserver which delivers static assets and provides API.
- Speedrunner (`run-speedtest.js`) - Daemon or One Shot process which performs the internet speed test.
- MongoDB - "Web Scale" persistence layer. 😜

The Webserver and MongoDB must always be running, however the Speedrunner can be either run as a daemon `/run-speedtest.js daemon` or invoked via cron or SystemD timer as a oneshot process `/run-speedtest.js`. Both the Webserver and Speedrunner share the common config within `/config/default.js`.
The Webserver and MongoDB must always be running, however the Speedrunner can be either run as a daemon `./run-speedtest.js daemon` or invoked via a schedule as a oneshot process `./run-speedtest.js`. Both the Webserver and Speedrunner share the common config within `config/default.json`.

## Configuration

All configuration is held within the `/config/default.js` directory. The following options are available:
Configuration for the logger & runner is in the `config/default.js` file. The following options are available:

| Leaf | Default | Description |
| ------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `webserver.listenPort` | `3000` | Port which the webserver will listen on |
| `webserver.listenHost` | `0.0.0.0` | Host which the webserver will listen on |
| `db.connectionString` | `mongodb://speedtest:speedtest@mongo:27017/speedtest` | Connection string the connection for the backend MongoDB compliant database. See: [Connection String URI Format](https://docs.mongodb.com/manual/reference/connection-string/) |
| `db.collection` | `speedtest` | Collection to use within MongoDB compliant database. |
| `speedtest.commandString` | `bin/speedtest -f json --accept-license` | Raw command to execute to perform speed test. Change this if you want it on a different path or specify a specific server. |
| `speedtest.intervalSec` | `43200` | Interval for which the speedtest will be run. This will be randomly skewed +/- 25% and limited to no less than 1800 (30 minutes) seconds between runs. |

MongoDB bootstraps itself from the `docker-entrypoint-initdb.d/mongo-init.js` file. Meaningful options to change, which should correspond to what you set in `config/default.json` for the Node.js app:

| Key | Default | Description |
| ------ | ----------- | ------------------------------------------------------------------- |
| `user` | `speedtest` | The user that is created in for access to the `speedtest` DB |
| `pwd` | `speedtest` | Password for the specified `user` for access to the `speedtest` DB. |

| Leaf | Default | Description |
| -- | -- | -- |
| `webserver.listenPort` | `3000` | Port which the webserver will listen on |
| `webserver.listenHost` | `0.0.0.0` | Host which the webserver will listen on |
| `db.connectionString` | `mongodb://speedtest:speedtest@mongo:27017/speedtest` | Connection string the connection for the backend MongoDB compliant database. See: [Connection String URI Format](https://docs.mongodb.com/manual/reference/connection-string/) |
| `db.collection` | `speedtest` | Collection to use within MongoDB compliant database. |
| `speedtest.commandString` | `bin/speedtest -f json --accept-license` | Raw command to execute to perform speed test. Change this if you want it on a different path or specify a specific server. |
| `speedtest.intervalSec` | `43200` | Interval for which the speedtest will be run. This will be randomly skewed +/- 25% and limited to no less than 1800 (30 minutes) seconds between runs. |
The `db.connectionString` in `config/default.js` should use the `user` & `pwd` you set in `docker-entrypoint-initdb.d/mongo-init.js`, like so:

```json
"connectionString": "mongodb://<user>:<pwd>@mongo/speedtest",
```

## Running Internet Speed Logger

### Container
A container is published to [Dockerhub](https://hub.docker.com/r/brennentsmith/internet-speed-logger) which contains both the webserver and test daemon.

```
git clone https://github.com/brennentsmith/internet-speed-logger.git
The images are published to GitHub Packages, and it contains both the webserver and test daemon.

```bash
git clone https://github.com/jeffbyrnes/internet-speed-logger.git
cd internet-speed-logger
docker compose up
```
In case you see errors from mongodb with:<br />
`mongodb | mkdir: cannot create directory '/bitnami/mongodb': Permission denied`<br />
`mongodb exited with code 1`

Try to set the persistent data directory rights:
```
docker-compose down
sudo chown -R 1001 mongo-persistence/
docker-compose up
The provided [`docker-compose.yml`](docker-compose.yml) can be modified to suit, or used as an example for your own Docker Compose setup with other services.

You may see some errors upon boot:

```plain
speedlogger-web_1 | MongoNetworkError: failed to connect to server
```

You may see some errors upon boot regarding `speedlogger-web_1 | MongoNetworkError: failed to connect to server` - these are normal as the web service will attempt to create the connection pool before MongoDB is ready. Once MongoDB is ready (~30s), all will work correctly.
these are normal as the web service will attempt to create the connection pool before MongoDB is ready. Once MongoDB is ready (~30s), all will work correctly.

### Using Forever to run locally

### Forever
Install the following:
- NodeJS: https://nodejs.org/en/download/package-manager/
- MongoDB: https://docs.mongodb.com/manual/installation/
- Forever: https://www.npmjs.com/package/forever (optional)

```
git clone https://github.com/brennentsmith/internet-speed-logger.git
- [Node.js](https://nodejs.org/en/download/package-manager/)
- [MongoDB](https://docs.mongodb.com/manual/installation/)

```bash
git clone https://github.com/jeffbyrnes/internet-speed-logger.git
cd internet-speed-logger
<< download latest version of Speedtest-CLI binary to `bin` dir within repo >>
# download latest version of Speedtest-CLI binary to `bin` dir within repo >>
npm ci
forever start index.js
forever start run-speedtest.js daemon
npm run webserver-daemon
npm run speedtest-daemon
```

## Updating

To get the latest Docker image, run:

```bash
docker compose stop
docker compose pull
docker compose up
```
9 changes: 4 additions & 5 deletions build-and-push.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,9 @@

set -ex

PARENT_DIR=$(basename "${PWD%/*}")
IMAGE_NAME="brennentsmith/internet-speed-logger"
IMAGE_NAME="ghcr.io/jeffbyrnes/internet-speed-logger"
TAG=$(git rev-parse --short HEAD)

docker build -t ${IMAGE_NAME}:${TAG} .
docker tag ${IMAGE_NAME}:${TAG} ${IMAGE_NAME}:latest
docker push ${IMAGE_NAME}
docker build -t "${IMAGE_NAME}:${TAG}" .
docker tag "${IMAGE_NAME}:${TAG}" "${IMAGE_NAME}:latest"
docker push "${IMAGE_NAME}"
6 changes: 3 additions & 3 deletions config/default.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@
"listenHost": "0.0.0.0"
},
"db": {
"connectionString": "mongodb://speedtest:speedtest@mongo:27017/speedtest",
"connectionString": "mongodb://speedtest:speedtest@mongo/speedtest",
"collection": "speedtest"
},
"speedtest": {
"commandString": "bin/speedtest -f json --accept-license --accept-gdpr",
"intervalSec": 43200
"intervalSec": 3600
}
}
}
5 changes: 3 additions & 2 deletions db/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,9 @@ const { MongoClient } = require('mongodb');

const mongoUrl = config.get('db.connectionString');

function connect(url) {
return MongoClient.connect(url).then((client) => client.db());
async function connect(url) {
const client = await MongoClient.connect(url);
return client.db();
}

module.exports = async () => {
Expand Down
38 changes: 0 additions & 38 deletions docker-compose-dev.yml

This file was deleted.

Loading