Concurrent URL Status Checker

A command-line tool that checks the HTTP status of many URLs concurrently using a bounded worker pool built from goroutines, channels, and context. Give it a list of URLs; it tells you which ones are UP or DOWN, how long each took, and can export the results as a table, JSON, or CSV.

Built with the Go standard library only — no third-party dependencies, nothing to go get. Clone it and run it.

Key Highlights

• Bounded concurrency using a configurable worker pool
• Goroutines and channels for concurrent URL processing
• context.Context for per-request and overall timeouts
• Graceful cancellation with Ctrl+C / SIGTERM
• Deterministic unit tests using httptest
• JSON and CSV report generation
• CI pipeline using GitHub Actions
• Standard-library-only implementation with no third-party dependencies

Features

• Concurrent checking via a configurable worker pool (default 10 workers)
• Context-based timeouts — a per-request timeout and an overall batch timeout, both cancellable early with Ctrl+C
• Graceful shutdown — interrupting the program marks in-flight/queued checks as DOWN instead of just dying mid-way
• Three output formats — aligned console table, JSON, or CSV
• Sensible exit codes — exits 1 if any URL is DOWN, so it's usable as a step in a CI job or cron-based monitor
• Input file supports comments (#) and URLs without a scheme (github.com is treated as https://github.com)
• Unit tests for every package, using httptest (no real network calls needed to run the test suite)

Project structure

url-status-checker/
├── go.mod                        # module definition (stdlib only)
├── README.md
├── LICENSE
├── Makefile                      # build / run / test / vet shortcuts
├── urls.txt                      # sample input file
├── cmd/
│   └── urlchecker/
│       └── main.go               # CLI entry point: flags, signals, wiring
└── internal/
    ├── checker/
    │   ├── checker.go            # checks ONE url, returns a Result
    │   └── checker_test.go
    ├── worker/
    │   ├── pool.go                # fans a URL list out across N goroutines
    │   └── pool_test.go
    └── report/
        ├── report.go              # table / JSON / CSV formatting
        └── report_test.go

This is a standard Go CLI layout: cmd/ holds the thin entry point, internal/ holds packages that are only meant to be imported from inside this module (the Go compiler enforces that). Each package has one job.

How it works (architecture)

                         urls.txt
                             │
                             ▼
                     main() loads URLs
                             │
                             ▼
                    worker.Pool.Run(ctx, urls)
                             │
              ┌──────────────┼──────────────┐
              ▼               ▼               ▼
        jobs channel ──> jobs channel ──> jobs channel      (buffered, len(urls))
              │               │               │
        ┌─────▼─────┐   ┌─────▼─────┐   ┌─────▼─────┐
        │ worker #1 │   │ worker #2 │   │ worker #N │   <- goroutines
        └─────┬─────┘   └─────┬─────┘   └─────┬─────┘
              │               │               │
              └──────► results channel ◄───────┘
                             │
                             ▼
                report.SortByURL + PrintTable / WriteJSON / WriteCSV

1. main.go reads the URL list and builds a single context.Context that is cancelled on Ctrl+C/SIGTERM or after -overall-timeout.
2. worker.Pool.Run starts -workers goroutines. All URLs are pushed onto a buffered jobs channel, then the channel is closed — this is the standard Go signal for "no more work is coming."
3. Each worker goroutine ranges over jobs until it's empty, wraps the shared context with a per-check timeout (context.WithTimeout), and calls checker.Check.
4. Every result (success or failure) is sent to a results channel. checker.Check never returns a Go error — every failure mode is captured inside the Result struct, so the concurrent fan-in code doesn't need special-case error handling.
5. sync.WaitGroup tracks when all workers have finished; once they have, results is closed and drained into a slice.
6. The results are sorted and printed/exported.

Requirements

• Go 1.22 or later (go.dev/dl) — check with go version
• No external packages, no internet access needed to build

Running it in VS Code

1. Install the official Go extension (golang.go) if VS Code prompts you.

2. Open this folder (File > Open Folder… → select url-status-checker).

3. The first time you open a .go file, VS Code may offer to install Go tools (gopls, dlv, etc.) — click Install All. This is optional; the project builds and runs fine without them.

4. Open a terminal in VS Code (Ctrl+`) and run:

   go run ./cmd/urlchecker

   That's it — it reads urls.txt from the project root by default.

Usage

# Basic run (uses urls.txt, 10 workers, table output)
go run ./cmd/urlchecker

# Custom input file and worker count
go run ./cmd/urlchecker -input=my-sites.txt -workers=20

# Shorter per-request timeout, export as JSON
go run ./cmd/urlchecker -timeout=2s -format=json -out=results.json

# Export as CSV
go run ./cmd/urlchecker -format=csv -out=results.csv

# Build a standalone binary
go build -o bin/urlchecker ./cmd/urlchecker
./bin/urlchecker -input=urls.txt

Flags

Flag	Default	Meaning
-input	urls.txt	Path to a text file, one URL per line
-workers	10	Number of concurrent goroutines
-timeout	5s	Timeout applied to each individual URL check
-overall-timeout	30s	Timeout for the entire batch
-format	table	table, json, or csv
-out	(none)	Output file path (used for json/csv)

Example output

Checking 6 URL(s) with 10 worker(s) (per-check timeout 5s, overall timeout 30s)...

URL                                                STATUS  CODE  LATENCY  DETAIL
---                                                ------  ----  -------  ------
https://github.com                                 UP      200   112ms    -
https://go.dev                                     UP      200   98ms     -
https://pypi.org                                    UP      200   145ms    -
https://this-domain-definitely-does-not-exist...    DOWN    0     23ms     dial tcp: lookup ...: no such host
https://www.google.com                              UP      200   87ms     -
https://www.wikipedia.org                           UP      200   134ms    -

6 checked | 5 UP | 1 DOWN | took 152ms

Note that the total time (152ms) is close to the slowest single request, not the sum of all six — that's the concurrency working.

Running the tests

go test ./...            # run all tests
go test ./... -v         # verbose output
make cover                # coverage report (or: go test ./... -coverprofile=coverage.out)

Tests use net/http/httptest to spin up local, in-process HTTP servers, so the test suite needs no real network access and is fully deterministic (including the timeout and "host not found" tests).

Design decisions worth knowing (useful for interviews / code review)

• Why a worker pool instead of one goroutine per URL? Unbounded goroutines (for _, u := range urls { go check(u) }) can open thousands of simultaneous connections and exhaust file descriptors / rate-limit you against the target servers. A worker pool caps concurrency to a known, configurable number.
• Why does Check never return an error? Because every failure mode (bad input, DNS failure, timeout, 5xx) is a valid, expected outcome for a status checker — a DOWN site isn't a bug in this program. Encoding that into the Result itself means the pool and report code can treat every check uniformly.
• Why context.WithTimeout per check, wrapping a parent context that has its own overall timeout? So a single slow URL can't stall the whole batch (-timeout), while the batch as a whole also has a hard ceiling (-overall-timeout), and both can be cut short instantly by Ctrl+C via signal.NotifyContext.
• Why a named return (func Check(...) (result Result)) with a defer? So Latency/LatencyMS are always set on every exit path — including early error returns — without repeating that logic at each return.
• Why exclude Latency from JSON but add LatencyMS? time.Duration marshals to raw nanoseconds by default, which isn't what a consumer of the JSON output expects; LatencyMS is explicit and human-friendly.

Possible extensions

• Retry failed checks N times with backoff before marking DOWN
• Slack/webhook notification when a previously-UP site goes DOWN
• -interval flag to keep re-checking on a schedule instead of running once
• Read URLs from stdin as an alternative to -input

License

MIT — see LICENSE.