Skip to content

Commit 86e59af

Browse files
verset229-engverset229-eng
committed
Google Maps Scraper: nearest-first distance ranking, multi-page UI, REST API, animated landing
- Nearest-first ranking with m/km distance (haversine) + automatic de-duplication - Multi-page UI: animated landing page, features, Manage dashboard, interactive map filter - API-only deploy mode (API_ONLY=1) to run headless without the UI - Removed sponsor/star popups; SEO-optimized README with screenshots and example output - Proxies are parameter-only (flag/form/API); no credentials committed
0 parents  commit 86e59af

250 files changed

Lines changed: 50564 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/FUNDING.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
github: gosom

‎.github/workflows/build.yml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
name: build
2+
3+
permissions: {}
4+
5+
on:
6+
push:
7+
branches: [main]
8+
pull_request:
9+
branches: [main]
10+
11+
jobs:
12+
run:
13+
name: Build
14+
runs-on: ubuntu-latest
15+
timeout-minutes: 7
16+
strategy:
17+
fail-fast: true
18+
matrix:
19+
go: ['1.26.5']
20+
21+
steps:
22+
- name: Check out code
23+
uses: actions/checkout@v4
24+
- name: Install Go
25+
uses: actions/setup-go@v4
26+
with:
27+
go-version: ${{ matrix.go }}
28+
check-latest: true
29+
- name: Go Format
30+
run: gofmt -s -w . && git diff --exit-code
31+
- name: Lint
32+
run: make lint
33+
- name: Go Vet
34+
run: go vet ./...
35+
- name: Go Tidy
36+
run: go mod tidy && git diff --exit-code
37+
- name: Go Mod
38+
run: go mod download
39+
- name: Go Mod Verify
40+
run: go mod verify
41+
- name: Go Vulnerability Check
42+
run: make vuln
43+
- name: Go Build
44+
run: go build -o /dev/null ./...

‎.github/workflows/publish.yml‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
name: publish
2+
3+
permissions:
4+
packages: write
5+
6+
on:
7+
release:
8+
types: [published]
9+
10+
jobs:
11+
push_playwright_image:
12+
name: Push Playwright Docker image to Docker Hub
13+
runs-on: ubuntu-latest
14+
steps:
15+
- name: Check out the repo
16+
uses: actions/checkout@v4
17+
18+
- name: Log in to Docker Hub
19+
uses: docker/login-action@v3
20+
with:
21+
username: ${{ secrets.DOCKER_USERNAME }}
22+
password: ${{ secrets.DOCKER_PASSWORD }}
23+
24+
- name: Extract metadata (tags, labels) for Docker
25+
id: meta
26+
uses: docker/metadata-action@v5
27+
with:
28+
images: gosom/google-maps-scraper
29+
30+
- name: Build and push Docker image
31+
uses: docker/build-push-action@v6
32+
with:
33+
context: .
34+
file: ./Dockerfile
35+
push: true
36+
tags: ${{ steps.meta.outputs.tags }}
37+
labels: ${{ steps.meta.outputs.labels }}
38+
no-cache: true
39+
40+
push_saas_image:
41+
name: Push SaaS Docker image to GHCR
42+
runs-on: ubuntu-latest
43+
steps:
44+
- name: Check out the repo
45+
uses: actions/checkout@v4
46+
47+
- name: Log in to GHCR
48+
uses: docker/login-action@v3
49+
with:
50+
registry: ghcr.io
51+
username: ${{ github.actor }}
52+
password: ${{ secrets.GITHUB_TOKEN }}
53+
54+
- name: Extract metadata (tags, labels) for Docker
55+
id: meta
56+
uses: docker/metadata-action@v5
57+
with:
58+
images: ghcr.io/gosom/google-maps-scraper-saas
59+
60+
- name: Build and push Docker image
61+
uses: docker/build-push-action@v6
62+
with:
63+
context: .
64+
file: ./Dockerfile.saas
65+
push: true
66+
tags: ${{ steps.meta.outputs.tags }}
67+
labels: ${{ steps.meta.outputs.labels }}
68+
no-cache: true

‎.gitignore‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
cache/
2+
vendor
3+
bin
4+
webdata/
5+
# local test credentials / private keys — never commit
6+
local/test_ssh_key
7+
local/test_ssh_key.pub
8+
*.log
9+
# scraper outputs / local data
10+
*.csv
11+
!testdata/**/*.csv
12+
!gmaps/testdata/**/*.csv
13+
examples/examples-api/map-outputs
14+
examples/examples-api/**/__pycache__/
15+
examples/examples-api/**/*.py[cod]
16+
gmaps-output

‎.golangci.yaml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
run:
2+
issues-exit-code: 1
3+
timeout: 3m
4+
linters-settings:
5+
errcheck:
6+
check-type-assertions: true
7+
goconst:
8+
min-len: 2
9+
min-occurrences: 3
10+
gci:
11+
sections:
12+
- prefix(github.com/gosom/ledger)
13+
custom-order: true
14+
gocritic:
15+
enabled-tags:
16+
- diagnostic
17+
- experimental
18+
- opinionated
19+
- performance
20+
- style
21+
govet:
22+
shadow: true
23+
nolintlint:
24+
require-explanation: true
25+
require-specific: true
26+
linters:
27+
disable-all: true
28+
enable:
29+
- bodyclose
30+
- dogsled
31+
- dupl
32+
- errcheck
33+
- copyloopvar
34+
- exhaustive
35+
- goconst
36+
- gocritic
37+
- gofmt
38+
- goimports
39+
- gocyclo
40+
- gosec
41+
- gosimple
42+
- govet
43+
- ineffassign
44+
- misspell
45+
- nolintlint
46+
- nakedret
47+
- prealloc
48+
- predeclared
49+
- revive
50+
- staticcheck
51+
- stylecheck
52+
- testpackage
53+
- thelper
54+
- tparallel
55+
- typecheck
56+
- unconvert
57+
- unparam
58+
- whitespace
59+
- wsl
60+

‎.pre-commit-config.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
repos:
2+
- repo: local
3+
hooks:
4+
- id: gobuild
5+
name: gobuild
6+
entry: go build -v -o /dev/null ./...
7+
language: golang
8+
types: [go]
9+
require_serial: true
10+
pass_filenames: false
11+
- id: golangci-lint
12+
name: golangci-lint
13+
entry: make lint
14+
language: golang
15+
types: [go]
16+
require_serial: true
17+
pass_filenames: false
18+
- id: gotest
19+
name: gotest
20+
entry: go test -v -race ./...
21+
language: golang
22+
types: [go]
23+
require_serial: true
24+
pass_filenames: false

‎AGENTS.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Development Guide for Agents
2+
3+
## Build/Test Commands
4+
- `make test` - Run all unit tests with race detection
5+
- `make test-cover` - Run tests with coverage statistics
6+
- `make lint` - Run golangci-lint with project configuration
7+
- `make vet` - Run go vet static analysis
8+
- `make format` - Format code with gofmt
9+
- `go test ./path/to/package` - Run tests for a specific package
10+
11+
## Code Style Guidelines
12+
- Use `gofmt` for formatting (spaces, not tabs)
13+
- Import order: standard library, third-party, local packages (prefix: github.com/gosom/google-maps-scraper)
14+
- Use descriptive variable names (e.g., `entry`, `cfg`, `ctx`)
15+
- Error handling: return errors, use `fmt.Errorf` with wrapping (`%w`)
16+
- Use struct tags for JSON marshaling: `json:"field_name"`
17+
- Constants use CamelCase (e.g., `RunModeFile`)
18+
- Interface names end with -er suffix (e.g., `Runner`, `S3Uploader`)
19+
- Use context.Context as first parameter in functions
20+
- Prefer early returns to reduce nesting
21+
- Use meaningful package names that reflect their purpose
22+
- Add godoc comments for exported types and functions
23+
- Use `nolint` comments sparingly with explanations
24+
- Avoid magic numbers, use named constants or comment them

‎Dockerfile‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Build stage for Playwright dependencies
2+
FROM ubuntu:20.04 AS playwright-deps
3+
ENV PLAYWRIGHT_BROWSERS_PATH=/opt/browsers
4+
ENV PLAYWRIGHT_DRIVER_PATH=/opt/ms-playwright-go
5+
ARG TARGETARCH
6+
ARG PLAYWRIGHT_GO_VERSION=v0.6100.0
7+
8+
RUN export PATH=$PATH:/usr/local/go/bin:/root/go/bin \
9+
&& apt-get update \
10+
&& apt-get install -y --no-install-recommends ca-certificates curl wget \
11+
&& if [ "$TARGETARCH" = "arm64" ]; then \
12+
GO_ARCH="arm64"; \
13+
else \
14+
GO_ARCH="amd64"; \
15+
fi \
16+
&& wget -q "https://go.dev/dl/go1.26.5.linux-${GO_ARCH}.tar.gz" \
17+
&& tar -C /usr/local -xzf "go1.26.5.linux-${GO_ARCH}.tar.gz" \
18+
&& rm "go1.26.5.linux-${GO_ARCH}.tar.gz" \
19+
&& apt-get clean \
20+
&& rm -rf /var/lib/apt/lists/* \
21+
&& go install github.com/mxschmitt/playwright-go/cmd/playwright@${PLAYWRIGHT_GO_VERSION} \
22+
&& mkdir -p /opt/browsers \
23+
&& playwright install chromium --with-deps
24+
25+
# Build stage
26+
FROM golang:1.26.5-trixie AS builder
27+
WORKDIR /app
28+
COPY go.mod go.sum ./
29+
RUN go mod download
30+
COPY . .
31+
RUN CGO_ENABLED=0 go build -ldflags="-w -s" -o /usr/bin/google-maps-scraper
32+
33+
# Final stage
34+
FROM debian:trixie-slim
35+
ENV PLAYWRIGHT_BROWSERS_PATH=/opt/browsers
36+
ENV PLAYWRIGHT_DRIVER_PATH=/opt/ms-playwright-go
37+
38+
# Install only the necessary dependencies in a single layer
39+
RUN apt-get update && apt-get install -y --no-install-recommends \
40+
ca-certificates \
41+
libnss3 \
42+
libnspr4 \
43+
libatk1.0-0 \
44+
libatk-bridge2.0-0 \
45+
libcups2 \
46+
libdrm2 \
47+
libdbus-1-3 \
48+
libxkbcommon0 \
49+
libatspi2.0-0 \
50+
libx11-6 \
51+
libxcomposite1 \
52+
libxdamage1 \
53+
libxext6 \
54+
libxfixes3 \
55+
libxrandr2 \
56+
libgbm1 \
57+
libpango-1.0-0 \
58+
libcairo2 \
59+
libasound2 \
60+
&& apt-get clean \
61+
&& rm -rf /var/lib/apt/lists/*
62+
63+
COPY --from=playwright-deps /opt/browsers /opt/browsers
64+
COPY --from=playwright-deps /opt/ms-playwright-go /opt/ms-playwright-go
65+
66+
RUN chmod -R 755 /opt/browsers \
67+
&& chmod -R 755 /opt/ms-playwright-go
68+
69+
COPY --from=builder /usr/bin/google-maps-scraper /usr/bin/
70+
71+
ENTRYPOINT ["google-maps-scraper"]

‎Dockerfile.saas‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Build stage
2+
FROM golang:1.26.5-alpine AS builder
3+
4+
WORKDIR /app
5+
6+
# Install build dependencies
7+
RUN apk add --no-cache git ca-certificates
8+
9+
# Copy go mod files
10+
COPY go.mod go.sum ./
11+
RUN go mod download
12+
13+
# Copy source code
14+
COPY . .
15+
16+
# Build the gmapssaas binary (stripped and optimized)
17+
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -o /gmapssaas ./cmd/gmapssaas
18+
19+
# Final stage - Debian slim with Playwright Chromium dependencies
20+
FROM debian:bookworm-slim
21+
LABEL org.opencontainers.image.source=https://github.com/gosom/google-maps-scraper
22+
23+
WORKDIR /app
24+
25+
# Install Playwright dependencies for Chromium only + cleanup in same layer
26+
# These are the system libraries that Playwright's Chromium needs to run
27+
# The browser itself is downloaded at runtime by playwright-go
28+
RUN apt-get update && apt-get install -y --no-install-recommends \
29+
ca-certificates \
30+
curl \
31+
libnss3 \
32+
libnspr4 \
33+
libatk1.0-0 \
34+
libatk-bridge2.0-0 \
35+
libcups2 \
36+
libdrm2 \
37+
libdbus-1-3 \
38+
libxkbcommon0 \
39+
libxcomposite1 \
40+
libxdamage1 \
41+
libxfixes3 \
42+
libxrandr2 \
43+
libgbm1 \
44+
libasound2 \
45+
libpango-1.0-0 \
46+
libcairo2 \
47+
fonts-liberation \
48+
&& rm -rf /var/lib/apt/lists/* /var/cache/apt/* \
49+
&& rm -rf /usr/share/doc /usr/share/man /usr/share/locale
50+
51+
# Copy binary from builder
52+
COPY --from=builder /gmapssaas /app/gmapssaas
53+
54+
# Copy migrations (needed for server mode)
55+
COPY --from=builder /app/migrations /app/migrations
56+
57+
EXPOSE 8080
58+
59+
# Default to serve mode, can be overridden with "worker" argument
60+
ENTRYPOINT ["/app/gmapssaas"]
61+
CMD ["serve"]

0 commit comments

Comments
 (0)