Skip to content

WalkThrough.wiki

Latest commit

 

History

History
404 lines (290 loc) · 24.2 KB

WalkThrough.wiki

File metadata and controls

404 lines (290 loc) · 24.2 KB

Table of Contents

ipfs-gate operator walkthrough

This is a noob-friendly guide to standing up your own ipfs-gate server. It assumes you can SSH into a Linux box and follow recipes — no programming background needed.

> Status: v0.1 first build. Working software but expect rough edges. Bugs found during real deployment get folded back into the Common Problems section below.

> Reference OS: Ubuntu Server 24.04 LTS (Noble). The recipes are tested on this OS specifically. Other distros (Ubuntu 22.04, Debian 12, etc.) will likely work but might have subtle differences. For reproducibility, stick to Ubuntu 24.04 LTS on a fresh VPS.

> Reference user: This guide assumes you SSH in as root on a fresh VPS (the default for most providers — Vultr, Hetzner, Linode, DigitalOcean, etc.). If you operate as a non-root user with sudo, swap sudo in where shown.

What you are setting up

A server that:

  • Accepts file uploads from web browsers
  • Charges users in CNOOBS (a Hive-Engine token) for hosting
  • Stores the files on IPFS (the InterPlanetary File System — a peer-to-peer storage network)
  • Hands users back a "CID" (a fingerprint of their file) that they share with whoever they want to give the file to
  • Auto-deletes files after 7 days (configurable)
You don't need to know how IPFS works internally. ipfs-gate sits in front of it, charges money, and unpins old files automatically.

What you need before starting

  1. A fresh Ubuntu 24.04 LTS VPS with at least:
    • 2 CPU cores, 2GB RAM, 20GB disk (more disk = more pinning capacity)
    • A public IP address and the ability to open ports 80 + 443
  2. A domain name you control (e.g. ipfs.v4call.com). It must point at your VPS's IP via an A record. Set this up at your DNS provider before starting — Let's Encrypt needs the domain to resolve.
  3. A dedicated Hive account for ipfs-gate (separate from any v4call account). Create one if needed. Fund it with about 2 HBD for refunds and Resource Credits.
  4. The active private key for that Hive account (NOT the owner key, NOT the posting key — the active key). You will paste it into the server's .env file. Keep it secret.
If you don't have CNOOBS to test with, ask CompleteNoobs in v4call or contact via the Hive blockchain.

This guide installs everything on one VPS (kubo, ipfs-gate, nginx). For v0.1 scale this is fine; splitting onto separate boxes can come later.

Step 1: Install Docker on Ubuntu 24.04

SSH into the VPS as root: 

ssh root@your-vps-ip

Update + install Docker (the official one-liner installer): 

apt update && apt upgrade -y
curl -fsSL https://get.docker.com | sh

Verify: 

docker --version
docker compose version

You should see Docker 27.x or newer and Docker Compose v2.x or newer.

(If you'd rather operate as a non-root user, also run usermod -aG docker your-user then log out and back in. The rest of this guide uses root; adjust paths under /root/... to /home/your-user/... as needed.)

Step 2: Clone ipfs-gate + prepare data directories

We'll deploy from /opt/IPFS-Gate (a standard location for operator-managed services): 

cd /opt
git clone https://github.com/CompleteNoobs/IPFS-Gate.git
cd /opt/IPFS-Gate

Critical permission step. The ipfs-gate container runs as UID 1000 (the node user inside the container). Because we're running docker as root, the host data directories get created owned by root, and the container can't write to them. Pre-create them and chown: 

mkdir -p data/kubo data/ipfs-gate data/letsencrypt data/letsencrypt-webroot
chown -R 1000:1000 data/

Skipping this leads to SQLITE_CANTOPEN errors in the ipfs-gate logs (see Common Problems #2).

Step 3: Configure .env

cp .env.example .env
nano .env

Fill in at minimum (the only three required fields):

  • IPFS_GATE_HIVE_ACCOUNT — your dedicated Hive account name (e.g. v4call-ipfs)
  • IPFS_GATE_ACTIVE_KEY — its active private key (starts with 5...)
  • ADMIN_KEY — pick a long random string. Generate one with openssl rand -hex 32 from another terminal.
Sensible defaults to leave alone:

  • BIND_HOST=0.0.0.0 — required for docker; do not change to 127.0.0.1
  • DISK_LIMIT_GB=5 — increase as your VPS allows
  • DEFAULT_TTL_DAYS=7 — accepts fractions for testing (e.g. 0.001 ≈ 1.4 min)
  • MAX_FILE_SIZE_MB=10 — per-upload cap
  • KUBO_DHT_MODE=none — private hosting; CID only servable from your own gateway. Set to client for public IPFS network availability
Save and exit (Ctrl-X, Y, Enter in nano).

Step 4: Edit nginx config for your domain

nano nginx/ipfs-gate.conf

Replace every instance of ipfs.example.com with your actual domain (e.g. ipfs.v4call.com).

Leave the HTTPS server block commented out for now — we'll enable it after the cert is in place in Step 6.

Step 5: Boot HTTP-only and get an SSL cert

Bring up the stack with HTTP only: 

docker compose up -d

Wait ~30 seconds for Kubo's first-boot identity generation. Then verify: 

docker compose ps
docker compose logs ipfs-gate | tail -20

You should see lines like: 

[quota] schema_version = 1
[server] kubo backend OK, version 0.41.0, used <N> bytes
[sweeper] starting, interval = 60000ms
ipfs-gate v0.1 listening on 0.0.0.0:3001
  operator account: @your-hive-account
  payment: 1 CNOOBS per upload (≤10MB, 7-day TTL)
  CORS origin: *

If you see listening on 127.0.0.1:3001 instead, your .env has the old BIND_HOST. Fix it (Common Problems #6) before proceeding — the cert step needs HTTP serving to work.

Now request the cert from Let's Encrypt: 

# Replace ipfs.your-domain.com with your actual domain in both places below.
docker run --rm -it \
  -v "$PWD/data/letsencrypt:/etc/letsencrypt" \
  -v "$PWD/data/letsencrypt-webroot:/var/www/certbot" \
  --entrypoint certbot \
  certbot/certbot \
  certonly --webroot --webroot-path=/var/www/certbot \
  -d ipfs.your-domain.com \
  --email you@your-domain.com --agree-tos --no-eff-email

On success: Successfully received certificate. The cert files land in data/letsencrypt/live/ipfs.your-domain.com/.

If this fails with a DNS error, double-check that your domain's A record actually points at this VPS's IP (try dig ipfs.your-domain.com from a different machine).

Step 6: Enable HTTPS

nano nginx/ipfs-gate.conf

Uncomment the entire server { listen 443 ssl; ... } block. Make sure every ipfs.example.com in that block is also replaced with your domain (it should already match what you set in Step 4).

Reload nginx: 

docker compose restart nginx
docker compose logs nginx | tail -10

Step 7: Verify it's working

From your laptop (NOT the VPS): 

curl https://ipfs.your-domain.com/

Expected JSON response: 

{"service":"ipfs-gate","version":"0.1.0-dev","operator":"your-hive-account","payment":{"currency":"CNOOBS","amount":"1","max_size_mb":10,"ttl_days":7}}

Test the admin endpoint too (substitute your real ADMIN_KEY): 

curl -H 'Authorization: Bearer YOUR_ADMIN_KEY' https://ipfs.your-domain.com/admin/stats

You should see a JSON response with disk, pin, payment, moderation, and kubo stats.

Note: an HTTP request (http://...) returns a 301 redirect to HTTPS. That's expected — nginx forces HTTPS for all real traffic. Use curl -L to follow the redirect or hit https:// directly.

Step 8: Schedule cert renewal

Let's Encrypt certs expire after 90 days. Add a cron job for root: 

crontab -e

Add this single line (adjust nothing — uses /opt/IPFS-Gate from this walkthrough): 

0 3 * * 1 cd /opt/IPFS-Gate && docker run --rm -v "$PWD/data/letsencrypt:/etc/letsencrypt" -v "$PWD/data/letsencrypt-webroot:/var/www/certbot" --entrypoint certbot certbot/certbot renew --quiet && docker compose restart nginx

Runs every Monday at 03:00 UTC. Certbot only actually renews when within 30 days of expiry — earlier runs are no-ops.

Updating to a new version

IPFS_GATE_DIR=/opt/IPFS-Gate
cd "$IPFS_GATE_DIR"

# 1. Back up your config + data BEFORE pulling
cp .env .env.bk-$(date +%Y%m%d)
cp -r data data.bk-$(date +%Y%m%d)
cp nginx/ipfs-gate.conf nginx/ipfs-gate.conf.bk-$(date +%Y%m%d)

# 2. Pull latest code (this WILL clobber .env and nginx/ipfs-gate.conf if you've edited them)
git fetch --all && git reset --hard origin/main

# 3. Restore your operator-specific files
cp .env.bk-$(date +%Y%m%d) .env
cp nginx/ipfs-gate.conf.bk-$(date +%Y%m%d) nginx/ipfs-gate.conf

# 4. Re-apply data ownership (in case new directories appeared)
chown -R 1000:1000 data/

# 5. Rebuild and restart (env-change or code-change BOTH need down/up, not restart)
docker compose down
docker compose build --no-cache
docker compose up -d

# 6. Watch logs to confirm clean start
docker compose logs -f ipfs-gate

Always back up .env, nginx/ipfs-gate.conf, and the data/ directory before any update. A git reset --hard will wipe any operator-edited files in the repo.

Admin tasks

All admin endpoints need Authorization: Bearer YOUR_ADMIN_KEY from your .env.

See server stats

curl -H 'Authorization: Bearer YOUR_ADMIN_KEY' https://ipfs.your-domain.com/admin/stats

Shows disk usage, pin counts, payment counts, recent moderation actions, Kubo health.

List a user's uploads

curl -H 'Authorization: Bearer YOUR_ADMIN_KEY' \
  "https://ipfs.your-domain.com/admin/uploads?account=guest33"

Ban an account

curl -X POST -H 'Authorization: Bearer YOUR_ADMIN_KEY' -H 'Content-Type: application/json' \
  -d '{"hive_account":"badguy","reason":"Spam","refund_policy":"none"}' \
  https://ipfs.your-domain.com/admin/ban

refund_policy can be "none" (default — banned user forfeits unused TTL) or "prorata" (calculate per-pin remaining-time refund; v0.1 records this intent but does NOT auto-issue the on-chain refund. The operator must transfer manually and then /admin/log-refund).

Unban an account

curl -X POST -H 'Authorization: Bearer YOUR_ADMIN_KEY' -H 'Content-Type: application/json' \
  -d '{"hive_account":"badguy"}' \
  https://ipfs.your-domain.com/admin/unban

Note: unban does NOT restore content that was unpinned. The user must re-upload + re-pay.

Take down a specific CID

curl -X POST -H 'Authorization: Bearer YOUR_ADMIN_KEY' -H 'Content-Type: application/json' \
  -d '{"cid":"bafkreig...","reason":"DMCA #12345"}' \
  https://ipfs.your-domain.com/admin/takedown

Adds the CID to the blocklist (can never be re-uploaded) and unpins from Kubo immediately.

Export the takedown list (for backup or peer sharing)

curl -H 'Authorization: Bearer YOUR_ADMIN_KEY' https://ipfs.your-domain.com/admin/takedowns > takedowns.json

Import a takedown list

curl -X POST -H 'Authorization: Bearer YOUR_ADMIN_KEY' -H 'Content-Type: application/json' \
  -d '{"takedowns":[{"cid":"bafk1...","reason":"From peer X"}]}' \
  https://ipfs.your-domain.com/admin/takedowns/import

View the audit log

curl -H 'Authorization: Bearer YOUR_ADMIN_KEY' \
  "https://ipfs.your-domain.com/admin/moderation/log?limit=50"

Append-only record of every moderation action with admin attribution and timestamps.

Orphan payments (manual refund needed)

If a user paid in wrong currency, wrong amount, or the Hive-Engine sidechain hadn't confirmed the transfer at upload time, the payment ends up in the orphan list: 

curl -H 'Authorization: Bearer YOUR_ADMIN_KEY' https://ipfs.your-domain.com/admin/orphan-payments

Operator reviews, decides whether to manually refund (out-of-band Hive transfer), and logs the action: 

# After manually transferring back to the user on Hive:
curl -X POST -H 'Authorization: Bearer YOUR_ADMIN_KEY' -H 'Content-Type: application/json' \
  -d '{"payment_id":88,"refund_tx_id":"ef56gh78...","reason":"Wrong amount"}' \
  https://ipfs.your-domain.com/admin/log-refund

Federation (deferred to v0.3+)

ipfs-gate is currently standalone. Each operator runs their own server with their own ban list, their own pricing, their own disk allocation. v0.3+ will add optional Nostr-based discovery and opt-in cross-operator banlist sharing — same architectural pattern as v4call's Nostr federation work.

Common problems

(This section grows as bugs are found and fixed during development. Ordered by first-boot likelihood.)

  1. Q: I ran docker compose up and Kubo doesn't respond for the first ~30 seconds. Is it broken?
    A: No. Kubo does a one-time identity generation on first boot. Takes 10–30s on a small VPS. Watch docker compose logs kubo to see when it's ready. The healthcheck in docker-compose.yml gates ipfs-gate startup on Kubo being healthy.
  2. Q: My ipfs-gate container won't start. I see one of:
    • EACCES: permission denied, mkdir '/app/data'
    • SqliteError: unable to open database file with code: 'SQLITE_CANTOPEN'
    • Container in restart loop with errors mentioning /app/data/ipfs-gate.db
    A: Same root cause: the container runs as UID 1000 (node user), but if you ran docker compose up as root, Docker created the host ./data/ directories owned by root. The container can't write there. Fix:
    chown -R 1000:1000 ./data/
docker compose down && docker compose up -d docker compose logs ipfs-gate | tail -20
  1. Confirmed during the v0.1 first-VPS test (2026-05-24). The Dockerfile correctly chowns /app/data at build, but the host volume mount overlays that with the host's ownership, which wins. Step 2 of this walkthrough does the chown proactively to avoid hitting this.
  2. Q: Boot logs say FATAL: IPFS_GATE_HIVE_ACCOUNT not set. Refusing to start.
    A: Your .env is missing or doesn't have IPFS_GATE_HIVE_ACCOUNT. Copy .env.example to .env and fill in the values (Step 3).
  3. Q: I changed something in .env and ran docker compose restart ipfs-gate, but the change didn't take effect (e.g. logs still show the old BIND_HOST or old TTL).
    A: docker compose restart reuses the existing container (and the env vars baked in at create-time). To pick up .env changes you must recreate the container:
    # Or force-recreate just one service:
docker compose up -d --force-recreate ipfs-gate
  1. This is parallel to v4call's "docker compose down required before rebuilding" gotcha — different cause (env change vs code change) but same surprise. Confirmed during v0.1 first-VPS test (2026-05-24).
  2. Q: My response JSON shows "ttl_days":0 even though I set DEFAULT_TTL_DAYS=0.001 for testing fast sweeper expiry. Or pins expire immediately.
    A: Pre-v0.1.1 code used parseInt which truncated 0.001 to 0. Fixed in v0.1.1: now uses parseFloat, so fractional days work. If you're on an old binary, git pull + rebuild. Reference: 0.001 day ≈ 1.44 minutes — handy for testing the sweeper.
  3. Q: Nginx returns 502 Bad Gateway when I curl my domain. The ipfs-gate container is running fine and logs show it's listening.
    A: Your .env has BIND_HOST=127.0.0.1. That binds ipfs-gate to the container's loopback only, which is unreachable from the nginx container. For Docker deployment, set BIND_HOST=0.0.0.0. The container is still isolated by the docker network — only nginx can reach it from inside the stack; the public only sees ports 80/443 on nginx. The shipped .env.example defaults to 0.0.0.0 — but pre-existing .env files from earlier installs still need the manual edit, then a full docker compose down && up -d (Common Problems #4).
  4. Q: Uploads succeed but recipients see "file not found" when fetching from a public IPFS gateway (e.g. ipfs.io).
    A: Check KUBO_DHT_MODE in your .env. If set to none (the v0.1 default), only your own ipfs-gate's gateway serves the file. Other public IPFS gateways won't know about it. This is intentional for privacy; switch to client if you want public-IPFS-network availability. Recipients fetching via https://ipfs.your-domain.com/ipfs/&lt;cid&gt; always work regardless of this setting.
  5. Q: Payment verification fails with memo mismatch.
    A: The browser must send the exact memo string returned by /reserve. Format is ipfs-gate:upload:&lt;16-hex-id&gt;. If the client edited it, this will fail. The reservation_id in the memo also must match the URL parameter.
  6. Q: Upload returns "payload.from undefined doesn't match sender &lt;username&gt;" even though the CNOOBS transfer landed on-chain.
    A: Pre-v0.1.2 had a redundant payload.from check in hive-verify.js that always failed: Hive-Engine's tokens/transfer contractPayload has no from field (the sender is implicit in the wrapping custom_json's required_auths, which extractTokenTransferOp already validates). Fixed in v0.1.2 by removing the check. If you're on an old binary, git pull + docker compose down && docker compose up -d --build. Confirmed during v0.1 first-client integration test (2026-05-25).
  7. Q: Uploads "succeed" with a CID but the gateway returns 404 / "CID not pinned here" a few minutes later (or even within seconds during testing).
    A: Almost always DEFAULT_TTL_DAYS still set to a sweeper-test value like 0.001. That's ~1.44 minutes; the sweeper unpins the file very fast. Check the live config:
    curl -s https://ipfs.your-domain.com/ | grep ttl_days
    If it's not your production value (default 7), edit .env, then docker compose down && docker compose up -d (env changes need recreate per Common Problems #4). Confirmed during v0.1 first-client integration test (2026-05-25) — sweeper-test TTL from earlier session lingered.
  8. Q: Upload returned 2xx with a CID even though my account didn't have enough CNOOBS to pay (so the Hive-Engine sidechain should have rejected the transfer).
    A: Pre-v0.1.3 had a soft balance check: it compared the escrow's current balance to the claimed payment amount. Since the escrow's existing balance always exceeds a single payment, the check passed even when 0 actually landed — and the file got pinned for free. Fixed in v0.1.3: /upload now polls Hive-Engine's getTransactionInfo RPC (at api.hive-engine.com/rpc/blockchain) for an authoritative success/fail signal. If the sidechain rejected the transfer, the upload is now hard-rejected with the actual sidechain error message and the reservation is cancelled. If git pull && docker compose up -d --build doesn't fix, check container logs for sidechain rejected tx ... entries. Confirmed during the same 2026-05-25 session when a user with 0.9 CNOOBS triggered a "successful" upload that was actually unpaid.
  9. Q: I ran git fetch --all && git reset --hard origin/main && docker compose ... --build && docker compose up -d and now the HTTPS site is unreachable (port 443 connection refused; HTTP 80 still works).
    A: git reset --hard clobbered your operator-specific nginx/ipfs-gate.conf (which had the real domain + cert path) back to the placeholder shipped in the repo. Nginx then can't load the cert, fails to bind 443, and the container restart-loops. Always back up nginx config before a hard reset. Working recipe:
    cp nginx/ipfs-gate.conf nginx/bk.ipfs-gate.conf       # one-time backup
git fetch --all && git reset --hard origin/main && \ docker compose down && docker compose build --no-cache && \ cp nginx/bk.ipfs-gate.conf nginx/ipfs-gate.conf && \ docker compose up -dIf you've already lost the file: restore it from nginx/ipfs-gate.conf.example (or the wiki Step 4 instructions) — replace v4call.com/placeholder with your real domain. Same gotcha exists on v4call. Confirmed during 2026-05-25 testing.
  1. Q: I deleted a CID (admin takedown / expiry) but my browser still serves it from cache for hours.
    A: The gateway sets Cache-Control: public, max-age=86400 (1 day) on /ipfs/:cid responses by default. Browsers respect this and skip the network entirely until the cache expires. Use /status/:cid from curl or an incognito window to see the gate's real state. For dev/testing, lower the cache window by setting GATEWAY_CACHE_MAX_AGE=3600 (1h) or even 60 (1min) in .env, then docker compose down && docker compose up -d (env changes need recreate per Common Problems #4). Pin state in the DB is independent of browser cache — the sweeper expires pins on schedule regardless. Documented in v0.1.4.
  2. Q: Uploads work but the gateway URL in the client bubble shows https://ipfs.localhost/ipfs/... instead of my real domain.
    A: You haven't set PUBLIC_GATEWAY_BASE in .env. The server falls back to https://ipfs.localhost when neither PUBLIC_GATEWAY_BASE nor SERVER_DOMAIN is set. Add this to your .env:
    PUBLIC_GATEWAY_BASE=https://ipfs.your-domain.com
    Then docker compose down && docker compose up -d (env changes need recreate per Common Problems #4). v4call's client has a safety net that substitutes the user's chosen gateway URL when the server returns a localhost one, so end-users can still fetch — but new clients won't have that fallback, so fix at the source. Was missing from .env.example pre-v0.1.2; documented in v0.1.2.
  3. Q: Server logs say kubo backend status: unreachable (fetch failed).
    A: Kubo container isn't healthy yet, or the KUBO_API_URL in .env doesn't match the docker-compose service name. Default is http://kubo:5001; only change if you renamed the kubo service.
  4. Q: The cert renewal cron fires but nginx still serves the old cert.
    A: Certbot writes new files but nginx doesn't auto-reload. The cron line in Step 8 includes docker compose restart nginx after renew — make sure you copied that part too.
  5. Q: Let's Encrypt certbot fails with "DNS problem" or "No A/AAAA record for host".
    A: Your domain's A record isn't pointing at this VPS, or DNS hasn't propagated yet. Verify with dig ipfs.your-domain.com from any machine. If it returns the wrong IP, fix at your DNS provider and wait 5-10 minutes for propagation, then retry the certbot command from Step 5.

Resources