fix(sync): correctness overhaul of QRZ + LoTW sync flows#185
Open
fix(sync): correctness overhaul of QRZ + LoTW sync flows#185
Conversation
Both sync paths had bugs causing silent data loss in production: QRZ download - OPTION param: send a single value with `;` separators, not two separate fields (FormData duplicates lose `TYPE:ADIF`) - MODSINCE (not MODIFIEDSINCE) — the date filter was being silently ignored, every "incremental" download pulled the full logbook - add STATUS:CONFIRMED to limit payload, extract APP_QRZLOG_STATUS - new matchQRZConfirmation: call+band+mode+station_callsign exact + ±15min tolerance (was call+date+time only with ±5min) LoTW download - URL params: qso_qslsince / qso_qslbefore (no underscores) — date filter was a no-op - add qso_qsldetail/qso_mydetail so LoTW returns enriched location fields (state, county, CQZ, ITUZ, DXCC, grid) - match: require call+band+mode+station_callsign exact, ±15min, proper UTC parsing, satellite-mode rule (sat_name must agree) - on confirmation: enrich state/cnty/cqz/ituz/dxcc/country/grid; persist stations.lotw_last_qsl_rcvd_date for incremental fetches LoTW upload (.tq8 signing) - replace the OpenSSL fallback stub (which logged a warning then returned UNSIGNED ADIF — every Vercel cron upload was rejected by LoTW) with a pure-Node node-forge implementation that produces wavelog-compatible .tq8 files (gzip + per-QSO RSA-SHA1 signature in <SIGN_LOTW_V2.0>, with the canonical sign-string echoed in <SIGNDATA>); runtime self-verify before each upload - multipart `upfile` POST + check for `<!-- .UPL. accepted -->` - filter unsupported prop_modes (INTERNET, RPT) — flagged 'I' - normalize callsigns (W1AW_P → W1AW/P) Cross-service sync - when LoTW confirms a QRZ-uploaded QSO, set qrz_qsl_sent='M' so the next QRZ sync re-uploads with OPTION=REPLACE; mirrored for the QRZ→LoTW direction Other - validateLoTWCredentials now actually parses the response body (the old response.url check always returned true) - new checkLotwCertCrl helper queries lotw.arrl.org/lotw/crl?serial= - vercel.json: maxDuration 300s on cron + LoTW routes Migration: migrations/sync_qrz_lotw_fixes.sql adds p12_password, cert_serial, crl_status to lotw_credentials; lotw_last_qsl_rcvd_date to stations; prop_mode/sat_name/band_rx/freq_rx/iota to contacts; CHECK constraints on the QSL-sent enums. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Discovered while applying to local + prod that: 1. Some envs initialized contacts without qrz_qsl_sent / qrz_qsl_rcvd (those were added by the in-app /install route, not the base schema). Add them with IF NOT EXISTS to make the migration self-sufficient. 2. The PG 18+ syntax `CREATE TRIGGER IF NOT EXISTS` in the existing postgres-lotw-migration.sql breaks on PG 15/17. Added a one-line transform in scripts/run-migration.mjs to rewrite it as DROP+CREATE so the legacy file works without editing it. scripts/run-migration.mjs is a small Node runner that takes DATABASE_URL via env and a SQL file via argv; it wraps everything in a transaction and rolls back on failure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The signer needs the P12 password to decrypt the private key. Until now the certificate-upload route stored only the .p12 file blob, so buildSignedTq8 had to assume an empty password — fine for unprotected TQSL exports, broken for password-protected ones. API: /api/lotw/certificate now accepts an optional p12_password form field. Before insert, it parses the P12 with node-forge using that password (via readCertMetadata). Bad password / corrupt file fails fast with a clear error rather than landing an unusable cert in the DB. We also persist cert_serial and cert_expires_at extracted from the parse, populating columns the migration just added. UI: both upload forms got a password input with show/hide toggle. Helper text explains the password is required to sign uploads, stored encrypted, and never returned to the browser. Empty is accepted for TQSL exports without a password. Side fix: the LoTW dashboard form was already broken — it didn't send cert_name (required by the API). Added a Certificate Name field there too, and restructured the 3-col grid into a 2x2 layout so all four fields fit cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Both QRZ and LoTW sync paths had bugs causing silent data loss in production. This PR ports wavelog's mature behavior into nextlog and replaces the broken LoTW signer with a Vercel-compatible pure-Node implementation.
Critical bugs fixed (all silent failures):
MODSINCEparam was misspelled (MODIFIEDSINCE) andOPTIONwas sent as two separate FormData fields, droppingTYPE:ADIF. Date filter now actually filters.qso_qsl_since→qso_qslsince). Detail flagsqso_qsldetail/qso_mydetailwere missing, so location enrichment was silently lost.console.warn('not fully implemented')then return unsigned ADIF. Replaced with a pure-Nodenode-forgesigner that produces wavelog-compatible.tq8files (gzip + per-QSO RSA-SHA1 signature in<SIGN_LOTW_V2.0>). Includes runtime self-verify before upload.Wavelog parity added: callsign normalization (
_→/),INTERNET/RPTprop_mode filter (TQSL ≥ 2.7.3 rejects), multipartupfile+<!-- .UPL. accepted -->response check, CRL helper (lotw.arrl.org/lotw/crl?serial=...), confirmation enrichment (state/county/CQZ/ITUZ/DXCC/grid), incremental download bookmark, robustvalidateLoTWCredentials.Cross-service sync: when LoTW confirms a QRZ-uploaded QSO, sets
qrz_qsl_sent='M'so the next QRZ sync re-uploads withOPTION=REPLACE. Mirrored for the QRZ→LoTW direction.Infrastructure:
migrations/sync_qrz_lotw_fixes.sql(idempotent),node-forge+@types/node-forgedeps,vercel.jsonmaxDuration: 300son cron + LoTW routes (RSA-2048 signing in JS scales as ~5–15 ms/QSO).Action required before deploying
migrations/sync_qrz_lotw_fixes.sqlagainst the databaselotw_credentials.p12_password; existing rows have NULL and will sign with empty password — fine for most TQSL exports but won't work for password-protected ones)Test plan
npm run buildpasses (verified locally)npm run lintclean on changed files (verified locally)since=<30 days ago>, inspect outbound request — confirm singleOPTIONvalue withTYPE:ADIF;STATUS:CONFIRMED;MODSINCE:.... Response should be incremental, not full logbook.qrz_qsl_sent='M'is set and the next QRZ sync re-uploads with REPLACE.Follow-ups (not in this PR)
buildSignedTq8is the runtime safety net for now.🤖 Generated with Claude Code