Skip to content

Commit 2b8d7cc

Browse files
committed
ci: split linux VirusTotal scan by architecture
The linux-builds artifact now contains both x86_64 and arm64 binaries (AppImage, deb, rpm), pushing the zipped payload past VirusTotal's ~650MB /files/upload_url limit. Uploads were failing with a cryptic "jq: parse error" because VirusTotal returns a non-JSON error body. Split the linux scan into two virtual matrix entries (linux-x86_64-builds and linux-arm64-builds) that both resolve to the same linux-builds artifact but zip only the files for their architecture, keeping each payload well under the VT limit. No change needed in the build, AWS upload, or e2e workflows. Also prints the zip size on upload and logs the raw VT response on failure so future size/rate-limit issues surface clearly instead of as a jq parse error. Made-with: Cursor
1 parent 6c9c4b3 commit 2b8d7cc

1 file changed

Lines changed: 36 additions & 11 deletions

File tree

.github/workflows/virustotal.yml

Lines changed: 36 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,19 @@ jobs:
5050
# Get list of artifacts
5151
ARTIFACTS=$(ls ./release)
5252
53-
# Conver list to json
54-
ARTIFACTS_JSON=$(echo "$ARTIFACTS" | jq -R -s -c 'split("\n")[:-1]')
53+
# Convert list to json.
54+
# linux-builds is split into two virtual scans (x86_64 and arm64) because
55+
# the combined zip exceeds VirusTotal's ~650MB upload limit once arm64
56+
# binaries are included. Both virtual scans still download the same
57+
# linux-builds artifact; the split happens at zip time via file filters.
58+
ARTIFACTS_JSON=$(echo "$ARTIFACTS" | jq -R -s -c '
59+
split("\n")[:-1]
60+
| reduce .[] as $a ([];
61+
if $a == "linux-builds"
62+
then . + ["linux-x86_64-builds", "linux-arm64-builds"]
63+
else . + [$a]
64+
end)
65+
')
5566
5667
echo "artifact_exists=true" >> $GITHUB_OUTPUT
5768
echo "artifact_names=$ARTIFACTS_JSON" >> $GITHUB_OUTPUT
@@ -71,27 +82,41 @@ jobs:
7182
if: needs.download_artifacts.outputs.artifact_exists == 'true'
7283
uses: actions/download-artifact@v4
7384
with:
74-
name: ${{ matrix.artifact }}
85+
# linux-x86_64-builds and linux-arm64-builds are virtual scan entries;
86+
# both resolve to the single linux-builds artifact uploaded by the build.
87+
name: ${{ (matrix.artifact == 'linux-x86_64-builds' || matrix.artifact == 'linux-arm64-builds') && 'linux-builds' || matrix.artifact }}
7588
path: ./release
7689

7790
- name: Send File to scan
7891
if: needs.download_artifacts.outputs.artifact_exists == 'true'
7992
run: |
8093
uploadZipFile="./${{ matrix.artifact }}.zip"
8194
82-
# Compress artifactes
83-
zip -r "${uploadZipFile}" "./release" ${{ startsWith(matrix.artifact, 'macos-') && '-x "*/redisstack/*" "*.tar.gz" "*.zip"' || '' }}
95+
# Compress artifacts. Per-scan exclusions keep each zip under
96+
# VirusTotal's ~650MB /files/upload_url limit:
97+
# - macos-* : exclude bundled redisstack and archives
98+
# - linux-x86_64-builds: exclude arm64/aarch64 binaries
99+
# - linux-arm64-builds : exclude x86_64/amd64 binaries
100+
zip -r "${uploadZipFile}" "./release" \
101+
${{ startsWith(matrix.artifact, 'macos-') && '-x "*/redisstack/*" "*.tar.gz" "*.zip"' || '' }} \
102+
${{ matrix.artifact == 'linux-x86_64-builds' && '-x "*arm64*" "*aarch64*"' || '' }} \
103+
${{ matrix.artifact == 'linux-arm64-builds' && '-x "*x86_64*" "*amd64*"' || '' }}
104+
105+
echo "File to upload: ${uploadZipFile} ($(du -h "${uploadZipFile}" | cut -f1))"
84106
85107
# Generate url to download zip file
86108
uploadUrl=$(curl -sq -XGET https://www.virustotal.com/api/v3/files/upload_url -H "x-apikey: $VIRUSTOTAL_API_KEY" | jq -r '.data')
87109
88-
echo "File to upload: ${uploadZipFile}"
89-
90-
# Upload zip file to VirusTotal
91-
analysedId=$(curl -sq -XPOST "${uploadUrl}" -H "x-apikey: $VIRUSTOTAL_API_KEY" --form file=@"${uploadZipFile}" | jq -r '.data.id')
110+
# Upload zip file to VirusTotal. Capture the raw response so a non-JSON
111+
# error (e.g. payload too large) produces a readable log instead of a
112+
# cryptic "jq: parse error".
113+
uploadResponse=$(curl -sq -XPOST "${uploadUrl}" -H "x-apikey: $VIRUSTOTAL_API_KEY" --form file=@"${uploadZipFile}")
114+
analysedId=$(echo "$uploadResponse" | jq -r '.data.id' 2>/dev/null || true)
92115
93-
if [ $analysedId == "null" ]; then
94-
echo 'Status is null, something went wrong';
116+
if [ -z "$analysedId" ] || [ "$analysedId" == "null" ]; then
117+
echo 'VirusTotal upload failed. Raw response (first 2KB):'
118+
echo "$uploadResponse" | head -c 2048
119+
echo
95120
exit 1;
96121
fi
97122

0 commit comments

Comments
 (0)