IPA files downloaded by ipatool are defect and can not be read in streaming mode

[jpstotz]

What happened?

The IPA files created by ipatool are defect and can not be read using the entry headers saved right before each ZIP entry. Only tools that use the central directory at the end of the file can read the IPA/ZIP file.

Steps to reproduce:

1. Download a sample app: ipatool download --purchase -i 1482894790
2. Save the following Java sample code to a file ZipTest.java

import java.io.InputStream;
import java.nio.file.*;
import java.util.zip.*;

public class ZipTest {
	public static void main(String[] args) {
		try (InputStream in = Files.newInputStream(Paths.get(args[0]))) {
			
			ZipInputStream zipIn = new ZipInputStream(in);
			ZipEntry entry;
			while ((entry = zipIn.getNextEntry()) != null) {
				System.out.println(entry.getName());
				zipIn.closeEntry();
			}
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
}

3. Execute the Java sample program on the downloaded file (require a JDK), e.g. 1482894790_1.3.1.ipa: java ZipTest.java 1482894790_1.3.1.ipa"
4. The code will fail because the downloaded IPA file is defect:

META-INF/
META-INF/com.apple.ZipMetadata.plist
java.util.zip.ZipException: only DEFLATED entries can have EXT descriptor
        at java.base/java.util.zip.ZipInputStream.readLOC(ZipInputStream.java:529)
        at java.base/java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:153)
        at ZipTest.main(ZipTest.java:13)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at jdk.compiler/com.sun.tools.javac.launcher.Main.execute(Main.java:484)
        at jdk.compiler/com.sun.tools.javac.launcher.Main.run(Main.java:208)
        at jdk.compiler/com.sun.tools.javac.launcher.Main.main(Main.java:135)

The error is not specific to Java, every ZIP tool that uses streaming mode will fail on the IPA files downloaded by ipatool.

AI description of the problem:

The error "only DEFLATED entries can have EXT descriptor" means that the ZIP file’s local file header has bit 3 of the general purpose flag set, which signals that the compressed entry is followed by a data descriptor (also called an extended descriptor or EXT descriptor) containing the CRC-32, compressed size, and uncompressed size. However, the entry’s compression method is not DEFLATE (method 8) — it might be stored (0), bzip2 (12), LZMA (14), or another method.
In the ZIP specification, the data descriptor flag is intended for streaming scenarios where the sizes are unknown until after compression. While the official APPNOTE.TXT does not explicitly restrict this flag to DEFLATE‑only, virtually all mainstream ZIP tools and libraries require that only entries using DEFLATE carry the descriptor. This is because other compression methods either need the sizes in the header for proper decoding or have different internal framing that makes the descriptor ambiguous.
Therefore, a ZIP file with bit 3 set and a non‑DEFLATE method is considered malformed. It violates the de facto compatibility rules of the ZIP format and will be rejected by programs like 7‑Zip, Info‑ZIP, and many Java/Python libraries that enforce this constraint. The file was likely produced by a buggy or non‑standard ZIP writer, or it may be corrupted. To fix it, the entry should use DEFLATE or the descriptor flag should be cleared.

I have developed PR #506 which fixes this problem, unfortunately the PR is ignored so far. Thus I created this issue to make it clear that the IPA files created by ipatool are not fully valid ZIP files.

Version

v2.3.2

Relevant log output

[bunnysayzz]

I would like to take this one. I traced the root cause: every download ends in applyPatches (appstore_download.go:102), which opens zip.NewWriter and rebuilds the archive via replicateZip (appstore_replicate_sinf.go:136). There header := file.FileHeader is passed to dst.CreateHeader(&header), and Go’s zip.Writer always sets the data-descriptor flag (bit 3 of the general purpose flag) because sizes are unknown at header-write time. For entries whose method is STORED (0), that combination is exactly what Java/7-Zip/Info-ZIP reject (“only DEFLATED entries can have EXT descriptor”), so streaming readers fail while central-directory readers succeed. The fix is to write stored entries with known sizes and the descriptor flag cleared (e.g. CreateRaw with the source header’s CRC/sizes, or setting the local header fields when the method is stored). I can open a PR with a repro test (a stored-entry zip that fails in streaming mode before the fix) once you green-light the approach.

[jpstotz]

@bunnysayzz What you describe is exactly what I have implemented in PR506. In the beginning I simply wanted to remove the unnecessary re-compression of the whole downloaded IPA file and later noticed that this also solved the "only DEFLATED entries can have EXT descriptor" problem.

You wrote that Go’s zip.Writer always sets the data-descriptor flag (bit 3 of the general purpose flag) because sizes are unknown at header-write time. Looking at the problem that way this can be considered of a bug in Go’s zip.Writer implementation. May be we should report that also to the Go project?

[bunnysayzz]

Makes sense, thanks for the pointer. Since PR #506 already implements exactly this fix (and predates my claim), I'll defer to it and won't open a competing PR. Happy to help with a rebase or anything else if it helps get it merged.

And agreed the Go behavior is worth a closer look: zip.Writer setting the descriptor flag unconditionally for STORED entries is what produces files that strict streaming readers reject, so a report to the Go project with a repro would benefit everyone. I'll look into whether there's a clean way to demonstrate it.