Skip to content

fix(utf8): strip only one leading BOM on decode - #415

Open
spokodev wants to merge 1 commit into
pillarjs:masterfrom
spokodev:fix/utf8-bom-double-strip
Open

fix(utf8): strip only one leading BOM on decode#415
spokodev wants to merge 1 commit into
pillarjs:masterfrom
spokodev:fix/utf8-bom-double-strip

Conversation

@spokodev

@spokodev spokodev commented Jul 9, 2026

Copy link
Copy Markdown

Decoding UTF-8 strips two leading BOMs instead of one, dropping a U+FEFF that belongs to the content.

For UTF-8 the internal TextDecoder is created with ignoreBOM: false, so it already removes one leading BOM. The decoder is then also wrapped by iconv-lite's StripBOM (added for every bomAware codec unless stripBOM: false), which removes a leading U+FEFF as well. When the byte stream is BOM followed by content that begins with U+FEFF, both strip and the content's U+FEFF is lost.

Repro:

const iconv = require("iconv-lite")
const buf = Buffer.concat([Buffer.from([0xEF, 0xBB, 0xBF]), Buffer.from("abc")])
iconv.decode(buf, "utf8")            // => "abc"     (U+FEFF dropped)
new TextDecoder("utf-8").decode(buf) // => "abc"

utf-16le/utf-16be/utf-32* all strip exactly one BOM here; only utf-8 strips two.

Fix: let StripBOM be the sole BOM stripper by setting ignoreBOM: true on the internal TextDecoder. The wrapper is present whenever a BOM should be removed (and absent for stripBOM: false, where the BOM should be kept), so a single strip is correct in every case. The stripBOM: false and stripBOM callback paths are unchanged.

Oracle: the WHATWG Encoding Standard / TextDecoder("utf-8") removes exactly one leading BOM.

Adds a regression test.

TextDecoder (ignoreBOM: false) already strips a leading BOM, and the
StripBOM wrapper strips one too, so decoding UTF-8 dropped a second
leading U+FEFF that is part of the content. Let the wrapper be the sole
BOM stripper (matches utf-16/utf-32, which strip exactly one).
@spokodev
spokodev requested a review from a team as a code owner July 9, 2026 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant