fix: decode PowerShell 5.1 UTF-16LE read_file output

[wonderwhy-er]

Summary

Fixes read_file for UTF-16LE text files produced by Windows PowerShell 5.1 redirection without changing the existing UTF-8/binary/PDF/image read paths.

Fixes #610. Supersedes #621.

Problem

Windows PowerShell 5.1 writes redirected text with a FF FE UTF-16LE BOM. read_file currently treats that text as UTF-8, which returns embedded \u0000 characters and can break downstream persistence (22P05).

Approach

• Detect the UTF-16LE BOM inside TextFileHandler with a 2-byte probe.
• For UTF-16LE files, stream with utf16le decoding and skip the BOM.
• Preserve existing positive-offset, negative-offset/tail, and [Reading ...] status semantics.
• Leave the normal UTF-8 and non-text handler paths unchanged.
• Keep readFileInternal/editing behavior out of scope so this fix does not silently transcode edited UTF-16 files to UTF-8.

Regression test

Adds test/test-utf16-powershell-read.js, which writes the exact PowerShell-style bytes (FF FE + UTF-16LE) and verifies:

• no embedded NULs or BOM in the result
• non-ASCII text decodes correctly
• line status metadata is preserved
• positive offsets work
• negative offsets return the correct tail lines

The new test fails on current main with read_file must not return embedded NULs and passes with this change.

Validation

• npm test: 59/59 tests passed on macOS
• Normal UTF-8 128 MB one-line read remains on the existing streaming path (~3 ms inside the read, ~153 MB max RSS in the local benchmark)
• Windows 11 / Windows PowerShell 5.1.26100.9168: actual @('alpha','Rīga','gamma') *> file.txt produced FF FE ...; this branch returned clean text with 0 NUL characters and the normal read-status header

Commit tested on Windows: 99f0062ba2c4f7cdeb964e3f9920153823912ad3.

Summary by CodeRabbit

• Bug Fixes
  • Added support for reading UTF-16LE text files, including files generated by Windows PowerShell.
  • Corrected line counts, pagination, and content when reading UTF-16LE files.
  • Prevented byte-order marks and embedded null characters from appearing in displayed text.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 7b528577-223b-4379-bf58-69334257ec64

📥 Commits

Reviewing files that changed from the base of the PR and between 56deabc and 99f0062.

📒 Files selected for processing (2)

• src/utils/files/text.ts
• test/test-utf16-powershell-read.js

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.

---

📝 Walkthrough

Walkthrough

TextFileHandler now detects UTF-16LE files by BOM, decodes them for metadata and positioned reads, skips the BOM during pagination, and validates start, middle, and tail reads with a 1,500-line PowerShell-style file.

Changes

UTF-16LE file reading

Layer / File(s)	Summary
Encoding detection and line counting src/utils/files/text.ts	TextFileHandler detects the UTF-16LE BOM and uses the detected encoding for file information and line counts. It removes the decoded BOM before counting lines.
Positioned UTF-16LE reads src/utils/files/text.ts, test/test-utf16-powershell-read.js	Smart positioning uses encoding-aware readline streams and skips the two-byte BOM. Tests verify reads from the start, middle, and end without embedded NUL characters or BOM content.

Priority: ➖ Normal — Schedule the PowerShell UTF-16LE compatibility fix because current decoding can introduce NUL characters and trigger PostgreSQL persistence errors.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 99f00

UTF-16LE PowerShell files now return decoded, BOM-free text with preserved pagination behavior, including offsets and tail reads. The change is ready to merge.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: decoding PowerShell 5.1 UTF-16LE output.
Linked Issues check	✅ Passed	The implementation detects the UTF-16LE BOM, decodes content correctly, skips the BOM, and adds regression coverage for issue #610 requirements.
Out of Scope Changes check	✅ Passed	The changes stay within the linked issue scope. They update UTF-16LE text handling and add focused regression tests without unrelated behavior changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 2…

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/windows-powershell-utf16-read

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}