fix: add UTF-16 encoding detection and conversion to prevent assertion failures #4347
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n failures Universal Ctags crashed with assertion failure in vStringPutImpl() when encountering files with UTF-16 encoding. The assertion `c >= 0 && c <= 0xff` failed because ctags expected all characters to fit within single byte range, but UTF-16 files contain multi-byte sequences that violate this assumption. This fix adds: - Detection of UTF-16 BOM (both LE and BE) in file reading - Automatic conversion from UTF-16 to UTF-8 using iconv when UTF-16 is detected - Force memory stream processing for UTF-16 files to enable conversion - Test cases for both UTF-16 LE and BE files Resolves issue universal-ctags#4342 Signed-off-by: Bernát Gábor <[email protected]>
gaborbernat
force-pushed
the
fix-utf16-encoding-crash
branch
from
55764aa to
c041872
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4347 +/- ##
=======================================
Coverage 85.87% 85.88%
=======================================
Files 252 252
Lines 62597 62631 +34
=======================================
+ Hits 53755 53789 +34
Misses 8842 8842 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Enables test execution for existing UTF-16 test files by adding the required args.ctags configuration file. This ensures the UTF-16 LE and UTF-16 BE files are processed during test runs, improving code coverage for the UTF-16 to UTF-8 conversion functionality. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Adds test for UTF-16 conversion failure path using malformed UTF-16 data with invalid surrogate sequences. This triggers the iconv() failure path and tests the fallback mechanism that preserves original data when UTF-16 to UTF-8 conversion fails. This ensures 100% coverage of the UTF-16 conversion error handling code including the eFree(converted_data) cleanup logic. Signed-off-by: Bernát Gábor <[email protected]>
Adds specific test for UTF-16 Big Endian BOM detection (FE FF) to ensure complete coverage of line 899: (bom[0] == 0xFE && bom[1] == 0xFF). This test completes 100% coverage of all UTF-16 BOM detection paths including both LE (FF FE) and BE (FE FF) byte order markers. Signed-off-by: Bernát Gábor <[email protected]>
Contributor
Author
|
@masatake any updates on this? |
Member
|
Sorry to be late to respond. I will work on this request next. |
Contributor
Author
|
Ideally you can just review and accept this PR. Anything wrong with the solution in it? 🤔 |
Member
|
The change for getMioFull() is excellent. This change requires new section like: I need time for thinking about the new test cases. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Universal Ctags crashed with assertion failure in
vStringPutImpl()when encountering files with UTF-16 encoding. The assertionc >= 0 && c <= 0xfffailed because ctags expected all characters to fit within single byte range, but UTF-16 files contain multi-byte sequences that violate this assumption.This fix adds:
Resolves issue #4342
Signed-off-by: Bernát Gábor [email protected]