Module text_encoding 

Text encoding detection + transcoding for the editor.

The editor used to read files with std::fs::read_to_string, which rejects anything that isn’t valid UTF-8 — so a Windows-1252 .ini, a UTF-16 file, Shift_JIS, etc. simply failed to open. This module reads bytes, detects the encoding (BOM → valid-UTF-8 → chardetng heuristic), and decodes to a Rust String for the editor; and re-encodes on save so files round-trip in their original encoding + BOM + line endings.

Spec: docs/specs/SPEC_EDITOR_FILE_ENCODINGS_2026_06_17.md

Structs

DecodedFile

Result of decoding a file’s bytes for the editor.

Functions

decode_file

Detect the encoding of bytes and decode to UTF-8.

decode_utf32 🔒

Decode UTF-32 (LE/BE) bytes by hand — encoding_rs has no UTF-32 codec.

detect_line_ending 🔒

The file’s line ending, from the first line break (what VS Code does). Using “any CRLF present” would flag a mostly-LF file as CRLF and convert all its LF lines on save.

encode_file

Encode editor text (\n-delimited UTF-8) back to bytes in encoding_label, applying bom and line_ending. Returns the bytes to write plus whether any character was unmappable in the target encoding (the caller may warn).

utf16_bytes 🔒

utf32_bytes 🔒