How to Handle Unicode and CJK Output Encoding on Windows with Console.OutputEncoding

OfficeCLI guarantees correct Unicode and CJK character display on Windows by temporarily switching Console.OutputEncoding to UTF-8 without BOM when writing to an interactive terminal, then automatically restoring the original code page on process exit.

When building cross-platform CLI tools in .NET, Windows consoles frequently corrupt Chinese, Japanese, and Korean (CJK) characters by defaulting to legacy code pages like CP936 or CP932. The OfficeCLI repository solves this through a robust encoding management strategy implemented in src/officecli/Program.cs that detects console attachment, switches to UTF-8, and guarantees cleanup.

Detecting Interactive Console Output

Before modifying encoding, OfficeCLI determines whether the process has a real console attached or if output is being piped to a file or another process. In Program.cs, the code validates both standard output and standard error streams:

if (!Console.IsOutputRedirected || !Console.IsErrorRedirected)
{
    // Console is interactive, safe to modify encoding
}

This check prevents encoding changes when running under redirection scenarios like officecli ... > output.txt or piping to | grep pattern, where the receiving process expects the original encoding.

Storing the Original Windows Code Page

To avoid permanently altering the user's terminal environment—a common source of "mojibake" in subsequent commands—the implementation immediately caches the current encoding:

var previous = Console.OutputEncoding;

This preserves the system default (often CP850, CP936, or CP1252 depending on Windows locale) so it can be restored after the CLI finishes execution.

Switching to UTF-8 Without BOM

When the existing code page differs from UTF-8 (code page 65001), OfficeCLI instantiates a UTF8Encoding with the BOM explicitly disabled to prevent byte-order mark artifacts in console output:

var utf8NoBom = new System.Text.UTF8Encoding(encoderShouldEmitUTF8Identifier: false);

if (previous.CodePage != System.Text.Encoding.UTF8.CodePage)
{
    Console.OutputEncoding = utf8NoBom;
    // Subsequent Console.Write* calls emit UTF-8 bytes
}

Setting encoderShouldEmitUTF8Identifier: false ensures that the first character written does not spit out invisible bytes that might confuse terminal emulators or downstream parsers.

Restoring Original Encoding on Process Exit

The most critical step prevents the console from remaining locked to UTF-8 after the application terminates. OfficeCLI attaches restoration logic to both normal exits and cancellation events:

void Restore(object? sender, EventArgs e)
{
    try { Console.OutputEncoding = previous; } 
    catch { /* console already gone */ }
}

AppDomain.CurrentDomain.ProcessExit += Restore;
Console.CancelKeyPress += Restore;

This Restore handler catches all exceptions silently because the console buffer may already be detached if the user closes the terminal window aggressively.

Handling Redirected Output Streams

When Console.IsOutputRedirected returns true, directly setting Console.OutputEncoding has no effect on the stream writer. Instead, OfficeCLI replaces Console.Out and Console.Error with StreamWriter instances using the same UTF-8 encoding. As demonstrated in ResidentServer.cs (lines 219-229), this ensures redirected bytes reach their destination unchanged regardless of the system code page:

// Pattern used in ResidentServer.cs for error stream management
var utf8Writer = new StreamWriter(Console.OpenStandardError(), utf8NoBom);
utf8Writer.AutoFlush = true;
Console.SetError(utf8Writer);

Implementing Unicode Support in Your Own .NET CLI

You can adopt this pattern in any .NET console application by wrapping the encoding logic at the entry point:

using System;
using System.Text;

class Program
{
    static void Main()
    {
        var utf8NoBom = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false);
        
        if (!Console.IsOutputRedirected && !Console.IsErrorRedirected)
        {
            var original = Console.OutputEncoding;
            
            if (original.CodePage != Encoding.UTF8.CodePage)
            {
                Console.OutputEncoding = utf8NoBom;
                
                AppDomain.CurrentDomain.ProcessExit += (_, __) =>
                {
                    try { Console.OutputEncoding = original; } catch { }
                };
            }
        }

        Console.WriteLine("日本語テキスト and 中文内容 display correctly");
    }
}

Summary

• Detection matters: Always check Console.IsOutputRedirected before modifying encoding to avoid breaking pipelines.
• UTF-8 without BOM: Use new UTF8Encoding(encoderShouldEmitUTF8Identifier: false) to prevent invisible prefix bytes.
• Cleanup is mandatory: Store the original Console.OutputEncoding and restore it via ProcessExit and CancelKeyPress handlers.
• Redirects need wrappers: Replace Console.Out with explicit StreamWriter instances when output is piped.
• Source files: The complete implementation lives in src/officecli/Program.cs (lines 5-30) with additional stream handling in ResidentServer.cs (lines 219-229) according to the iOfficeAI/OfficeCLI source code.

Frequently Asked Questions

Why does Windows show garbled text with CJK characters?

Windows consoles default to legacy code pages (CP936 for Chinese GBK, CP932 for Japanese Shift-JIS) that only support a limited glyph subset. When a .NET application outputs UTF-8 bytes but the console interprets them using these legacy code pages, multi-byte sequences split incorrectly producing "mojibake" gibberish.

What is Windows code page 65001?

Code page 65001 is the Windows identifier for UTF-8. When you assign Console.OutputEncoding to a UTF-8 encoding instance, the Win32 console API switches to this code page, enabling proper rendering of Unicode characters including CJK glyphs and emoji.

Why disable the UTF-8 BOM for console output?

The Byte Order Mark (BOM) is a three-byte prefix (0xEF 0xBB 0xBF) that signals UTF-8 encoding in files, but in console output it manifests as invisible characters or garbled symbols at the start of the first printed line. Disabling it with encoderShouldEmitUTF8Identifier: false ensures clean terminal output.

How do I handle Unicode output when piping to another program?

When output is redirected (piped), use new StreamWriter(Console.OpenStandardOutput(), utf8Encoding) to wrap the raw stream. This bypasses the console code page entirely and writes UTF-8 bytes directly to the pipe, ensuring the receiving process receives uncorrupted Unicode data regardless of the terminal's current encoding.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}