Harmony Protocol in Rust: Parsing OpenAI's GPT-OSS Response Format

OpenAI’s Harmony is the response format for gpt-oss models. We reverse-engineered it and implemented it in Rust. This is the deep dive.

What is Harmony?

Harmony is a structured response format that gpt-oss models use. It is not ChatML — it is a different, more structured format with “channels” that separate different types of content (analysis, final answer, etc.).

The key difference from ChatML: Harmony channels are semantic. The model can write analysis in one channel and the final answer in another. The consumer (the agent runtime) can choose which channel to surface to the user.

The channel semantics

A Harmony response looks like:

<|channel|>analysis<|message|>Let me think about this step by step...<|end|>
<|channel|>final<|message|>The answer is 42.<|end|>

The analysis channel is the model’s reasoning. The final channel is the answer. The runtime can choose to show both, show only final, or use analysis for debugging.

The Rust implementation

Our Rust implementation is at github.com/sarkar-dipankar/harmony-protocol. It is a zero-copy parser that handles:

1. Streaming: parse partial messages as they arrive. Essential for real-time UX.
2. Channel routing: route different channels to different consumers. The analysis channel goes to the debugger; the final channel goes to the user.
3. Error recovery: if the model produces malformed Harmony (which it does ~2% of the time), recover gracefully and continue parsing.
4. Comparison with ChatML: our implementation also parses ChatML, so it works as a drop-in replacement for existing ChatML consumers.

Why Rust for this

The parser runs in the hot path of every LLM response. Latency matters. A Python parser adds 5-20ms per response. The Rust parser adds <0.1ms. For an agent runtime handling hundreds of concurrent sessions, this is the difference between responsive and sluggish.

How to use it

use harmony_protocol::{HarmonyParser, Channel};

let mut parser = HarmonyParser::new();
let chunks = parser.feed(&response_bytes);

for chunk in chunks {
    match chunk.channel {
        Channel::Analysis => { /* log for debugging */ }
        Channel::Final => { /* send to user */ }
        _ => {}
    }
}

The parser is also available as a Python module via PyO3, for teams that want the Rust performance without abandoning Python.

The bigger picture

Harmony Protocol is one piece of the Neul Labs agent infrastructure stack. It sits in the “runtime” layer — the part that turns a model response into something the agent can act on. The Substrate Pattern sits above it (safety), and the tool substrate sits below it (what the agent can call).

— Dipankar Sarkar, Founder of Neul Labs