Pipecat src/pipecat/utils/context/

Three texts,
one cursor.

How Pipecat keeps what the LLM wrote, what the screen renders, and what the TTS speaks in sync — word by word, while the audio is still playing.

Learn these three colours now. They label every table, diagram and cursor for the rest of the deck.

1.1 One response, three consumers

The same sentence has to be three different strings.

A bot prompted to wrap cards in <card> and code in <code> produces exactly one thing:

1.2 What you get back

The provider reports only the words it actually spoke.

No offsets. No segment ids. No tags. Just tokens, in order, with timestamps — and they are the only signal available. Everything in this deck exists to answer three questions for each one.

Three questions, three layers — TextSegmentMap, WordCompletionTracker, AggregatedFrameSequencer. But first, what happened when they didn't exist.

2.1 The context drifted from what the LLM wrote

The bot slowly forgets its own conventions.

The text appended to the context was the text that came back from the TTS — so everything that made the output speakable was lost on the round trip.

Why this one is the worst: it fails silently and slowly. Nothing errors, nothing logs, and the regression shows up as "the model got lazy" several turns later.

2.2 Skipped frames arrived out of order

A frame with no audio has nothing to wait for.

Configure skip_text_types=["code"] and that frame is never synthesized — no audio, no word events. So it was pushed the moment it appeared, landing before the sentence that precedes it.

LLM:      "Run this:"  →  <code>npm install</code>  →  "Then reload."

Context:  <code>npm install</code>      ← arrived first, nothing to wait for
          "Run this:"                   ← spoken later
          "Then reload."

The transcript reads out of order — and on interruption it is worse: text that was never spoken could still be recorded as if it had been.

2.3 Highlighting spoken words was not possible

Two streams, no correspondence between them.

The client could not turn a stream of words into a highlight moving through a rendered sentence. There was simply no key linking the two.

2.4 RTVI bot_output_transforms were useless word-by-word

You cannot redact a card number that arrives in four pieces.

Client-side transforms operate on a segment. Receiving text word by word — with no segment identity, and no notion of "spoken so far" versus "remaining" — left nothing coherent to transform.

Four disconnected events, no segment, no split. To produce XXXX-XXXX-XXXX-3456 a transform needs the whole segment plus the current spoken split — which is exactly what did not exist.

2.5 Problems found along the way

Then reality: providers, streaming, concurrency.

These four are why tests/test_word_completion_tracker.py is 203 tests, most of them a regression corpus of real provider behaviour.

2.6 And a feature: text transformations

Rewritten for the provider. For nobody else.

TTS engines read what you give them, and written text is often not what you want spoken. $42.50 can come out "dollar forty two point five zero"; 1994 as "one thousand nine hundred ninety four" when the sentence means a year. So Pipecat ships a bundle of transforms that rewrite the text on its way to the provider — and nowhere else.

III

How they're solved

The solution has two halves — and the first one is what makes the second possible.

Rebuilding the context from what the TTS spoke is exactly what caused 2.1. Once the three texts exist separately from the moment they are created, there is nothing to reverse-engineer — the alignment problem becomes "where are we in each", not "what did this used to say".

3.0 Every problem, and what answers it

The map for the rest of the deck.

3.1 Three texts, produced at two split points

Nobody authors three strings. Two split points create them.

The middle channel is the AggregatedTextFrame itself — and it is never rewritten. With a SimpleTextAggregator and no transformers, all three are the same string; they diverge only where something acted.

3.2 What that looks like for one response

One sentence in. Six frames out.

Only the two tagged frames do anything interesting. Frame 2's delimiters moved into raw_text; its transformer wrapped the digits in <spell>. Frame 5 is never spoken — which is what creates the ordering problem in 2.2. text_type is the routing key throughout: it picks the transformer, decides whether the frame is spoken at all, and selects the RTVI redaction.

3.3 The guarantee that makes it useful

One rule holds after every single word.

Exactly — character for character. Never a paraphrase, never a normalised copy, never off by a space. The two halves are always a clean split of the string the segment frame is already carrying.

• Highlight the first half, leave the second plain → word highlighting
• Redact both halves → redaction that keeps advancing
• Concatenate them → the original back, exactly

That is what makes progress frames usable by anything, without coordination. A consumer holding the segment frame never has to guess how the text was transformed on its way to the TTS.

3.4 The three layers

Each owns one concern. None knows about the layer above.

  TTS provider ── word-timestamp events ──┐
                                          ▼
 ┌────────────────────────────────────────────────────────────────┐
 │  AggregatedFrameSequencer            one per TTS service       │
 │   ordered slot queue:  [ spoken ][ skipped ][ spoken ] …       │
 │                    ┌────────────────┐   ┌────────────────┐     │
 │                    │ WordCompletion │   │ WordCompletion │     │
 │                    │ Tracker        │   │ Tracker        │     │
 │                    │  ┌───────────┐ │   │  ┌───────────┐ │     │
 │                    │  │ TextSeg   │ │   │  │ TextSeg   │ │     │
 │                    │  │ mentMap   │ │   │  │ mentMap   │ │     │
 │                    │  └───────────┘ │   │  └───────────┘ │     │
 │                    └────────────────┘   └────────────────┘     │
 └────────────────────────────────────────────────────────────────┘
             │                                    │
             ▼                                    ▼
      TTSTextFrame                   AggregatedTextProgressFrame
   → conversation context                → RTVI → the client

★ TextSegmentMap · one real cursor drives three

Watch the followers hold, then jump.

Only raw_pos is a real cursor. Through an unchanged segment the other two move by a count — however many letters and digits the word spent. Through a rewritten one they are held, then jump to the end of the original span in one step, because no position halfway through $42.50 means anything. Nothing ever speaks a </card>, so the walk ends on a beat that is not a word at all: once the frame is done, the tracker claims whatever llm text has left — which is how a closing tag reaches the conversation context. (↑ / ↓ also step.)

4.1 Building the map

Diff the two texts once, at word level.

difflib.SequenceMatcher compares TTS text against segment text, tokenized on whitespace keeping the whitespace as tokens so offsets stay exact. Each opcode becomes one TextSegment.

original = 'Your balance is $42.50'
tts      = 'Your balance is forty-two
            dollars and fifty cents'

equal    'Your balance is '
replace  '$42.50'
         → 'forty-two dollars…'

TextSegment(
  original='$42.50',
  tts='forty-two dollars…',
  original_start=16,
  original_end=22)

4.2 is_transformed · split_markup_runs

All-or-nothing, and how that cost is contained.

split_markup_runs("I love to count <spell>1234</spell>.")
# → ["I love to count ",
#    "<spell>1234</spell>."]

4.3 advance_by_alnums

Counting is what attaches tags to words.

Nothing decides which words own which tags. One loop counts letters and digits and steps over everything else — where the tags land simply follows from that.

llm: Your card is <card>4111</card>
                  ↑ costs 0    ↑ 4 digits = 4
llm_pos = 23, not 17

provider reports:  "Yeah"
context records:   "Yeah,"

The closing tag is the one thing the loop deliberately stops short of — it is swept up by WordCompletionTracker when the frame completes. Why no diff for llm_text? It holds the same letters and digits as the segment text, in the same order, differing only in what wraps them. So the map never finds positions in it — it only counts.

4.4 _classify_hop

Challenge: Provider tokens are messy.

Three strategies, first hit wins.

4.5 The four outcomes

Two end the walk. Two carry the token onward.

tts='Hello five'  original='Hello 5'

token 'Hello five'
  ├─ seg0: CROSSES, 6 chars → completes
  └─ seg1: PLACED → user_facing_pos 7

# one token completed two segments and
# crossed a transform, in one call

5.1 WordCompletionTracker

The map says where. The tracker decides what that means.

One call in, several answers out — each read back by the sequencer to build the frames it pushes.

A single process_word call reads six of these to build one TTSTextFrame plus one AggregatedTextProgressFrame. Note the right-hand column: only add_word_and_check_complete() ever calls advance_word() — one entry point moves the map, and every other accessor just reads the state that call left behind. And is_complete is delegated, not tracked — forwarded to the segment map, except on a force-completed slot where the map was deliberately never advanced.

5.2 Span attribution

Recovering structure the TTS never saw.

5.3 When providers misbehave

Never stall. Emit the remainder and move on.

5.4 AggregatedFrameSequencer

Every frame takes a slot — spoken or not.

5.5 Concurrent contexts

Two utterances in flight, one timeline.

Two back-to-back TTSSpeakFrames may overlap run_tts returns before synthesis finishes. Ordering still has to hold across both.

seq.process_word("beta",  pts=1000, context_id="ctxB")   # → TTSTextFrame('beta')
seq.process_word("alpha", pts=1000, context_id="ctxA")   # → TTSTextFrame('alpha')

ctxB's word reaches ctxB's slot even though ctxA's slot sits earlier and is still incomplete. A word for a context with no live entry is dropped as stale — that is what stops timestamps delivered seconds after an interruption from bleeding into the next turn. A None context id matches any slot (legacy providers, where concurrency cannot occur).

5.6 Token mode (streaming)

The sentence is always known one token late.

In TextAggregationMode.TOKEN the service dispatches word-sized chunks — but tracking and progress need a sentence, and a boundary is only confirmed by lookahead: the first non-whitespace character of the next sentence.

6.1 Where they're wired up

All three are driven from TTSService.

The service owns one sequencer; the sequencer builds a tracker per slot; each tracker builds one segment map. src/pipecat/services/tts_service.py

Two ordering problems stay with the service, because both concern queues the sequencer cannot see: push_frame stamps the will_be_spoken anchor with max(_word_timestamp_tracker.last_pts, clock.now()) so it rides the clock queue alongside its progress frames; and _push_sequencer_frames routes sequencer output through the audio-context queue when streaming, so a single consumer emits anchors, words and audio in order.

6.2 One sentence, end to end

Step through every call the three layers make.

the LLM wrote "Your card is <card>4111</card>" · the aggregator split the delimiters into raw_text, so "Your card is 4111" is what reaches the TTS

Captured from a real run. Note word_belongs_here runs twice per word — once for the sequencer to pick the slot, once inside add_word_and_check_complete to decide force-completion — and that advance_word is the only call that moves a cursor. Everything after it reads back what it left. (↑ / ↓ also step.)

6.3 code-helper

Four settings. Four independent channels.

# 1. Aggregate tagged segments separately
llm_text_aggregator.add_pattern(
    type="credit_card",
    start_pattern="<card>", end_pattern="</card>",
    action=MatchAction.AGGREGATE)

# 2. Never speak code blocks
tts = CartesiaTTSService(..., skip_text_types=["code"])

# 3. Rewrite what the TTS receives, per segment type
tts.add_text_transformer(spell_out_text, "credit_card")
tts.add_text_transformer(strip_url_protocol, "link")

# 4. Redact what the client renders, per segment type
rtvi_observer_params=RTVIObserverParams(
    bot_output_transforms=[("credit_card", obfuscate_credit_card)])

const cur = this.botSpans[data.segment_id];
cur.innerHTML =
  `<strong>${acc}</strong>${rem}`;

And the code block, never spoken, still lands in the transcript after the sentence that precedes it — because it waited its turn in the slot queue.

6.4 Where to look

Code coverage.

uv run pytest tests/test_text_segment_map.py tests/test_word_completion_tracker.py \
  tests/test_aggregated_frame_sequencer.py tests/test_tts_frame_ordering.py \
  tests/test_cartesia_tts.py tests/test_soniox_tts.py

Full write-ups in docs/architecture/ — one document per layer, plus RTVI integration and known rough edges.