JIT context rehydration via request_context tool

t-677·WorkTask·
·
·
Created1 month ago·Updated1 month ago·pipeline runs →

Description

Edit

Add a JIT (just-in-time) context expansion mechanism to the PromptIR compilation pipeline. Currently each agent turn is a full AOT compilation: ContextRequest → hydrate → PromptIR → compile → CompiledPrompt → LLM. The model cannot request additional context mid-turn. This task adds a mid-turn re-entry point.

Design

1. New tool: request_context

Add a tool the model can call mid-turn to trigger incremental context hydration. Arguments:

  • query (string): semantic description of what context is needed
  • scope (string, optional): hint — temporal, semantic, knowledge
  • time_range (string, optional): for temporal scoping

When the interpreter encounters this tool call, instead of executing an external command, it: 1. Constructs a new ContextRequest with the model's query as observation and the scope hint adjusting ContextStrategy 2. Calls hydrate for just that scoped request (incremental, not full recompile) 3. Produces new Section(s) and splices them into the existing PromptIR 4. Runs budget enforcement on the delta 5. Returns the hydrated context as a tool result the model sees inline

2. Incremental hydration (not full recompile)

Hydrate.hydrate already builds sections independently. The JIT path should only hydrate the new source the model requested, produce new Section(s), and splice them into the existing PromptIR. Budget enforcement runs again on just the delta. This avoids throwing away KV cache and is analogous to a JIT optimization pass — patch-compile just the hot section.

3. Interpreter change (Sequential.hs)

In Sequential.hs, the tool-call handler needs a new branch: when the model calls request_context, the interpreter: 1. Runs incremental hydration (scoped to the query) 2. Splices new section into the running PromptIR 3. Re-compiles (just formatting, since sections are already assembled) 4. Returns the new context as the tool result 5. Continues the LLM conversation with the new context visible inline

The model's conversation history now has the expanded context as a tool response. No restart needed.

4. What the model sees

Turn starts with compiled prompt (AOT). Model is reasoning. It realizes it needs details from, say, a conversation last Tuesday. It calls:

{"tool": "request_context", "args": {"query": "conversation about billing changes", "scope": "temporal", "time_range": "2026-02-17"}}

Gets back the actual messages (lossless). Continues reasoning with that new context in the same turn.

5. Key properties

  • Composes with existing Priority, Budget, Provenance, and composition mode machinery — the dynamically-fetched section gets the same treatment as compile-time sections
  • It is a late-bound section, not a special case
  • Unlike LCM's summary DAG navigation, the model just describes what it wants (semantic query) and the system finds it
  • Lossless by default — returns actual messages/data, not summaries

Context

This is the "JIT" extension to the compiler/interpreter analogy: PromptIR is a compiler (AOT, whole-program optimization, inspectable artifacts), RLM is an interpreter (dynamic, model-driven), LCM is a VM (structured opcodes). The JIT combines static compilation via PromptIR with model-triggered recompilation mid-turn when the model hits a "cache miss" on its compiled context.

Related: t-399 (rate-distortion pruning), t-432 (compaction), t-345 (RLM integration research)

Timeline (0)

No activity yet.