Predicted Outputs made regeneration latency depend on a declared reference string
OpenAI released Predicted Outputs on November 4, 2024 to reduce latency when much of a document or code response is known ahead of time.
Historical reporting: OpenAI’s API changelog records Predicted Outputs on November 4, 2024, and OpenAI’s Python SDK added a prediction parameter on November 5. This article does not infer present model support, pricing, latency or compatibility.
Key takeaways
- Predicted Outputs target regeneration where a substantial expected string is already known.
- Latency claims require the reference, match rate, model and stream conditions to be measured together.
- A predicted reference is input data and needs the same review boundary as other prompt material.
The feature targeted known-output regeneration[source 1][source 2]
OpenAI’s API changelog records the release of Predicted Outputs on November 4, 2024 for responses where much of the content was known ahead of time, identifying document and code regeneration with minor changes as the common case. On November 5, OpenAI’s Python SDK commit “fix: add new prediction param to all methods” provided a contemporaneous technical record that the request field had been added to the SDK.
This is a specific performance technique, not a general claim that every completion becomes faster. It depends on how much of the produced output matches the supplied reference, the model path, request shape and service conditions. A historical feature release is not a substitute for a current latency measurement.
The reference string becomes a reviewed input[source 1][source 2]
A predicted string may contain source code, draft text, templates or other sensitive material. Treat it as request content: classify it, minimise it, log an appropriate digest and apply the same access and retention rules as comparable prompt inputs. Do not assume an optimisation field is harmless merely because it is not a user-facing instruction.
For code regeneration, bind the reference to a file revision and record whether the returned content matched, diverged or stopped early. A fast result that silently preserves an obsolete clause or discards an intended edit is not a success. This is original engineering analysis and does not describe OpenAI’s current data handling or product controls.
Analysis: measure latency with match quality[source 1][source 2]
A useful benchmark records model identifier, region or endpoint where relevant, input size, prediction size, output size, streaming setting, time to first token, completion time and match or divergence behaviour. Include several edit shapes: a small local change, a large contiguous rewrite and a request that differs substantially from the reference. One demo result cannot establish an application’s tail latency.
Pair speed metrics with correctness checks. Parse generated code, run focused tests, check diff size and reject unsupported file changes. For prose, check required sections, citations and unintended carryover. Optimising regeneration should not weaken the acceptance rule for the regenerated artefact.
Analysis: preserve feature history without inferring current API behaviour[source 1][source 2]
This article records the November 4, 2024 changelog event and the November 5 SDK commit. It does not assert current endpoint parameters, supported models, price, rate limits, retention or measured performance. Those facts require current official documentation and a live test in the intended account.
The durable lesson is to make both expected output and measured result inspectable. A latency optimisation remains a product decision only when its reference input, configuration and quality evidence can be reviewed together. When a reference includes user content, use a deletion and access policy that can be demonstrated rather than assumed. Report cache and queue conditions, which can otherwise dominate a small prediction benefit.
Sources
- OpenAI API changelog: Predicted Outputs (November 4, 2024)
- OpenAI Python SDK commit: add new prediction param to all methods (November 5, 2024)
Continue reading
Related reading
News analysisRunway introduced Dev Playground for testing API integrations3 min read
News analysisOpenAI, Nscale and Aker announced Stargate Norway as a planned Narvik facility, not live European compute3 min read
News analysisLangfuse v3 separated transactional metadata from high-volume LLM observations3 min read