Why do long AI API requests time out: streaming or non-streaming?

In short: Timeouts on long answers usually happen because the client gives up while waiting for the first byte or for the whole response. With a streaming request (stream: true) text arrives as it's written, which makes it the safest choice for long jobs. For a non-streaming request the Wiozen API sends the response headers after 30 seconds and keeps the connection open with whitespace; on your side, set the client's total timeout close to 10 minutes and check the body for an error field.

What's the difference between streaming and non-streaming?

In a non-streaming request the model writes the whole answer and you get all of it as one JSON document. If the model thinks for two minutes and writes for three, you have nothing for five minutes. In a streaming request the answer arrives piece by piece as Server-Sent Events; the first words show up as soon as the model starts writing.

Billing is the same: the same tokens are counted either way. The difference is how the wait is spent. A stream carries data the whole time, a non-streaming response only at the end. Most timeouts happen during that silent wait.

How does a streaming request stay open?

Before its first token the model may be thinking or working through a long prompt. Meanwhile the Wiozen API sends the 200 headers within 12 seconds at the latest, then writes a : ping comment line every 10 seconds. Under the SSE rules a line starting with a colon is a comment; SDKs skip it, but the connection never looks idle.

If the model produces nothing at all for 120 seconds, the stream is treated as broken and closed. A single request can run for 10 minutes at most.

from openai import OpenAI

client = OpenAI(base_url="https://api.wiozen.com/v1", api_key="sk-gw-...", timeout=600)

stream = client.chat.completions.create(
    model="wiozen-anthropic",
    messages=[{"role": "user", "content": "Rewrite this 300-line file"}],
    max_tokens=32000,
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Editor extensions and terminal agents mostly stream already. If your own code expects long output (rewriting a file, a long report, a translation), turning streaming on causes the fewest problems.

What happens to a non-streaming request after 30 seconds?

If the request hasn't finished in 30 seconds, the Wiozen API sends the 200 headers and a single space without waiting for the answer. While the model keeps writing, another space goes out every 15 seconds, and when the answer is done the JSON body is appended after those spaces. JSON allows whitespace before a value, so json.loads, JSON.parse and the SDKs read the body as usual.

There's a price for this: once the headers are out, an error that happens later can't change the status code, which stays 200. The error then arrives in the body as an error object. If you write your own HTTP code, don't rely on the status code alone; also check whether the body has an error field.

import requests

r = requests.post(
    "https://api.wiozen.com/v1/chat/completions",
    headers={"Authorization": "Bearer sk-gw-..."},
    json={"model": "wiozen-anthropic", "messages": [...], "max_tokens": 32000},
    timeout=(10, 120),  # 10 s to connect, at most 120 s between bytes
)
data = r.json()
if "error" in data:  # even when the status is 200
    raise RuntimeError(data["error"]["message"])
print(data["choices"][0]["message"]["content"])

The timeout=(10, 120) in the example waits at most 120 seconds between two bytes. Since a space arrives every 15 seconds, it never runs out. If your client caps the total time instead (most SDK timeout settings work that way), set it close to 600 seconds.

What should I do about a 504 "Upstream did not respond in time"?

This response means the model produced no answer in time. It comes with a retry-after: 5 header and the request is not charged. Waiting a few seconds and sending the same request again is usually enough.

If the same request keeps getting 504, make it smaller. A very long chat history or megabytes of attached files make the model take longer to reach its first token. Sending only the context you actually need cuts both the wait and the cost.

Do I pay if I'm the one who drops the connection?

When your client times out and closes the connection, the model may already have done work. If the stream had started, the input processed and the output produced up to that point are charged. A request abandoned before any answer arrived is free on Wiozen's own models such as wiozen-anthropic; on the Claude, Gemini and DeepSeek models, the tokens reported as processed up to that point are charged.

That's why a short client timeout combined with constant retries can get expensive: every attempt processes the input again. Give it a long timeout, turn streaming on, and don't send the same request twice at once. How the charge is worked out is in the API cost guide.

Frequently asked questions

To stream, add stream: true to your request. Every parameter you can use is in the docs.

Create your account

More guides: How can you use an AI API from Turkey without a credit card? · How do you connect an OpenAI-compatible API to a tool you already use? · How do you calculate the cost of an AI API? · What should I do if my AI API key has leaked? · How do you generate images with an AI API, and what does one image cost? · Why does my AI API return a 429 error, and how do I fix it? · Anthropic Messages API or OpenAI Chat Completions: which one should I use? · What is a context window, and what is a 1 million token context good for? · Which AI model should I choose: deciding by price, context and quality · Why does a reasoning model return an empty answer, and why are thinking tokens charged? · Why is my API balance running out faster than expected? Tracking spend request by request · How do you convert a web page to Markdown to feed it to an AI model? · How do you get AI to write and publish articles in WordPress automatically? · How do you find blog topics from RSS feeds and Google Trends? · Does AI-written content show up on Google and in AI answers? · How do you get internal links into AI-written WordPress articles? · How do you get AI to generate a different featured image for every WordPress post? · How do you write a content brief when an AI writes your blog posts? · What should you watch for when an AI writes recipes for your food blog? · How do you fix errors when connecting a WordPress plugin to your AI API account? · Why does my AI article bot stop halfway in WordPress? "Run is stuck" and LiteSpeed · Why are my scheduled WordPress posts late? WP-Cron and a real server cron · How do I get an AI article bot to write blog posts in English or another language? · What does "Content Bot cannot generate articles" mean? Reading the bot's log