# Batch Inference

Batch inference takes many chat-completion requests in one call and answers
them later. You submit the whole job, check on it from time to time, and page
through the results once it has finished. FreeInference runs the requests in
the background, a few at a time per model, and gives way to interactive
traffic.

The API has the shape of OpenAI's Batch API: the same batch objects, statuses
and result lines. The difference is that the requests travel inline in the
create call, and the results are read a page at a time instead of downloaded as
a file.

> **Internal accounts only:** Batch inference is currently available to
> internal accounts. Creating a batch with any other key returns `403`. It may
> be opened to other account tiers later.

## When to use it

Use a batch for a large job that does not need its answers right away: an
evaluation run, labelling or summarizing a dataset, generating synthetic data.
Compared with sending the requests yourself:

- **One call submits the whole job**, up to 50,000 requests by default. You do
  not hold thousands of connections open, and you can disconnect once it is
  submitted.
- **FreeInference paces and retries the work.** Requests wait for a free slot
  instead of failing on your concurrency limit, and transient errors are
  retried for you, so you write no rate-limit or retry logic.
- **The results wait for you.** A finished batch and its results are kept for
  7 days by default.

Send the requests yourself when you need answers within seconds, need
streaming, or call an endpoint other than chat completions. A batch has no
fixed turnaround: its requests start when their models have room, and any that
have not started when its completion window closes (24 hours by default)
expire. Batch requests are not discounted either: each one counts against your
daily quota exactly as the same request sent directly would.

## How batches run

- **Nothing runs during the create call.** FreeInference checks every request,
  stores the batch and answers at once, with the status `validating`. The
  requests start in the background as soon as their models have room.
- **A few at a time, behind interactive traffic.** Each model runs only a few
  batch requests at once, and fewer while it is busy with interactive requests.
  Batches take turns, so a large batch does not hold up a smaller one submitted
  after it. Depending on how busy its models are, a batch can finish in minutes
  or take hours. Requests run in no particular order; their results are listed
  in the order you submitted them.
- **Like your own requests.** Each request is served exactly as if you had sent
  it to `/v1/chat/completions` yourself, under your account, and counts toward
  your daily quota. While it runs it holds one of your concurrent-request
  slots, so a running batch leaves fewer for your interactive work. When every
  slot is in use, the batch request waits.
- **Retried when the error may pass.** A request that fails with a timeout, a
  `429` or a `5xx` is retried with backoff, up to eight attempts in all. A
  request rejected as invalid (any other `4xx`) fails at once.
- **Checked against your key every time.** Before each request runs,
  FreeInference checks the API key that created the batch again:
  - if the key has been revoked or has expired, or the account was suspended
    or deleted, the batch fails;
  - if your daily quota is used up, your batches pause until it renews at
    00:00 UTC;
  - if your account no longer has access to batch inference, your batches wait
    until it has again.

**Keep the key you created a batch with until the batch finishes.** The
completion window keeps running while a batch is paused or waiting. And while
your daily quota is used up, every call made with your key returns `429` until
the quota renews, checking on and reading your batches included.

## Endpoints

Every endpoint is under the base URL `https://freeinference.org/v1` and takes
your FreeInference API key, as `Authorization: Bearer hyi-your-api-key` (or
`X-API-Key`; see the [API Headers Reference](api_headers.md)).

| Request | What it does |
|---|---|
| `POST /v1/batches` | [Create a batch](#create-a-batch) |
| `GET /v1/batches/{batch_id}` | [Check on a batch](#check-on-a-batch) |
| `GET /v1/batches/{batch_id}/results` | [Read its results](#read-the-results), a page at a time |
| `POST /v1/batches/{batch_id}/cancel` | [Cancel it](#cancel-a-batch) |
| `DELETE /v1/batches/{batch_id}` | [Delete it](#delete-a-batch), with its results |
| `GET /v1/batches` | [List your batches](#list-your-batches) |

The examples read your key from an environment variable:

```bash
export FREEINFERENCE_API_KEY="hyi-your-api-key"
```

## Create a batch

`POST /v1/batches` with the requests inline:

```bash
curl https://freeinference.org/v1/batches \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "requests": [
      {
        "custom_id": "q-1",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
          "model": "glm-5.3-flash",
          "messages": [{"role": "user", "content": "Summarize the plot of Hamlet in two sentences."}]
        }
      },
      {
        "custom_id": "q-2",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
          "model": "qwen3.6-35b",
          "messages": [{"role": "user", "content": "Translate \"good morning\" into French."}],
          "max_tokens": 50
        }
      }
    ],
    "metadata": {"project": "eval-42"}
  }'
```

| Field | Required | Meaning |
|---|---|---|
| `requests` | Yes | The requests, at least one |
| `requests[].custom_id` | Yes | Your name for the request: 1–512 characters, unique within the batch. Its result carries it back |
| `requests[].method` | No | `POST`, the default and the only method accepted |
| `requests[].url` | No | `/v1/chat/completions`, the default and the only endpoint a batch can call |
| `requests[].body` | Yes | A chat-completion request body, exactly as you would send it to `/v1/chat/completions`. `stream` must not be `true` |
| `metadata` | No | Up to 16 string key-value pairs for your own use; keys up to 64 characters, values up to 512 |

One batch can use several models. Name each one by a model ID from
[Available Models](models.md): auto models (`auto`, `auto-<tier>`) cannot be
used in a batch.

FreeInference checks every request before it stores anything, and refuses the
whole batch with `400` if any request is invalid. The error message names the
request at fault, for example
`requests[1] (custom_id 'q-2'): Model 'no-such-model' not found`. As elsewhere
in the API, a model your key may not use is reported exactly like one that does
not exist.

The answer is the new batch:

```json
{
  "id": "batch_6f1c0a7e2b8d4e55a1c3f9d2e4b6a8c0",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "errors": null,
  "input_file_id": null,
  "completion_window": "24h",
  "status": "validating",
  "output_file_id": null,
  "error_file_id": null,
  "created_at": 1792929600,
  "in_progress_at": null,
  "expires_at": 1793016000,
  "finalizing_at": null,
  "completed_at": null,
  "failed_at": null,
  "expired_at": null,
  "cancelling_at": null,
  "cancelled_at": null,
  "request_counts": {"total": 2, "completed": 0, "failed": 0},
  "metadata": {"project": "eval-42"},
  "cost_usd": 0.0,
  "error": null
}
```

The other endpoints take the batch's `id`; the examples below read it from
`$BATCH_ID`. Timestamps are Unix seconds, and `expires_at` is when the
completion window closes. The file fields and `errors` are always `null`. Two
fields are not in OpenAI's batch object: `cost_usd`, what the batch's finished
requests have counted against your daily quota so far (an accounting figure,
not a bill), and `error`, which says why the whole batch failed.

## Check on a batch

`GET /v1/batches/{batch_id}` returns the batch object again:

```bash
curl https://freeinference.org/v1/batches/$BATCH_ID \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"
```

Poll it until `status` is `completed`, `failed`, `expired` or `cancelled`, the
four final statuses:

| Status | Meaning |
|---|---|
| `validating` | Stored; no request has started yet |
| `in_progress` | Requests are running |
| `completed` | Every request has finished, successfully or not. Read each one's outcome in the results |
| `failed` | The batch stopped because the key or account that created it is no longer valid; its `error` says so (`owner_unauthorized`). Requests that had not started fail with `batch_failed` |
| `expired` | The completion window closed before every request had started. Those requests expire with `batch_expired`; the others keep their results |
| `cancelling` | You cancelled the batch while requests were running; they are finishing |
| `cancelled` | Cancelled. Requests that had not started never will |

`request_counts.total` is the number of requests in the batch.
`request_counts.completed` counts the requests that succeeded, and
`request_counts.failed` those that failed, including the ones a failed or
expired batch never ran. Cancelled requests are in neither.

## Read the results

`GET /v1/batches/{batch_id}/results` pages through the batch's requests in the
order you submitted them, each with its outcome:

```bash
curl "https://freeinference.org/v1/batches/$BATCH_ID/results?limit=100" \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"
```

```json
{
  "object": "list",
  "data": [
    {
      "id": "batch_req_1041",
      "custom_id": "q-1",
      "status": "completed",
      "response": {
        "status_code": 200,
        "request_id": "req_2c9f6a1d8e7b4c03a5f1e2d3c4b5a697",
        "body": {
          "id": "chatcmpl-9b1e",
          "object": "chat.completion",
          "model": "glm-5.3-flash",
          "choices": [
            {
              "index": 0,
              "message": {"role": "assistant", "content": "Prince Hamlet ..."},
              "finish_reason": "stop"
            }
          ],
          "usage": {"prompt_tokens": 16, "completion_tokens": 120, "total_tokens": 136}
        }
      },
      "error": null
    },
    {
      "id": "batch_req_1042",
      "custom_id": "q-2",
      "status": "failed",
      "response": null,
      "error": {"code": "invalid_request", "message": "...", "status": 400}
    }
  ],
  "has_more": false,
  "first_id": "batch_req_1041",
  "last_id": "batch_req_1042"
}
```

`limit` is 1–1000 (default 100). When `has_more` is `true`, ask for the next
page with `after` set to this page's `last_id`:

```bash
curl "https://freeinference.org/v1/batches/$BATCH_ID/results?limit=100&after=batch_req_1042" \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"
```

Every request of the batch is listed, finished or not. A request that is still
`queued` or `running` has no `response` yet; if it has already failed once and
is being retried, its `error` shows what went wrong last time. So read the
results once the batch has reached a final status, or read them again later.

| Result `status` | Meaning |
|---|---|
| `queued` | Waiting to run, or waiting to be retried |
| `running` | Running now |
| `completed` | Succeeded. `response.body` is the chat completion, as `/v1/chat/completions` would have returned it |
| `failed` | Failed, or never ran because the batch failed. `error` says why |
| `cancelled` | Never ran: the batch was cancelled |
| `expired` | Never ran: the completion window closed first |

An `error` has a `code` and a `message`, plus the HTTP `status` behind it when
there was one:

| `error.code` | Meaning |
|---|---|
| `invalid_request` | The request was rejected as invalid (`400`, `422`) |
| `model_not_found` | The model is gone, or your key may no longer use it (`404`) |
| `forbidden` | The request was refused (`403`) |
| `rate_limited` | Still rate-limited after every retry (`429`) |
| `timeout` | Still timing out after every retry (`504`) |
| `service_unavailable` | No provider could serve it after every retry (`503`) |
| `upstream_error` | Another server error, after every retry |
| `batch_failed` | Not run: the batch failed |
| `batch_expired` | Not run: the completion window closed first |

## Python example

The OpenAI SDK's batch helpers expect an uploaded input file, so call the
endpoints over plain HTTP. This script submits a batch, waits for it to finish
and collects every result, using [`requests`](https://requests.readthedocs.io/):

```python
import os
import time

import requests

API = "https://freeinference.org/v1"
session = requests.Session()
session.headers["Authorization"] = f"Bearer {os.environ['FREEINFERENCE_API_KEY']}"


def call(method, path, **kwargs):
    response = session.request(method, API + path, timeout=120, **kwargs)
    response.raise_for_status()
    return response.json()


prompts = ["Summarize the plot of Hamlet in two sentences.", "Name three prime numbers."]
batch = call(
    "POST",
    "/batches",
    json={
        "requests": [
            {
                "custom_id": f"q-{i}",
                "method": "POST",
                "url": "/v1/chat/completions",
                "body": {
                    "model": "glm-5.3-flash",
                    "messages": [{"role": "user", "content": prompt}],
                },
            }
            for i, prompt in enumerate(prompts)
        ]
    },
)

# Wait for a final status.
while batch["status"] not in ("completed", "failed", "expired", "cancelled"):
    time.sleep(30)
    batch = call("GET", f"/batches/{batch['id']}")

# Page through every result.
results, after = [], None
while True:
    params = {"limit": 1000}
    if after:
        params["after"] = after
    page = call("GET", f"/batches/{batch['id']}/results", params=params)
    results.extend(page["data"])
    if not page["has_more"]:
        break
    after = page["last_id"]

for result in results:
    if result["status"] == "completed":
        print(result["custom_id"], result["response"]["body"]["choices"][0]["message"]["content"])
    else:
        print(result["custom_id"], result["status"], result["error"])
```

## Cancel a batch

`POST /v1/batches/{batch_id}/cancel` stops a batch and returns it:

```bash
curl -X POST https://freeinference.org/v1/batches/$BATCH_ID/cancel \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"
```

Requests that have not started are cancelled; running ones finish and keep
their results. The batch is `cancelling` until they do, then `cancelled`.
Cancelling a batch that has already finished returns it unchanged.

## Delete a batch

`DELETE /v1/batches/{batch_id}` deletes a batch with its requests and results:

```bash
curl -X DELETE https://freeinference.org/v1/batches/$BATCH_ID \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"
```

```json
{"id": "batch_6f1c0a7e2b8d4e55a1c3f9d2e4b6a8c0", "object": "batch.deleted", "deleted": true}
```

A batch can be deleted in any status, but not while one of its requests is
running: the answer is then `409`. Cancel the batch, wait for its running
requests to finish, then delete it.

FreeInference also deletes a finished batch on its own, 7 days after it
finished by default, so collect its results before then.

## List your batches

`GET /v1/batches` lists your batches, newest first:

```bash
curl "https://freeinference.org/v1/batches?limit=20" \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"
```

`limit` is 1–100 (default 20). The answer has the shape of a page of results,
with batch objects in `data`; for the next page, pass `after` set to the
previous page's `last_id`.

Reading, listing, cancelling and deleting your batches work even if your
account can no longer create them, so you can always collect and remove what
you submitted. A batch that belongs to another account reads as not found.

## Limits

| Limit | Default |
|---|---|
| Requests in one batch | 50,000 |
| Size of the create request's body | 32 MiB |
| Unfinished batches per account (`validating`, `in_progress` or `cancelling`) | 10 |
| Completion window: how long a batch's requests have to start | 24 hours |
| How long a finished batch and its results are kept | 7 days |

These are the defaults, and FreeInference may change them. Split a larger job
into several batches.

## Errors

| Status | When |
|---|---|
| `400` | Creating: the body is not valid JSON or not a batch; a request is invalid, uses an unknown, unavailable or auto model, or repeats a `custom_id`; or the batch holds too many requests. Reading results: `after` is not a result ID such as `batch_req_42` |
| `401` | The API key is missing, invalid or expired |
| `403` | Creating: your account may not create batches (internal accounts only, for now) |
| `404` | The batch does not exist, was deleted, or belongs to another account; or batch inference is unavailable at the moment. The message says which |
| `409` | Deleting: one of the batch's requests is still running. Cancel the batch and wait for it to finish first |
| `413` | Creating: the body is larger than the size limit |
| `429` | Creating: you already have the most unfinished batches allowed. Any call: your daily quota is used up until it renews at 00:00 UTC |

## Coming from OpenAI's Batch API

- Put the requests in the create call's `requests` array, each entry shaped
  like a line of an OpenAI batch input file. There is no file upload: no
  `input_file_id`, and no `output_file_id` or `error_file_id` to download.
  Read the results from `/v1/batches/{batch_id}/results`.
- The OpenAI SDK's `batches.create` needs an uploaded file, so it cannot create
  a batch here; use plain HTTP, as in the [Python example](#python-example).
- The completion window is FreeInference's (24 hours by default); a
  `completion_window` you send is ignored.
- Only `/v1/chat/completions` can be batched, and without streaming.
- The status `finalizing` never occurs, and `DELETE /v1/batches/{batch_id}`
  exists.
- There is no batch discount: batch requests count against your quota like any
  other.

## See also

- [Available Models](models.md) — the model IDs a batch can use
- [API Headers Reference](api_headers.md) — authentication headers
- [IDE & Coding Agent Integrations](integrations.md) — send chat requests
  directly with the OpenAI SDK or curl
- [Quick Start](quickstart.md) — create an API key