OpenAI Evals is closing

Bring your evals across.

OpenAI Evals goes read-only on 31 October 2026 and shuts down on 30 November.¹ Paste your export and Runback's importer converts it here, in your browser: every test it keeps, and every grader it can't.

Choose a file

Items file (JSONL): {"item": …} lines, or legacy {"input", "ideal"}

{"item":{"ticket_text":"My monitor won't turn on!","correct_label":"Hardware"}}
{"item":{"ticket_text":"I'm in vim and I can't quit!","correct_label":"Software"}}
{"item":{"ticket_text":"Best restaurants in Cleveland?","correct_label":"Other"}}
{"item":{"ticket_text":"My keyboard is missing keys.","correct_label":"Hardware"}}
{"item":{"ticket_text":"Excel crashes every time I open the Q3 file.","correct_label":"Software"}}
{"item":{"ticket_text":"The office printer jams on double-sided jobs.","correct_label":"Hardware"}}
{"item":{"ticket_text":"Can you recommend a good podcast?","correct_label":"Other"}}
{"item":{"ticket_text":"Zoom says my camera is in use by another app.","correct_label":"Software"}}
{"item":{"ticket_text":"My laptop battery dies after 40 minutes.","correct_label":"Hardware"}}
{"item":{"ticket_text":"The VPN client won't install on macOS.","correct_label":"Software"}}
{"item":{"ticket_text":"What time does the cafeteria open?","correct_label":"Other"}}
{"item":{"ticket_text":"The second docking-station port gives no signal.","correct_label":"Hardware"}}

Eval and run objects eval + run loaded

Eval object (JSON, from GET /v1/evals/{id})

{
  "object": "eval",
  "id": "eval_illustrative_0001",
  "name": "IT Ticket Categorization (illustrative)",
  "testing_criteria": [
    {
      "type": "string_check",
      "name": "Match output to human label",
      "input": "{{ sample.output_text }}",
      "operation": "eq",
      "reference": "{{ item.correct_label }}"
    },
    {
      "type": "string_check",
      "name": "Mentions the category",
      "input": "{{ sample.output_text }}",
      "operation": "ilike",
      "reference": "{{ item.correct_label }}"
    },
    {
      "type": "score_model",
      "name": "Helpfulness",
      "model": "gpt-4o-mini",
      "input": [
        {
          "role": "system",
          "content": "Score from 1 to 7 how well the response categorises the ticket."
        },
        {
          "role": "user",
          "content": "Ticket: {{ item.ticket_text }}\nResponse: {{ sample.output_text }}"
        }
      ],
      "range": [
        1,
        7
      ],
      "pass_threshold": 5.5
    },
    {
      "type": "text_similarity",
      "name": "Close to label",
      "input": "{{ sample.output_text }}",
      "reference": "{{ item.correct_label }}",
      "evaluation_metric": "fuzzy_match",
      "pass_threshold": 0.8
    },
    {
      "type": "python",
      "name": "Custom check",
      "source": "def grade(sample, item):\n    return 1.0"
    }
  ]
}

Run object (JSON, from GET /v1/evals/{id}/runs/{run_id})

Illustrative sample, written by Runback in the formats OpenAI documents. Not a real export. Runs in this tab with Runback's own importer. Nothing is sent or stored.

Sample · Evals API export

12 tests kept.

Of 12 rows. Each became a dataset item with checks: exact match, contains and LLM judge.

Not carried over: Close to label (text_similarity), Custom check (python). Listed on every row it applied to, never silently dropped.

My monitor won't turn on!exact match · contains · LLM judge · gpt-4.1
I'm in vim and I can't quit!exact match · contains · LLM judge · gpt-4.1
Best restaurants in Cleveland?exact match · contains · LLM judge · gpt-4.1

Start free →Already on Runback? Import now →

Exactly what moves

What carries over. What doesn't.

Each line is how the importer above behaves. Up to 10,000 rows and 4 MB per import; split anything bigger.

string_check, eqAn exact-match check against the row's reference.Carries over
string_check, like / ilikeA contains check. ilike stays case-insensitive.Carries over
score_modelAn LLM-judge check. The rubric is the grader's own messages, and the pass threshold is rescaled from its range to 0–1.Carries over
label_modelAn LLM-judge check that passes on your passing labels.Carries over
The run's prompt templateRendered per row into each item's request. developer becomes system; the run's model is kept.Carries over
Legacy { input, ideal }A starts-with pattern, as the registry's Match did. Includes is an option at import.Carries over
The original rowKept beside each item so a reviewer sees what was translated (the first 16,000 characters).Carries over
string_check ne, text_similarity, python, multiNamed on every row it applied to, along with any grader that reads something other than the model's output. Never dropped silently.Not carried over
A row with no grader Runback can runLeft out and reported. A test with no checks passes whatever the model says, which is worse than no test.Not imported
Images, files and other non-text contentThe row is reported and left out.Not imported
Past run results and scoresThe importer reads test cases and graders. Outputs from your OpenAI runs are not brought across.Not imported
The model each item replays againstMust be on your deployment's replay allowlist. This preview cannot see that list, so the app may refuse rows the preview kept.Checked at import

Every imported item lands pending review.²

Then trace one real run.

Your imported tests check a model's answers. A traced run shows what your agent saw and did at every step. Both live in the same workspace.

Start free →Already on Runback? Import now →