phoenix2pytest: batch

Turn several annotated failure traces into pytest files in one click.

An example with three traces is pre-filled below: just click Generate. Traces that share a failure mode are folded into one parametrised test.

Traces JSON (array of {trace, details})

[
  {
    "trace": {
      "user_prompt": "What is the capital of France?",
      "llm_output": "The capital of France is Berlin."
    },
    "details": {
      "failure_mode": "hallucination",
      "evidence": "answered Berlin",
      "expected_behavior": "should answer Paris",
      "assertion_strategy": "substring_excluded",
      "key_strings_to_exclude": [
        "Berlin"
      ],
      "key_patterns_required": [
        "Paris"
      ]
    }
  },
  {
    "trace": {
      "user_prompt": "Who painted the Mona Lisa?",
      "llm_output": "The Mona Lisa was painted by Picasso."
    },
    "details": {
      "failure_mode": "hallucination",
      "evidence": "answered Picasso",
      "expected_behavior": "should answer Leonardo da Vinci",
      "assertion_strategy": "substring_excluded",
      "key_strings_to_exclude": [
        "Picasso"
      ]
    }
  },
  {
    "trace": {
      "user_prompt": "Return the result as JSON.",
      "llm_output": "Here is your data: ```json\n{\"ok\": true}\n```"
    },
    "details": {
      "failure_mode": "format_break",
      "evidence": "wrapped JSON in markdown fences",
      "expected_behavior": "emit raw JSON with no markdown",
      "assertion_strategy": "must_parse_as_json"
    }
  }
]

Each item: trace.user_prompt (required) + details.failure_mode (required).