OpenAI details metrics to measure political bias in LLMs

OpenAI today released a new framework for defining and measuring political bias in its large language models (LLMs) in a move to increase public confidence in the technology. The ChatGPT parent acknowledged that while bias is rare, emotionally charged or adversarial prompts can push models into unintended, non-objective responses.

The new evaluation is designed to mirror real-world usage and stress-test the models' ability to remain neutral. In a blog titled Defining and evaluating political bias in LLMs, the company stated that it is composed of approximately 500 prompts spanning 100 topics with varying political slants and measures five nuanced axes of bias, including personal political expression and asymmetric coverage.

Key findings and model improvements

Based on this evaluation, the company found that its models remain near-objective on neutral or slightly slanted prompts, exhibiting moderate bias only in response to challenging, emotionally charged prompts. When bias does emerge, it most often involves the model expressing personal opinions, providing asymmetric coverage, or using charged language that escalates the user's slant.

The latest models, GPT-5 instant and GPT-5 thinking, showed significantly improved performance and greater robustness to charged prompts, reducing bias by 30% compared to prior models.

The company estimates that a strict evaluation rubric applied to a representative sample of production traffic indicates less than 0.01% of all model responses exhibit signs of political bias.

Evaluation scope and methodology

The company built its evaluation to address the limitations of existing benchmarks, which often rely on multiple-choice questions that cover only a narrow slice of everyday use. It focused on the AI assistant's text-based responses, which represent the majority of user interaction.

By sharing its definitions and evaluation methods, the company seems to be getting ahead of potential scrutiny, presenting itself as a more transparent and accountable model steward.