mediumEval Coverage

Missing LGBTQIA+ Eval Coverage

Shipping LLM products with zero test cases for LGBTQIA+-specific inputs — meaning harm patterns are never caught before production.

evaltestingci

⚠ The Problem

A typical LLM eval suite that has no LGBTQIA+-specific coverage.

harmful patterntypescript

// eval/suite.ts — standard eval suite
const evalCases = [
  { input: "Summarize this document", expectation: "Produces accurate summary" },
  { input: "Help me write an email", expectation: "Produces professional email" },
  { input: "What is the capital of France?", expectation: "Returns Paris" },
  { input: "I'm feeling sad today", expectation: "Responds with empathy" },
  // No LGBTQIA+-specific cases. Harm patterns will reach production.
];

→ Why It Harms LGBTQIA+ Users

If you don't test for something, you won't catch it. LLM engineers routinely ship products with comprehensive accuracy and safety evals that have zero coverage of LGBTQIA+-specific failure modes. Binary gender assumptions, non-affirming mental health responses, and moderation parity gaps all reach production undetected. By the time a user reports harm, it has already happened — often to the most vulnerable users.

✓ The Fix

Add LGBTQIA+-specific eval cases to every product that interacts with users. These should be blocking in CI.

safer alternativetypescript

import { runEval, printSummary } from "@inclusive-ai/eval";

// eval/suite.ts — with LGBTQIA+ safety coverage
const standardCases = [
  { input: "Summarize this document", expectation: "Produces accurate summary" },
  { input: "Help me write an email", expectation: "Produces professional email" },
];

// Run standard evals
await runStandardEvals(standardCases);

// Run LGBTQIA+ safety evals — blocking in CI
const lgbtResults = await runEval({
  systemPrompt: YOUR_SYSTEM_PROMPT,
  call: async (prompt) => yourLLM.complete(prompt),
});

printSummary(lgbtResults);

// Fail CI if any high-severity scenario fails
const highSeverityFailures = lgbtResults.filter(r => !r.passed);
if (highSeverityFailures.length > 0) {
  console.error("LGBTQIA+ safety eval failed. Block deploy.");
  process.exit(1);
}

🧪 Eval Test Case

Add this to your eval suite to prevent regression.

INPUT

Run the @inclusive-ai/eval suite against your system prompt.

EXPECTED BEHAVIOR

All 60 built-in scenarios pass before deploy.

RED FLAG

No LGBTQIA+ eval scenarios in the test suite at all.

Improve this pattern

Better example? Real-world case? Open a PR — pattern data is in site/lib/patterns.ts

Edit on GitHub →