Skip to main content

Common Mistakes in AI for QA

Prerequisites: You should already understand every module in Sections 1–5 of this path. Leads to: After this, you'll be ready for the AI for QA Capstone.

Every module in this path named mistakes specific to its own concern. This module steps back and asks which mistakes recur across concerns, visible only once you've seen the whole path. These aren't new rules — they're the handful of underlying habits that, left unchecked, undermine almost any concern this path taught.

Pattern 1: Treating Review as a Formality Instead of a Genuine Checkpoint​

A team has an AI-review policy, but the actual review is a fast skim confirming output looks coherent, not a check against a specific, defined target.

How this leads to an escaped defect: Responsible AI Usage and Human-in-the-Loop QA's own opening example — a batch of AI-drafted KYC test cases, reviewed only for fluency, shipped with confidently invented validation rules the actual requirement never specified.

Prevention: define a specific verification target for every category of AI-assisted work, and make checking against it the actual review action — not "does this look right."

Pattern 2: Trusting AI Output's Confident Tone as Evidence of Correctness​

A team judges AI-generated content's trustworthiness by how fluent, detailed, or assured it sounds.

How this leads to an escaped defect: Reviewing AI Output and Recognizing Hallucinations's own fabricated-endpoint example — a hallucinated API call, written in the exact confident style of real endpoints, passed a plausibility-based review that a direct-verification check would have caught in under a minute.

Prevention: verify every specific, checkable claim directly against its real source — never judge correctness by tone or fluency.

Pattern 3: Accepting an AI-Drafted Test Case Set Because It Covers the Requirement's Obvious Shape​

A team treats an AI-drafted test case set as complete once it covers a requirement's clearly-stated scenarios.

How this leads to an escaped defect: AI-Assisted Test Case Generation's own $15,000 transfer-limit example — an AI draft covering "under," "over," and "at" the limit missed the actual boundary-adjacent values where a real off-by-one defect concentrated.

Prevention: apply Boundary Value Analysis and Equivalence Partitioning explicitly and systematically to every AI-drafted test case set, not a general completeness read.

Pattern 4: Testing an AI-Generated Response With an Exact-Match Assertion​

A team tests AI-generated feature output the same way they'd test any deterministic feature — a fixed expected value, pass or fail.

How this leads to an escaped defect: Testing AI-Driven Features's own loan-status example — a test that "failed" repeatedly due to legitimate response-phrasing variation, confusing the team about whether a real defect existed, when the actual problem was the test method itself.

Prevention: separate an AI feature's deterministic layer (exact-match testable) from its AI-generation layer (rubric-evaluated), and apply the correct method to each.

Pattern 5: Sending Unsafe Data to an External AI Tool​

A team pastes real customer data, credentials, or proprietary code into an external AI tool as a routine, unquestioned part of getting help.

How this leads to an escaped defect: AI Security and Privacy Awareness's own opening example — a real customer transaction record sent to a public AI tool for debugging help, a genuine data-exposure incident that synthetic test data would have avoided entirely, with equal debugging value.

Prevention: default to synthetic or sanitized data for any AI-assisted task; treat "can I send this?" as a real question every time, not an assumed-safe routine action.

Pattern 6: Knowing the Right Standards but Applying Them Inconsistently​

A team's individual testers each know the correct review standards, but nothing structurally ensures any given artifact actually gets the right one.

How this leads to an escaped defect: Human Review Workflows and AI Quality Assurance's own opening example — sound individual knowledge, applied inconsistently across a team, meant review quality depended on luck rather than a reliable process.

Prevention: build one unified, lightweight workflow routing every AI-assisted artifact type to its correct review standard automatically, logged every time — not reliant on individual memory.

Every pattern above has the same underlying shape: a shortcut that feels reasonable in the specific moment it's taken — trusting AI output because reviewing it deeply takes real time, pasting real data because it's faster than generating synthetic data, skipping systematic technique review because a draft "looks thorough enough." Recognizing the temptation itself as a signal worth pausing on, not just knowing the list of patterns, is what actually prevents them from recurring — the same closing lesson every prior Common Mistakes module in TestAtlas has reached.

Key Takeaways​

  • Six recurring patterns cut across this path's individual modules: formality review, trusting confident tone, accepting obvious-shape test coverage, exact-match testing of AI output, unsafe data sharing, and inconsistent standard application.
  • Each pattern is a shortcut that feels reasonable in the specific moment it's taken — recognizing the moment of temptation is the actual skill, more than memorizing the list.
  • Every pattern traces back to a real, worked defect example from earlier in this path — these are patterns that have already produced real, specific consequences throughout this path's own teaching examples.
  • Prevention in every case is a specific, learnable habit, not a vague call for more thoroughness.

What You Just Learned​

  • Six mistake patterns that recur across this path's individual concerns, not within any single one
  • How each pattern, left unchecked, produced a real, specific defect in this path's own worked examples
  • The specific prevention habit for each pattern
  • Why recognizing the moment of temptation to take a shortcut matters more than memorizing a list of mistakes

Next: AI for QA Capstone

Interview Questions​

Q1: What's a common mistake you've seen (or made) when using AI in QA work, and how would you prevent it?

What to look for: A specific, real pattern (not a generic "not being careful enough") with a concrete prevention habit attached — ideally one of this module's six, explained in the candidate's own words, showing genuine understanding rather than a memorized list.

Q2: Why might a team that knows all the right AI-review practices still ship defects that those practices should have caught?

What to look for: A candidate who directly names Pattern 6 — that knowing a standard and having it reliably, consistently applied across an entire team are different things, and that a structural workflow, not just individual knowledge, closes that gap.

Glossary​

Escaped Defect: A defect that reached production despite testing having occurred, typically traceable to a specific gap in test design or execution rather than an absence of testing effort.

Quick Revision​

Remember these five points:

✓ Define a specific verification target for every review — a formality check with no target catches almost nothing.

✓ Verify specific, checkable claims directly — never trust AI output's confident tone as evidence of correctness.

✓ Apply BVA/Equivalence Partitioning explicitly to AI-drafted test cases — obvious-shape coverage isn't real boundary coverage.

✓ Separate an AI feature's deterministic layer from its AI-generation layer — apply exact-match and rubric evaluation to the right one.

✓ Default to synthetic/sanitized data for AI tools, and build one consistent, logged workflow — don't rely on individual memory.