Skip to main content

Applying AI for QA: AtlasBank AI Support Assistant Validation

Prerequisites: You should already understand every module in Sections 1–4 of this path. Leads to: After this, you'll be ready for Applying AI for QA: AI-Assisted End-to-End QA Workflow.

Every module so far taught one concern at a time — a rubric dimension, a defect class, a governance element. This module combines Section 3's full toolkit against one real, integrated target: the AtlasBank AI Support Assistant, tested end to end within its documented six-category scope.

The Feature: AtlasBank AI Support Assistant, Full Validation Pass​

A complete validation pass across all six of the Assistant's documented categories — transaction questions, card support, loan FAQs, KYC guidance, account information, payment help — combining the deterministic/AI-quality distinction, prompt testing, rubric evaluation, and the four named defect classes from Section 3.

Deterministic and AI-Quality Layers Together​

Applying Testing AI-Driven Features: each of the six categories is tested on two tracks. The deterministic track (does the Assistant call the right API, retrieve the right record) is verified with standard exact-match techniques across all six categories cleanly. The AI-quality track requires the rest of this module's work.

Prompt Variation and Rubric Scoring Across All Six Categories​

Applying Prompt Testing and Evaluation: each category gets a representative question tested across formal, informal, typo-containing, and ambiguous phrasing, scored against the four-dimension rubric (accuracy, completeness, relevance, tone).

A real finding surfaces here: payment-help category responses score consistently lower on completeness specifically for ambiguous phrasing — a customer asking generally about "payment problems" without specifying whether they mean a failed transfer, a declined card payment, or a missed loan payment receives a response addressing only one of the three possibilities, chosen seemingly at random rather than asking a clarifying question. No other category shows this specific pattern as strongly, since payment-help genuinely spans the widest range of underlying causes behind a similarly-phrased ambiguous question.

Hallucination, Bias, Safety, and Reliability Across All Six Categories​

Applying Hallucinations, Bias, Safety, and Reliability: grounding tests (known-unavailable data), framing-consistency tests, scope-boundary tests (a request just outside each category's edge), and reliability tests (repeated identical queries) are run systematically across all six categories, not just one.

A second real finding surfaces here: the safety/scope-boundary test reveals that a request blending a legitimate KYC question with an adjacent, out-of-scope request (asking about KYC document requirements and general account-opening eligibility for a new product line in the same message) is answered in full, including the out-of-scope portion — the Assistant doesn't recognize the mixed-scope request as needing to decline part of it, a gap the six categories' individual-question testing alone hadn't surfaced, since a genuinely mixed, partially-out-of-scope message is a distinct condition from either a fully in-scope or fully out-of-scope one.

What This Application Module Shows​

Two real, genuinely distinct findings — a completeness gap specific to payment-help's inherent ambiguity, and a scope-boundary gap specific to mixed in-scope/out-of-scope requests — were found by applying Section 3's full toolkit systematically across all six categories, not by testing any single category or technique in isolation. Both are exactly the kind of finding this path's central theme predicts: AI-generated content needs deliberate, structured testing precisely because its failure modes don't announce themselves the way a deterministic defect does.

Common Mistakes​

Mistake 1: Testing only one or two of the six documented categories thoroughly, assuming the others behave similarly. The payment-help completeness gap was specific to that category's particular ambiguity profile — a pattern that wouldn't have generalized from testing only, say, card support.

Mistake 2: Testing scope boundaries only with fully out-of-scope requests, not mixed requests. The mixed-scope gap specifically required testing a message blending an in-scope and out-of-scope element together — a distinct condition from either extreme tested alone.

Mistake 3: Treating a full validation pass as complete after running the deterministic-layer checks alone. Both of this module's real findings live entirely in the AI-quality layer — a deterministic-only pass would have reported the Assistant as fully validated while missing both.

Best Practices​

Practice 1: Apply Section 3's full toolkit systematically across every documented category, not just a representative sample. This is what surfaced the payment-help-specific completeness gap — a pattern invisible if only one or two categories had been tested in depth.

Practice 2: Specifically test mixed in-scope/out-of-scope requests as their own condition, not just each extreme separately. The scope-boundary gap this module found required exactly this test shape — neither a fully in-scope nor fully out-of-scope test alone would have found it.

Practice 3: Treat a validation pass as incomplete until both the deterministic and AI-quality layers have been explicitly covered. Per Testing AI-Driven Features's own foundational distinction, applied here at full-feature scale.

Key Takeaways​

  • Applying Section 3's full toolkit systematically across every documented category (not just a sample) finds category-specific patterns that spot-testing would miss.
  • Mixed in-scope/out-of-scope requests are a distinct test condition from either extreme tested alone, and need their own explicit test coverage.
  • A validation pass isn't complete until both the deterministic and AI-quality layers have been explicitly, separately covered.

What You Just Learned​

  • How to apply Section 3's full toolkit systematically across every documented category of a real AI feature
  • How a category-specific completeness gap was found in payment-help's inherently ambiguous question space
  • How a scope-boundary gap specific to mixed in-scope/out-of-scope requests was found through deliberate, targeted testing
  • Why a validation pass needs explicit coverage of both the deterministic and AI-quality layers to be considered complete

Next: Applying AI for QA: AI-Assisted End-to-End QA Workflow

Interview Questions​

Q1: When validating an AI feature with multiple distinct capabilities or categories, why might testing only one or two categories thoroughly be insufficient?

What to look for: A candidate who explains that AI quality issues can be category-specific — a pattern strong in one capability area may not generalize — and that thorough validation needs systematic coverage across every documented category, not a representative sample assumed to generalize.

Q2: Why might a request blending an in-scope and out-of-scope element need its own dedicated test, separate from testing each in isolation?

What to look for: A candidate who recognizes that a mixed request is a genuinely distinct condition — a feature correctly declining fully out-of-scope requests and correctly answering fully in-scope ones can still fail specifically on a blended request, which needs to be tested as its own case.

Glossary​

No new terms are introduced in this module — every concept used above is defined in its linked source module.

Quick Revision​

Remember these five points:

✓ Apply Section 3's full toolkit systematically across every documented category, not a representative sample.

✓ AI quality issues can be category-specific — a pattern in one capability area may not generalize to others.

✓ Test mixed in-scope/out-of-scope requests as their own distinct condition, not just each extreme separately.

✓ A validation pass isn't complete until both the deterministic and AI-quality layers are explicitly covered.

✓ Systematic, full-scope testing finds real, specific gaps that spot-testing a feature's most obvious cases would miss.