Test Data Design
Prerequisites: You should already understand Boundary Value Analysis, Equivalence Partitioning, and Requirement Traceability Matrix (RTM). Leads to: After this, you'll be ready for Exploratory Testing Fundamentals.
A perfectly designed test case, executed against sloppy or unrealistic data, produces false confidence — the test technically ran, but it never actually exercised the risk it was meant to. Test Data Design is the discipline of choosing data as deliberately as the test case itself, applying the same rigor Section 2's techniques taught for what to test to the separate, often under-considered question of what data to test it with.
Why This Matters
A team using whatever data happens to be lying around. A QA team at AtlasBank tests a new account-statement export feature using whatever test accounts already exist in the staging environment — accounts with tidy, simple transaction histories, mostly round numbers, no unusual characters in payee names, nothing older than a few months. Every test passes. The feature ships. Within weeks, real customer complaints reveal the export silently truncates statements for accounts with more than 500 transactions, and fails entirely for payee names containing an ampersand. Neither condition existed anywhere in the staging environment's convenient, accidentally-narrow set of test accounts.
A team designing test data deliberately. A different team, before testing the same feature, deliberately constructs test data covering specific conditions: an account with a very high transaction count (deliberately exceeding what "normal" staging data happens to have), payee names with special characters, and a genuinely old account with years of history. Testing against this deliberately constructed data catches both defects before release — not because the second team got lucky, but because they treated test data as something to design, not something to inherit from whatever already existed.
Both teams applied the same test case rigor. Only one of them applied the same rigor to the data behind it.
What Test Data Design Covers
Effective test data spans several distinct categories, each targeting a different kind of risk:
| Category | What It Means | Example (AtlasBank Loan Application) |
|---|---|---|
| Valid data | Data representing normal, expected, correct input | Applicant age 35, income $65,000, requested loan $20,000 |
| Invalid data | Data that should be rejected, testing that validation actually works | Applicant age −5, income "not a number," requested loan $0 |
| Boundary data | Data at the exact edges of a valid range, per Boundary Value Analysis | Applicant age exactly 18 (minimum) and exactly 75 (maximum) |
| Null data | Missing or absent values in optional and required fields | Middle name left blank (should be allowed); annual income left blank (should be rejected, if required) |
| Duplicate data | The same value appearing where uniqueness is expected, or repeated submissions | Two loan applications submitted with the identical applicant ID within the same session |
| Production-like data | Data reflecting the real scale, messiness, and variety production data actually has | An account with 800+ transactions, names with apostrophes and hyphens, realistic date ranges spanning years |
The opening scenario's two missed defects map directly onto two of these categories: the transaction-count failure was a production-like-data gap (staging data was too small and clean), and the ampersand failure was a data variety gap (special characters weren't represented). Neither would have been caught by valid, invalid, or boundary data alone — they required deliberately reaching for realistic, messy data on purpose.
A beginner tests with whatever data is convenient — usually whatever already exists in a shared test environment. A senior tester treats test data as a deliberate design decision, asking specifically: does this data represent the real scale and messiness production data has, not just the happy-path shape that's easiest to set up? The habit of asking "would this data actually exist in production" before trusting a test result is what catches the class of defect this module's opening scenario describes.
Privacy Considerations
Test data must never be real customer or patient data, even when real production data would technically make the most realistic test data possible. This isn't a minor compliance footnote — using real customer data in a test environment is a genuine privacy and regulatory risk (GDPR, HIPAA, and similar regulations all have direct requirements here), independent of how much it might improve test realism. The right approach is synthetic data that's realistic without being real: generated data that mimics production's real scale, variety, and messiness (the exact gap that caused the opening scenario's defects) without ever containing an actual customer's or patient's real information. Data masking or anonymization of a copy of production data is a common middle ground, but even then, genuine anonymization is harder to get right than it looks — a poorly anonymized dataset can sometimes still be re-identified, which is its own real risk.
Data Maintenance
Test data isn't a one-time setup — it degrades in value over time exactly like a stale test case does. Data seeded once and never revisited stops reflecting current business rules (a boundary that was 60 in a requirement two years ago might be 65 now), and static, unchanging test accounts stop resembling production's genuinely evolving messiness. Treat test data with the same ownership discipline Test Case Organization and Naming applies to test cases themselves — someone should be responsible for noticing when test data has gone stale relative to current rules, not leaving it to accumulate quietly incorrect.
When NOT to Use Extensive Test Data Design
Full, deliberate test data design across every category isn't always the right investment:
- Very small prototypes or proof-of-concept work: if the goal is validating a rough idea before real investment, a couple of convenient, hand-picked values are often entirely sufficient — building out a full valid/invalid/boundary/null/duplicate/production-like dataset for throwaway exploratory work is effort spent on the wrong thing at that stage
- One-off internal utilities with no real users or compliance exposure: the cost of a data-related defect here is low enough that convenient, ad hoc data is a reasonable, deliberate choice, not a shortcut to feel guilty about
- Features already covered by a well-maintained, shared test data set: reinventing test data from scratch when a team already has good, current, deliberately designed data for the relevant domain is wasted effort — the point is designing data deliberately once and reusing it, not redesigning it every single time
The discipline this module teaches is about deliberate choice, not maximum effort applied uniformly everywhere — matching investment to risk and context is the same theme this entire learning path returns to repeatedly.
How This Works on Two Real Projects
Banking (AtlasBank): Testing a new fraud-detection rule for the Mobile App, a tester deliberately constructs test data across every category: valid transactions (should not trigger fraud review), a transaction at the exact dollar-amount threshold that should trigger review (boundary data), a duplicate transaction submitted twice in rapid succession (testing whether the system correctly treats it as one event, not two separate suspicious transactions), and — critically — production-like data reflecting a genuinely old account with years of varied transaction history, since the fraud rule's logic depends partly on comparing a transaction against historical patterns. Testing against a brand-new, empty test account (the "convenient" default) would never have exercised that historical-comparison logic at all; the deliberately aged, realistic account is what actually tests the rule as it would behave for a real customer.
Healthcare: Testing a patient-matching feature (confirming two records referring to the same patient get correctly linked), a tester deliberately includes production-like name variety: hyphenated surnames, patients with the exact same name but different dates of birth (testing that the system doesn't incorrectly merge two different people), and a patient record with a null middle name alongside one with a full middle name for the same person (testing that a missing optional field doesn't incorrectly prevent a correct match). None of this data is real patient information — it's synthetically constructed specifically to mimic the real variety and edge cases patient records actually contain, without the genuine privacy risk real records would carry.
Both examples show the same underlying discipline: test data chosen specifically to exercise a real risk, not chosen because it happened to be the easiest data available.
Common Mistakes
Mistake 1: Using whatever data is already conveniently available instead of designing it deliberately. As the opening scenario shows, convenient staging data is often too clean and too small to actually exercise the risks that matter most in production.
Mistake 2: Using real customer or patient data in a test environment. This is a genuine privacy and regulatory risk, not a shortcut — synthetic, realistic data is the correct alternative, even when it takes more effort to construct.
Mistake 3: Letting test data go stale as business rules change. A boundary value in test data that no longer matches the current requirement silently undermines exactly the kind of boundary testing this data was meant to support.
Mistake 4: Testing only with "clean" data and never with production-like messiness. The transaction-count and special-character defects in this module's opening scenario both specifically required messy, realistic data to catch — clean, convenient data structurally cannot reveal them.
Best Practices
Practice 1: Design test data across all six categories deliberately, not just valid and invalid. Boundary, null, duplicate, and production-like data each target a distinct, real class of defect the others don't.
Practice 2: Use synthetic data that mimics production's real scale and messiness, never real customer data. This satisfies both the realism goal and the privacy requirement at once — the two aren't actually in tension if synthetic data is constructed thoughtfully.
Practice 3: Maintain test data with the same ownership discipline as test cases. Assign responsibility for noticing when test data no longer reflects current business rules, the same way Module 11 recommends for test case staleness.
Practice 4: Match the depth of test data design to the feature's actual risk. Full deliberate design across every category for high-risk, regulated, or high-scale features; a lighter touch is a reasonable, deliberate choice for low-risk, throwaway, or prototype work.
On a fintech project, a currency-conversion feature passed every test using clean, round test amounts ($100, $500, $1,000). A real production incident revealed a rounding defect that only appeared with amounts carrying more decimal precision, of the kind real transactions actually have far more often than round numbers do (a real purchase converted from a foreign receipt, for instance, rarely lands on an exact round number). The team's test data had been unconsciously biased toward "clean" numbers simply because they were easier to read and reason about while writing test cases — an unintentional but real gap between what was convenient to write and what production data actually looks like.
Mini Challenge
Scenario: A user-registration feature at AtlasBank has a "date of birth" field, used to confirm the applicant is at least 18 years old.
Your task: Design a small test dataset covering at least four of this module's six categories (valid, invalid, boundary, null, duplicate, production-like) for this field. For each data point, state which category it belongs to and what specific risk it's designed to catch.
Key Takeaways
- Test data deserves the same deliberate design rigor as the test case it supports — convenient, already-available data often can't exercise the risks that matter most.
- Six categories — valid, invalid, boundary, null, duplicate, and production-like — each target a distinct class of defect.
- Real customer or patient data should never appear in a test environment; synthetic data can be realistic without carrying real privacy risk.
- Test data needs ongoing maintenance, with clear ownership, the same way test cases do — stale test data silently undermines the tests it's meant to support.
What You Just Learned
- The six categories of deliberate test data, and the specific class of defect each one targets
- Why production-like, realistically messy data catches defects clean, convenient data structurally cannot
- Why real customer or patient data should never be used in testing, and what synthetic data offers instead
- When lighter, more convenient test data is a reasonable, deliberate choice rather than a shortcut
Downloadable Resources (Coming Soon): A ready-to-use Test Data Template, structured around this module's six categories, is planned — tracked in NOT_NOW.md.
Next: Exploratory Testing Fundamentals
Related Topics
- Boundary Value Analysis — Boundary data, applied here as its own deliberate test data category
- Requirement Traceability Matrix (RTM) — Where non-functional considerations like data privacy would also need explicit tracking
- Writing Clear Test Cases — Where specific, deliberately chosen test data becomes part of a complete, repeatable test case
Interview Questions
Q1: What categories of test data would you consider when testing a form with several input fields?
What to look for: A candidate who names multiple categories (valid, invalid, boundary, null, duplicate, production-like) rather than just "valid and invalid data" — the more complete answer shows a fuller grasp of what test data design actually covers.
Q2: Why shouldn't you use real production data for testing, even when it would be the most realistic option?
What to look for: A clear statement of the privacy and regulatory risk (GDPR, HIPAA, or similar), plus awareness that synthetic data can achieve realism without that risk — a candidate who only says "it's against policy" without explaining why hasn't fully internalized the reasoning.
Many candidates answer "what test data would you use" by naming only valid and invalid examples, stopping there. That's incomplete — it misses boundary, null, duplicate, and especially production-like data, which is often where the most damaging real-world defects hide, as this module's currency-rounding and transaction-count examples both show. A strong answer names the fuller set of categories, not just the two most obvious ones.
Q3: How would you keep test data from becoming stale as a project evolves?
What to look for: A candidate who mentions ownership and periodic review tied to requirement changes — treating test data with the same maintenance discipline as test cases — not someone who treats test data as a one-time setup task.
Section 3 Complete
You've finished Writing and Organizing Tests, the third section of Manual Testing. You now know how to turn what Section 2 taught you to find into artifacts a real team can execute, trust, and maintain:
✔ Writing Clear Test Cases — precise, atomic test cases someone else can execute correctly ✔ Test Case Organization and Naming — keeping thousands of test cases findable and maintainable at scale ✔ Requirement Traceability Matrix (RTM) — proving coverage and finding gaps, not just assuming they don't exist ✔ Test Data Design — choosing data as deliberately as the test case it supports
How they build on each other: a well-written test case (Module 10) is only maintainable at scale with real organization (Module 11); organized test cases with stable IDs are what makes an RTM's mapping reliable (Module 12); and every one of those test cases needs deliberately designed data behind it to actually exercise the risk it claims to test (Module 13). Section 2 taught you what to test; Section 3 taught you how to turn that into something a real team can execute, trust, and prove.
Technique / Artifact Selection Matrix:
| Need | Reach For | Why |
|---|---|---|
| A test idea needs to become executable | Writing Clear Test Cases | Precision and atomicity make it usable by someone besides the author |
| A suite has grown past what one person can navigate | Test Case Organization and Naming | Feature-based structure and naming keep it findable and trustworthy at scale |
| You need to prove every requirement was tested | Requirement Traceability Matrix | Turns coverage from an impression into a checkable claim |
| A test case needs realistic, risk-targeted input | Test Data Design | Convenient data often can't exercise the risk a test case is meant to catch |
Next section: Exploratory and Ad Hoc Testing, starting with Exploratory Testing Fundamentals — where structured, scripted testing gets deliberately contrasted against simultaneous, discovery-driven testing.
Section 3 Knowledge Check
Five realistic scenarios. For each, decide which artifact or practice from this section applies, and why. No answers are provided here — this is a chance to apply the section's toolkit yourself before moving on. Solutions: Section 3 Solutions.
Scenario 1: A tester has just designed a full set of test ideas for a new feature using several Section 2 techniques, and now needs to turn them into something a teammate can execute next sprint.
Scenario 2: A QA team's test suite has grown to over 2,000 test cases over three years, with no consistent naming, and a new hire can't find existing coverage for a feature they're about to test.
Scenario 3: A compliance auditor asks a team to prove that a specific regulatory requirement was tested in each of the last three releases.
Scenario 4: A test case for a currency-conversion feature has only ever been run with round, clean dollar amounts, and a tester wants to know if that's actually sufficient.
Scenario 5: A requirement changes mid-project, and a tester needs to know exactly which existing test cases might now be testing an outdated rule.
Glossary
Boundary Data: Test data positioned at the exact edges of a valid range, applying Boundary Value Analysis specifically to data construction.
Production-Like Data: Test data reflecting the real scale, variety, and messiness of actual production data, as opposed to clean, convenient, artificially tidy test data.
Synthetic Data: Artificially generated data designed to mimic real data's realistic characteristics without containing any actual customer or patient information.
Data Masking: The process of obscuring or anonymizing real data (often a copy of production data) to reduce privacy risk while attempting to preserve realistic structure.
Quick Revision
Remember these five points:
✓ Test data deserves the same deliberate design as the test case it supports — convenient data often can't exercise the risks that matter most.
✓ Six categories matter: valid, invalid, boundary, null, duplicate, and production-like data.
✓ Production-like, realistically messy data catches defects clean data structurally cannot — as with rounding errors and special-character handling.
✓ Never use real customer or patient data in testing — synthetic data can be realistic without the privacy risk.
✓ Test data needs ongoing maintenance and clear ownership, the same way test cases do.