# TEST-CASES.md - `02_text_cleaner.py` Test Corpus **Version**: 1.0 **Last updated**: April 29, 2026 **Companion to**: TECHNICAL.md Section 9 (script boundaries) and the per-script functional spec template introduced in TECHNICAL.md Section 10.1. ## Purpose of this document Defines the complete set of behaviors `02_text_cleaner.py` is expected to exhibit, with one test fixture per behavior. Used as: 1. The build target when porting the (currently skeleton) script to working state. 2. The pytest input set once the script ships. 3. The acceptance criteria for the GUI port (every fixture must produce its expected output through both CLI and Streamlit GUI). Each test case has an input file in `test_data/` and (where exact-diff comparison applies) an expected-output file in `expected/`. --- ## 1. Scope boundary (what 02 owns vs what it doesn't) This is the load-bearing decision. Every contested case routes back to it. **02 owns: character-level hygiene only.** - Whitespace normalization (outer trim + internal collapse for text columns). - Unicode normalization (NFC by default, NFKC opt-in). - Smart-punctuation ASCII-fication (curly quotes, em/en dash, ellipsis, primes). - Invisible / zero-width character stripping. - Control character stripping (with explicit allowlist for tab/newline inside quoted cells). - BOM detection on input, never written on output. - Line-ending normalization at the file level AND inside multi-line cells. - Optional case operations (per-column, opt-in only). **02 does NOT own:** | Concern | Owned by | |---|---| | Detecting and replacing nulls / sentinel codes | `04_missing_value_handler` | | Reformatting dates, currencies, phones, names, addresses | `03_format_standardizer` | | Outlier detection or domain-rule violations | `06_outlier_detector` | | Renaming or reordering columns | `05_column_mapper_enforcer` | | Deduplication (even though dedup normalizes internally) | `01_deduplicator` | | File encoding detection on read | The shared I/O layer in `src/core/io.py` | **Invariant 02 must preserve:** after running 02, the schema (column count, column order, row count) is unchanged. 02 changes cell *content*, never *structure*. The one nuance: a cell containing only whitespace becomes an empty string, but the cell still exists and the row is not dropped. --- ## 2. Default configuration assumed by these tests Tests assume the default config below. Any test that exercises a non-default flag explicitly says so in its description. | Setting | Default | Notes | |---|---|---| | `--trim` | on | Strip leading/trailing whitespace including Unicode whitespace (NBSP, NNBSP, ideographic space, etc.) | | `--collapse-internal` | on (text columns only) | Collapse runs of internal whitespace to a single ASCII space, ONLY in cells that don't parse as numeric, date, or phone-shaped | | `--unicode-form` | NFC | NFKC available as opt-in; folds ligatures and fullwidth | | `--smart-quotes` | on | Curly to straight, em/en dash to hyphen, ellipsis to `...`, primes to `'`/`"` | | `--strip-zero-width` | on | ZWSP, ZWJ, ZWNJ, LRM, RLM, soft hyphen, word joiner | | `--strip-controls` | on | Strip C0 (except `\t\n\r` inside quoted cells) and DEL | | `--strip-bom` | on | BOM removed on read; never written on output | | `--line-endings` | LF | File-level AND embedded-cell line endings normalized to LF | | `--case` | none | Case operations are opt-in per column | | `--fix-mojibake` | off | Logged as warning by default; opt-in repair via ftfy | | `--columns` | all | All text columns processed; `--columns name,email` restricts | **Idempotency requirement:** for any input X, `clean(clean(X)) == clean(X)`. This is a property test, not a fixture-comparison test. Every fixture below should be run through the cleaner twice and produce identical output both times. --- ## 3. Test case index | # | File | Category | What it tests | Diff-testable | |---|---|---|---|---| | 01 | `01_whitespace_basic.csv` | Whitespace | ASCII space + tab, leading/trailing/internal | Yes | | 02 | `02_whitespace_unicode.csv` | Whitespace | NBSP, narrow NBSP, ideographic, em/thin space | Yes | | 03 | `03_smart_punctuation.csv` | Punctuation | Curly quotes, em/en dash, ellipsis, primes | Yes | | 04 | `04_unicode_forms.csv` | Unicode | NFC vs NFD, ligatures, fullwidth, presentation forms | Yes | | 05 | `05_zero_width_invisible.csv` | Invisible | ZWSP, ZWJ, ZWNJ, LRM, RLM, soft hyphen | Yes | | 06 | `06_control_characters.csv` | Control | NUL, BEL, BS, VT, FF, ESC, DEL | Yes | | 07 | `07_bom_utf8.csv` | Encoding | UTF-8 BOM at file start | Yes (byte-exact) | | 08 | `08_line_endings_crlf.csv` | Line endings | All CRLF (Windows) | Yes (byte-exact) | | 09 | `09_line_endings_cr.csv` | Line endings | All CR (classic Mac) | Yes (byte-exact) | | 10 | `10_line_endings_mixed.csv` | Line endings | CRLF + LF + CR mixed in one file | Yes (byte-exact) | | 11 | `11_embedded_newlines.csv` | Line endings | Newlines inside quoted cells (preserve, normalize) | Yes | | 12 | `12_case_variations.csv` | Case | Mixed case across name/email/product columns | 3 outputs (default + 2 modes) | | 13 | `13_non_latin_scripts.csv` | Preservation | Chinese, Japanese, Arabic, Russian, emoji | Yes | | 14 | `14_mojibake.csv` | Encoding | Double-encoded UTF-8 (warn-by-default; fix opt-in) | 2 outputs (default + fixed) | | 15 | `15_whitespace_only_cells.csv` | Boundary (vs 04) | Cells containing only whitespace become empty | Yes | | 16 | `16_dirty_headers.csv` | Headers | Headers themselves have whitespace, BOM, smart quotes | Yes | | 17 | `17_preserve_intended.csv` | Negative | Things 02 must NOT touch | Yes | | 18 | `18_empty_file.csv` | Edge | Zero-byte file | Yes | | 19 | `19_headers_only.csv` | Edge | Headers but no data rows | Yes | | 20 | `20_kitchen_sink.csv` | Integration | Everything combined in one file | Yes | | 21 | `21_excel_pollution.xlsx` | Excel-specific | Multi-sheet, Alt+Enter cells, force-text, copy-paste pollution | No (manual) | --- ## 4. Per-test details ### 01 - Whitespace basic **File**: `test_data/01_whitespace_basic.csv` -> `expected/01_whitespace_basic.csv` Tests the core whitespace contract on ASCII space and tab characters. Every kind of placement: leading-only, trailing-only, both, internal-multiple, tab-padded, multiple internal multi-space runs in one cell, all of the above combined. **Expected behavior:** - Leading and trailing whitespace stripped from every cell. - Internal runs of whitespace collapsed to a single ASCII space. - Tabs treated as whitespace by both rules. **Why it matters:** This is the highest-frequency real-world pollution. Trailing-space pollution alone is what the v1.5 audit identified as the gap that motivated creating script 02 in the first place (DECISIONS.md v1.5 entry). --- ### 02 - Whitespace, Unicode **File**: `test_data/02_whitespace_unicode.csv` -> `expected/02_whitespace_unicode.csv` The whitespace pretenders. Python's `str.strip()` with no argument actually does strip these in 3.x, but a lot of cleaners written by people who were burned in 2.x explicitly pass `' \t\n'` and miss them. Excel and Word produce these constantly when you copy from a styled document. Characters covered: NBSP (U+00A0), narrow NBSP (U+202F), ideographic space (U+3000), em space (U+2003), thin space (U+2009). **Expected behavior:** treated identically to ASCII space - trimmed at edges, collapsed internally. **Why it matters:** "It looks fine but the join doesn't match" debugging sessions almost always end here. NBSP-padded keys are the silent killer. --- ### 03 - Smart punctuation **File**: `test_data/03_smart_punctuation.csv` -> `expected/03_smart_punctuation.csv` Curly quotes, dashes, ellipsis, primes - the autocorrect-as-you-type damage from Word/Excel. ASCII-fy where round-trip-safe. | Input | Output | Notes | |---|---|---| | `\u201c` `\u201d` (curly double) | `"` | | | `\u2018` `\u2019` (curly single) | `'` | Includes apostrophe | | `\u2014` (em-dash) | `-` | | | `\u2013` (en-dash) | `-` | | | `\u2026` (ellipsis) | `...` | | | `\u2032` (prime) | `'` | | | `\u2033` (double prime) | `"` | | | `\u00ab` `\u00bb` (guillemets) | `"` | | | `\u00d7` (multiplication sign) | **preserved** | Not safely round-trip-able to ASCII; `x` would be wrong | | `\u00b1` (plus-minus) | **preserved** | Same reasoning | **Why it matters:** smart-quote pollution breaks regex, breaks downstream parsers, and breaks string equality joins. The two preservation cases (multiplication, plus-minus) are deliberate - they have no faithful ASCII equivalent and forcing one is destructive. --- ### 04 - Unicode normalization forms **File**: `test_data/04_unicode_forms.csv` -> `expected/04_unicode_forms.csv` `café` can be encoded two ways: - NFC: `caf\u00e9` (one code point, e-acute as a unit) - NFD: `cafe\u0301` (two code points, plain e + combining accent) These render identically. They compare unequal. They have different lengths. macOS filesystem defaults to NFD, which means a CSV exported from a Mac and joined against a CSV from Excel can silently fail. Default normalization: NFC (most compact, what Excel emits, what most Western databases expect). **Cases covered:** - Pre-composed (NFC) e-acute and i-diaeresis. - Decomposed (NFD) versions of the same. - The `\uFB03` `ffi` ligature - **preserved** under NFC (NFKC would fold it to `ffi`). - Fullwidth Latin letters (`\uFF21\uFF22\uFF23` = `ＡＢＣ`) - **preserved** under NFC. - Roman numeral nine character (`\u2168`) - **preserved** under NFC. After cleaning, rows 1 and 2 must produce identical bytes (NFC and NFD both normalized to NFC). Same for rows 3 and 4. **Why it matters:** Mac-vs-Windows data joins. Catches "they look the same but won't match" bugs. **Opt-in `--unicode-form=NFKC` test:** not provided as a fixture but should exist as a unit test. Under NFKC, ligature folds to `ffi`, fullwidth folds to ASCII `ABC`, roman numeral folds to `IX`. NFKC is destructive for some legitimate text (mathematical notation, some CJK content) so it stays opt-in. --- ### 05 - Zero-width and invisible characters **File**: `test_data/05_zero_width_invisible.csv` -> `expected/05_zero_width_invisible.csv` These bytes show up from rich-text copy/paste, from RTL text, from accidentally-included U+FEFF in the middle of a cell (yes, this happens), and from some web-form pastes. Characters covered: U+200B (ZWSP), U+200C (ZWNJ), U+200D (ZWJ), U+200E (LRM), U+200F (RLM), U+00AD (soft hyphen), U+2060 (word joiner). **Expected behavior:** all stripped unconditionally. None of these has a legitimate role in tabular data cells, even when there's a domain reason for them in prose (typesetting Arabic, hyphenation hints in long-form text). For a CSV, they're noise. **Why it matters:** these are the *truly invisible* polluters. You can stare at the cell forever and not see them. They break joins, they bloat string lengths, they hash differently. The first time a buyer hits a zero-width-space in a customer name, this test is what saves them. --- ### 06 - Control characters **File**: `test_data/06_control_characters.csv` -> `expected/06_control_characters.csv` The C0 controls (U+0000 through U+001F) plus DEL (U+007F). Test cases: NUL, BEL, BS, VT, FF, ESC, DEL, and a multi-control combination. **Expected behavior:** all stripped from cell content. **The exception:** tab (U+0009), LF (U+000A), and CR (U+000D) are NOT stripped from inside quoted cells. Tab might be intentional formatting; LF/CR are handled by line-ending normalization (case 11). Outside of quoted cells, tab is whitespace and gets normalized like space. **Why it matters:** real-world exports from broken systems, half-corrupted database dumps, copy-paste from terminals (including ANSI escape sequences starting with ESC), and binary data accidentally exported as text all leave these in cells. A NUL byte mid-string breaks C-string-based parsers; a BEL makes terminals beep when you `cat` the file; ESC sequences corrupt logs. --- ### 07 - UTF-8 BOM **File**: `test_data/07_bom_utf8.csv` -> `expected/07_bom_utf8.csv` (byte-exact comparison) File starts with the three-byte sequence `EF BB BF`. Excel writes UTF-8 with BOM by default. Pandas `read_csv` usually handles this but leaves the BOM as part of the first column header name unless you pass `encoding='utf-8-sig'`. Result: a mystery column called `\ufeffid` that breaks every `df["id"]` lookup downstream. **Expected behavior:** - BOM stripped on read. - First column header is the clean string `id`, not `\ufeffid`. - Output file is written WITHOUT a BOM. **Diff target:** byte-for-byte equality with `expected/07_bom_utf8.csv`. The expected file must NOT have the BOM. **Why it matters:** Excel-origin data is the dominant input for the target buyer. Getting BOM handling wrong silently breaks the rest of the pipeline. --- ### 08, 09, 10 - Line endings: CRLF, CR-only, mixed **Files**: `08_line_endings_crlf.csv`, `09_line_endings_cr.csv`, `10_line_endings_mixed.csv` - 08: every line ends with CRLF (`\r\n`). Standard Windows. - 09: every line ends with CR (`\r`) only. Classic Mac. Rare but seen. - 10: same file contains all three: CRLF, LF, CR, CRLF, LF. **Expected behavior on output:** all lines end with LF (`\n`). Byte-exact match to the expected files. **Why LF as the default output:** it's what Linux uses, what every modern code editor handles, what Git stores by default, and what Streamlit / pandas write by default. CRLF is an option for buyers who specifically need Windows-style output, but the default minimizes round-trip surprises. **Why it matters:** mixed line endings cause "ghost rows" in some parsers, blank lines in some editors, and silent data loss in any tool that splits on one specific newline pattern. Case 10 is the disaster scenario - multi-source concat - and is the most important of the three. --- ### 11 - Embedded newlines inside quoted cells **File**: `test_data/11_embedded_newlines.csv` -> `expected/11_embedded_newlines.csv` The trap. File-level line-ending normalization must NOT collapse intentional newlines inside multi-line cells (addresses, notes columns). But the embedded line endings *should still* be normalized to LF for consistency. **Expected behavior:** - File-level line endings: LF. - Embedded CRLF inside a quoted cell: normalized to LF. - Embedded CR inside a quoted cell: normalized to LF. - Cell stays multi-line; the newline character count inside the cell is preserved. **Why it matters:** an address column with `123 Main St\r\nApt 4B\r\nNew York` is the canonical legitimate multi-line cell. A naive `text.replace('\r\n', '\n')` works correctly. A naive `text.split('\n')` to "remove blank lines" destroys the address. The cleaner must understand CSV quoting. --- ### 12 - Case operations (opt-in) **Files**: input `12_case_variations.csv`; three expected outputs: - `expected/12_case_variations__default.csv` (no flag - identity) - `expected/12_case_variations__email_lower.csv` (`--case email=lower`) - `expected/12_case_variations__name_title.csv` (`--case name=title`) Default behavior is **preserve case**. Case operations are opt-in per column because: - Lowercasing emails is almost always right (emails are case-insensitive per RFC 5321 local-part-aside). - Title-casing names is almost always right (`ALICE SMITH` -> `Alice Smith`), but must handle apostrophes correctly (`O'Connor` -> `O'Connor`, not `O'connor`). - Lowercasing product codes is almost always WRONG (`SKU-A1B2` is a code, not prose). So the tool offers per-column case ops, never a global one. The expected outputs cover the two most common configurations. **Tricky case to verify:** row 4 name `DAN O'CONNOR`. Under `--case=title` this must become `Dan O'Connor`, not `Dan O'connor`. Python's `str.title()` gets this wrong. Implementations should use `string.capwords()` or a regex that respects apostrophes inside words. **Why it matters:** dedup quality (case 01 in the deduplicator) depends on consistent case in the comparison columns. Buyers running 02 before 01 expect this to "just work" for the email column. --- ### 13 - Non-Latin scripts and emoji (preservation negative test) **File**: `test_data/13_non_latin_scripts.csv` -> `expected/13_non_latin_scripts.csv` Negative test: cleaning must not damage characters outside the Latin/punctuation block. Trim and NFC still apply (row 1 has leading and trailing space, which gets trimmed). Coverage: Chinese (Beijing), Japanese (katakana test), Arabic RTL, Cyrillic Russian, multi-codepoint emoji (party popper U+1F389, rocket U+1F680), accent + emoji combo (`café ☕`). **Expected behavior:** only whitespace and NFC normalization apply. All script-significant characters preserved exactly. **Why it matters:** the cleaner must be safe on international buyer data. Stripping "weird-looking" characters because they're outside ASCII is a textbook bug. Emoji in particular are in the supplementary planes (above U+FFFF) and naive byte-level filters often mangle them. --- ### 14 - Mojibake **Files**: input `14_mojibake.csv`; two expected outputs: - `expected/14_mojibake__default.csv` (no flag - bytes preserved, warning logged) - `expected/14_mojibake__fixed.csv` (`--fix-mojibake` - heuristic repair) Mojibake is the result of UTF-8 bytes being interpreted as cp1252 or Latin-1 and re-saved as UTF-8. Classic patterns: - `café` becomes `café` - `München` becomes `München` - `naïve` becomes `naïve` - The smart-apostrophe in `don't` becomes `don't` **Default behavior: warn, do NOT auto-fix.** Reasoning: mojibake repair is heuristic, and the heuristic can false-positive on legitimate strings that happen to contain `Ã` followed by another Latin-1 character. The right call for a tool sold to non-experts is to flag the suspicious pattern in the log and let the user opt in. **With `--fix-mojibake` (uses ftfy or equivalent):** repair attempted. The expected output shows fully repaired text including the smart-apostrophe-via-cp1252 case, which ftfy specifically handles. **Why it matters:** mojibake is silent corruption. The customer doesn't know it happened until a name shows up wrong on a printed invoice. Flagging it is the responsible default. --- ### 15 - Whitespace-only cells (the 02-vs-04 boundary) **File**: `test_data/15_whitespace_only_cells.csv` -> `expected/15_whitespace_only_cells.csv` Per TECHNICAL.md Section 9.3: 02 trims whitespace first, leaving an empty string. Script 04 then detects empty strings as disguised null. So 02's job in this file is to convert `" "`, `"\t\t"`, `"\u00A0\u00A0"`, and mixed-whitespace cells all into `""`. **What 02 does NOT do here:** - Does not decide whether the cell is "missing." That's 04's call. - Does not write `NaN` or `N/A` or any other sentinel. Just produces empty string. - Does not drop the row. Schema is invariant. **Expected behavior:** every whitespace-only cell becomes empty. Row count unchanged. Headers untouched. **Why it matters:** this is the single most-relitigated boundary in the bundle. Documenting it via fixture prevents drift. --- ### 16 - Dirty headers **File**: `test_data/16_dirty_headers.csv` -> `expected/16_dirty_headers.csv` Headers themselves are subject to all the same pollution as data cells. A header `" Email "` (NBSP-padded) breaks `df["Email"]` lookups because the actual column name has NBSP padding. Smart-quoted header `"\u201cEmail\u201d"` is even worse. **Expected behavior:** headers cleaned by the same rules as data. Note that the smart-quoted header `"Email"` (with surrounding quotes) becomes a header value containing literal ASCII double quotes, which then requires CSV-quoting in the output. The expected file is written with proper CSV escaping. **Why it matters:** broken column names break every downstream join, every selectbox in the GUI, and every CLI flag that takes a column name. Cleaning headers is non-negotiable. --- ### 17 - Preserve-intended (negative tests) **File**: `test_data/17_preserve_intended.csv` -> `expected/17_preserve_intended.csv` The negative-test file. Things 02 must NOT touch because they belong to other scripts: | Cell content | What 02 does | What 02 does NOT do | |---|---|---| | ` 100 ` | Trims to `100` | Doesn't reformat as `$100.00` (that's 03) | | `1 234` | Preserves as `1 234` | Doesn't collapse internal space (looks numeric, European thousand-sep) | | `$1,500.00` | Trims outer whitespace | Doesn't reformat currency (that's 03) | | `2024-01-15` | Trims outer whitespace | Doesn't reformat date (that's 03) | | `(555) 123-4567` | Trims outer whitespace | Doesn't reformat phone (that's 03); does not collapse internal space | | `+1 555 123 4567` | Trims outer whitespace | Same; phone-shaped, leave internal spacing alone | | `N/A` | Trims to `N/A` | Doesn't replace with empty or NaN (that's 04) | | `nan` | Trims to `nan` | Doesn't replace with empty or NaN (that's 04) | The internal-whitespace heuristic: if a cell parses as numeric, looks like a date, or matches a phone-shape regex (digits + common separators), do NOT collapse internal whitespace. Only collapse in cells classified as free text. This requires a per-cell check; document it in the implementation. **Why it matters:** scope discipline. If 02 starts reformatting dates because "while we're trimming whitespace anyway", it stops being 02 and starts being a worse 03. The DECISIONS.md Section 4a rule (functional scope) cuts the other way too: 02 must not reach into other scripts' territory. --- ### 18 - Empty file **File**: `test_data/18_empty_file.csv` (zero bytes) -> `expected/18_empty_file.csv` (zero bytes) **Expected behavior:** graceful no-op. Either produces an empty output file with a logged warning, or emits a clean error message naming the problem ("Input file is empty"). What it MUST NOT do: crash with `pandas.errors.EmptyDataError` traceback in the GUI. **Why it matters:** error UX standard from DECISIONS.md Section 4b - errors that name the problem and the fix, not stack traces. --- ### 19 - Headers only (no data rows) **File**: `test_data/19_headers_only.csv` -> `expected/19_headers_only.csv` Just headers, no data. Headers themselves are dirty (whitespace + NBSP + ZWSP). **Expected behavior:** headers cleaned, output is clean headers + no data rows. No crash, no warning required (it's a legitimate state). **Why it matters:** template files often look like this. The buyer might be cleaning a template before populating it. Don't punish them for it. --- ### 20 - Kitchen sink (integration) **File**: `test_data/20_kitchen_sink.csv` -> `expected/20_kitchen_sink.csv` The integration test. Combines: - UTF-8 BOM at file start. - CRLF line endings throughout. - Headers with leading/trailing space, NBSP, smart quotes, ZWSP. - Data cells with NBSP, internal multi-space, smart quotes, em-dash, ellipsis, primes (foot/inch markers). - A whitespace-only cell that should become empty. - Multiplication sign (preserved). **Expected output:** every transformation applied correctly, schema unchanged, file written as UTF-8 (no BOM) with LF line endings. **Why it matters:** this is the one fixture that catches transformation-order bugs. If smart-quote replacement runs before whitespace trim, you get different output than the other order. Picking and locking the order is part of the implementation; the fixture verifies it. **Recommended transformation pipeline order** (informative, not normative): 1. Decode bytes -> strip BOM at file level. 2. Normalize file-level line endings -> LF. 3. Parse CSV (with proper quoting for embedded newlines). 4. Per cell, in order: a. Unicode NFC normalize. b. Strip zero-width and control characters. c. Strip BOM if it appears mid-cell. d. Smart-quote ASCII-fy. e. Normalize embedded line endings to LF. f. Whitespace trim (outer). g. Internal whitespace collapse (text columns only - check after trim). h. Per-column case op (if configured). 5. Headers go through the same per-cell pipeline. 6. Write as UTF-8, LF line endings, no BOM. --- ### 21 - Excel pollution (multi-sheet XLSX) **File**: `test_data/21_excel_pollution.xlsx` (no expected file - manual / programmatic verification per sheet) Four sheets, each isolating an Excel-specific concern: **Sheet `Customers`** - dirty headers (NBSP, smart quotes, ZWSP) and dirty data cells (NBSP padding, tab padding, smart apostrophe in `O'Connor`, em-dash). One whitespace-only `name` cell to verify the 02/04 boundary applies on XLSX too. **Sheet `Notes`** - multi-line cells from Alt+Enter (LF inside cell), plus a cell with mixed CRLF inside (from someone pasting Windows text into Excel). Cells have wrap_text formatting set so the line breaks render in Excel. After cleaning, all in-cell line breaks should be LF. **Sheet `International`** - non-Latin scripts and emoji with surrounding whitespace. Verifies the preservation contract from case 13 holds for XLSX. **Sheet `ForceText`** - leading-zero IDs (e.g., `0001234`). These must not be stripped of leading zeros (that's not 02's job - it doesn't change semantic content). Row 3 has a leaked apostrophe (`'9999999`) from a force-text cell - this is a judgment call but the default is to preserve it; trying to detect "leaked apostrophe" is too error-prone. **Why it matters:** XLSX has pollution patterns that don't appear in CSV (Alt+Enter cells, force-text apostrophes, sheet structure). The XLSX reader path needs the same cleaning logic as the CSV reader path; this fixture verifies that. --- ## 5. What this corpus does NOT cover Listed so the gap is explicit, not hidden: 1. **Encoding detection** (cp1252 input, Latin-1 input, UTF-16). That's the I/O layer's job, not 02's transformation logic. Once the reader produces a Python `str`, 02 operates the same regardless of source encoding. Add I/O-layer fixtures separately when that layer is built. 2. **Performance / large files**. No multi-GB fixture is included because it bloats the repo. Add a benchmark (not a unit test) targeting a 500MB CSV; verify processing completes without OOM via chunked reads. 3. **Streamlit UI behavior**. The fixtures verify cleaning correctness; verifying the GUI shows the right preview, applies the right defaults, and renders cleaning in the diff view is a separate test layer (probably manual, possibly Playwright). 4. **Concurrency / file-locking** (e.g., user has the input file open in Excel). Expected to fail with a clean error, not corrupt data. Add a manual test, not a fixture. 5. **CLI argument parsing** for the various flags. Each flag should have a Typer-level test, separate from the fixtures here. --- ## 6. How to use this corpus ### As a build target Each fixture is one piece of the spec. Implement the cleaner against fixture 01, run, diff, fix, repeat. Move to 02. By the time fixture 20 passes, the script is done. ### As pytest fixtures ```python import pytest from pathlib import Path from src.core.text_cleaner import clean_csv CORPUS = Path("tests/corpus") # wherever this folder lands @pytest.mark.parametrize("name", [ "01_whitespace_basic", "02_whitespace_unicode", "03_smart_punctuation", "04_unicode_forms", "05_zero_width_invisible", "06_control_characters", "07_bom_utf8", "08_line_endings_crlf", "09_line_endings_cr", "10_line_endings_mixed", "11_embedded_newlines", "13_non_latin_scripts", "15_whitespace_only_cells", "16_dirty_headers", "17_preserve_intended", "18_empty_file", "19_headers_only", "20_kitchen_sink", ]) def test_default_config(name, tmp_path): inp = CORPUS / "test_data" / f"{name}.csv" expected = (CORPUS / "expected" / f"{name}.csv").read_bytes() out = tmp_path / "out.csv" clean_csv(inp, out) # default config assert out.read_bytes() == expected # Cases 12 and 14 have multiple expected files; parametrize them separately # with the relevant flags. # Idempotency property test - applies to every fixture: @pytest.mark.parametrize("name", [...same list...]) def test_idempotent(name, tmp_path): inp = CORPUS / "test_data" / f"{name}.csv" out1 = tmp_path / "out1.csv" out2 = tmp_path / "out2.csv" clean_csv(inp, out1) clean_csv(out1, out2) assert out1.read_bytes() == out2.read_bytes() ``` ### Regenerating fixtures If a default policy changes (e.g., switch the default Unicode form from NFC to NFKC, which would be a meaningful policy decision), the fixtures in `expected/` need regenerating. Edit `generate_test_data.py` and re-run. Document the policy change in DECISIONS.md before doing this.