{ "_comment": "Expected results of the four scripts on the fictional Acme Cloud fixtures. Run each command from the skill folder after changing a script, a fixture, a threshold or the check catalogue, and compare: every expected line must appear, no must_not_appear line may appear, and the exit code must match. Recorded 2026-11-06 with Python 3.9.6.", "score_audit": { "command": "python3 scripts/score_audit.py assets/example-acme-findings.csv --config assets/example-acme-config.json --as-of 2026-11-01", "exit_code": 1, "expected_lines": [ "As of 2026-11-01. 52 checks from 62 findings: 11 PASS, 25 WARN, 14 FAIL, 1 N/A, 1 NOT CHECKED.", "- O1 Paid media daily (paid_media_performance): Do not use until fixed. Blocking FAIL: FRS-03, FRS-05, KEY-08, MET-07, CHN-07. Caveats: MET-01, REC-02, REC-04, CHN-06.", "- O2 Pipeline by source (pipeline_and_lead_source): Do not use until fixed. Blocking FAIL: KEY-04, KEY-06, IDN-05, MET-02. Caveats: KEY-03, MET-01, CHN-08.", "- O3 Revenue by channel (revenue_attribution): Use with caveats. Caveats: IDN-03, MET-01, MET-05, REC-06, CHN-08.", "- O4 Qualified-demo upload to Google Ads (activation_audience_or_upload): Do not use until fixed. Blocking FAIL: IDN-05, OPS-04. Also not assessed: OPS-08. Caveats: IDN-01, REC-08.", "- O5 Exec KPIs (executive_kpi): Do not use until fixed. Blocking FAIL: KEY-08, MET-02, MET-04. Caveats: MET-01, REC-07.", "Overall 45 (44.52), Failing, coverage 97.5%, graded.", "- FRS Freshness and completeness: 34.4, coverage 100.0%", "- KEY Grain, keys and values: 40.5, coverage 100.0%", "- IDN Identity: 60.0 (capped from 63.3), coverage 100.0%", "- MET Metric definitions and owners: 32.4, coverage 100.0%", "- REC Reconciliation: 62.5, coverage 100.0%", "- CHN UTM and channel classification: 43.8, coverage 100.0%", "- OPS Controls and ownership: 38.5, coverage 81.3%", "Priorities: P1 11, P2 4, P3 12, P4 15.", "Nothing was changed." ], "p1_findings": [ "F03", "F09", "F16", "F19", "F21", "F27", "F30", "F32", "F35", "F51", "F57" ], "p2_findings": [ "F05", "F06", "F49", "F54" ], "p3_findings": [ "F15", "F23", "F25", "F29", "F33", "F37", "F39", "F41", "F42", "F50", "F53", "F62" ] }, "score_audit_rejects_template_text": { "command": "python3 scripts/score_audit.py assets/example-acme-findings.csv --config assets/audit-config-template.json", "exit_code": 2, "expected_stderr_contains": "config: placeholder text on line 3 (audit_name: )" }, "reconcile_exports": { "command": "python3 scripts/reconcile_exports.py assets/example-acme-ads-platform.csv assets/example-acme-ads-warehouse.csv --key platform,account_id,date --measures spend --currency-col currency --label-a platform --label-b warehouse --rollup platform,account_id,month --month-from date --mature-through 2026-10-28", "exit_code": 1, "expected_lines": [ "Keys: platform 84, warehouse 81, matched 81, missing in warehouse 3, unexpected in warehouse 0.", "Duplicates: platform 0 keys (0 excess rows), warehouse 0 keys (0 excess rows).", "Currency mismatches: 28 keys", " - linkedin_ads, 500000001: 28 keys, platform EUR, warehouse USD", "Value mismatches (spend): 1 of 81 matched keys", " - facebook_ads, act_1234567890, 2026-10-27: platform 1,500.00, warehouse 1,477.50, difference -22.50 (-1.50%)", " - google_ads, 1234567890, 2026-10-10: platform spend 2,300.00", " - google_ads, 1234567890, 2026-10-11: platform spend 2,300.00", " - google_ads, 1234567890, 2026-10-12: platform spend 2,300.00", " - USD: platform 106,400.00, warehouse 110,677.50, signed difference +4,277.50, absolute difference 18,122.50", " - EUR: platform 11,200.00, warehouse 0.00, signed difference -11,200.00, absolute difference 11,200.00", " - USD: platform 99,500.00, warehouse 99,477.50, difference -22.50 (-0.02%)", " - currency-mismatched keys, not compared: 28 keys; platform 11,200.00 EUR; warehouse 11,200.00 USD", " - facebook_ads, act_1234567890, 2026-10: platform 42,000.00 USD, warehouse 41,977.50 USD, difference -22.50 (-0.05%), matched-only -0.05%, flagged keys 1: PASS", " - google_ads, 1234567890, 2026-10: platform 64,400.00 USD, warehouse 57,500.00 USD, difference -6,900.00 (-10.71%), matched-only 0.00%, missing keys 3: FAIL", " - linkedin_ads, 500000001, 2026-10: platform 11,200.00 EUR, warehouse 11,200.00 USD, difference not comparable (currencies differ), currency mismatch keys 28: FAIL (currency)", "Result: 3 missing, 0 unexpected, 0 duplicate keys, 28 currency mismatches, 1 value mismatches; rollups 1 PASS, 0 WARN, 2 FAIL." ], "must_not_appear": [ " - EUR: platform 11,200.00, warehouse 11,200.00, difference 0.00 (0.00%)" ], "exceptions_rows_with_flag": 32 }, "classify_channels": { "command": "python3 scripts/classify_channels.py assets/example-acme-sessions-by-source.csv --registry assets/example-acme-utm-registry.csv --warehouse-map assets/example-acme-warehouse-channel-map.csv", "exit_code": 1, "expected_lines": [ "Rows 18, sessions 157,950, key events 2,810, click-ID sessions 41,310.", "- CHN-01 vocabulary: unapproved pairs 5,050 sessions (3.20%); pairs at 0.5% or more: mailchimp / newsletter 2,000 (1.27%), linkedin / paid-social 1,800 (1.14%). Suggested: WARN", "- CHN-02 Unassigned: 3,350 sessions (2.12%), 45 key events (1.60%). Suggested: WARN", "- CHN-03 click IDs: 41,310 click-ID sessions, 40,950 in paid channels (99.13%); outside paid: (direct) / (none) / (direct) (Direct) 360. Suggested: PASS", "- CHN-04 variants: linkedin / paid_social spelled 3 ways (linkedin / paid_social, linkedin / paid-social, LinkedIn / paid_social); 2,500 sessions in non-canonical spellings (1.58%). Suggested: WARN", "- CHN-05 macros: 1,100 sessions (0.70%): linkedin / paid_social / {{campaign.name}} (Paid Social). Suggested: FAIL", "Suggested statuses: CHN-01 WARN, CHN-02 WARN, CHN-03 PASS, CHN-04 WARN, CHN-05 FAIL, CHN-07 FAIL." ], "chn07": "agreement 151,550 of 157,950 sessions (95.95%); paid mapped to organic or direct: linkedin / paid-social / 2026_10_pipeline_webinar as Organic Social; unknown mapped to Direct: (not set) / (not set) / (not set) as Direct; suggested FAIL" }, "classify_channels_macro_cases": { "command": "python3 scripts/classify_channels.py assets/regression-macro-cases.csv", "exit_code": 1, "expected_lines": [ "- google / cpc / [Brand] Search - US: 100, Paid Search; auto-tagged: verify network type", "- CHN-05 macros: 700 sessions (77.78%): google / cpc / {campaignid} (Paid Search), google / cpc / %7Bcampaignid%7D (Paid Search), google / cpc / [[campaign]] (Paid Search), newsletter / email / *|CAMPAIGN_UID|* (Email), newsletter / email / %%campaign%% (Email), linkedin / paid_social / __CAMPAIGN_NAME__ (Paid Social), facebook / paid_social / {{site_source_name}} (Paid Social). Suggested: FAIL" ], "note": "Single square brackets are a naming convention, not a macro: the [Brand] row carries no macro flag and stays out of the CHN-05 line." }, "check_dbt_artifacts": { "command": "python3 scripts/check_dbt_artifacts.py assets/example-acme-dbt-manifest.json --run-results assets/example-acme-dbt-run-results.json --sources assets/example-acme-dbt-sources.json --config assets/example-acme-config.json --adapter bigquery --as-of 2026-11-01", "exit_code": 1, "expected_lines": [ "Critical nodes (audit config outputs[].models): 7 models, 7 sources (7 of them found through depends_on).", "Not in the manifest: none.", "Model fct_deals", "Model mart_sessions", "Source linkedin_ads.ad_analytics_by_campaign (identifier ad_analytics_by_campaign; read by stg_ads__daily_spend, fct_paid_media_daily)", " Note: gclid has no not_null test (not a grain test; click IDs can be empty when gbraid or wbraid is used, so measure coverage with IDN-01)", "- OPS-01 FAIL: no grain test on mart_pipeline_by_source (no tests at all)", "- OPS-02 WARN: 1 test warning(s) on critical models (fct_invoices: accepted_values on currency: warn, 3 failures); confirm an owner and date for each", "- All 7 freshness results pass; a pass on a metadata-based check proves little (FRS-04)." ], "frs04": "FAIL: metadata-based freshness on ga4.events (wildcard identifier events_*, unreliable on BigQuery) and stripe.invoice; ingestion timestamp _fivetran_synced on google_ads.campaign_stats and linkedin_ads.ad_analytics_by_campaign; hubspot.deal, hubspot.contact and facebook_ads.basic_ad read data timestamps (PASS)" }, "check_dbt_artifacts_tag_route": { "command": "python3 scripts/check_dbt_artifacts.py assets/example-acme-dbt-manifest.json --run-results assets/example-acme-dbt-run-results.json --sources assets/example-acme-dbt-sources.json --adapter bigquery --as-of 2026-11-01", "exit_code": 1, "expected_lines": [ "Critical nodes (tag:critical): 5 models, 7 sources (3 of them found through depends_on).", "Source linkedin_ads.ad_analytics_by_campaign (identifier ad_analytics_by_campaign; read by stg_ads__daily_spend, fct_paid_media_daily; not tagged critical)" ], "note": "Tags in the project miss fct_deals and mart_sessions, so the tag route checks 5 of the 7 models the outputs read; the config route is the one the procedure uses." } }