CSV Data Wrangler

Purpose

When to Use

• Processing large CSV files efficiently
• Cleaning and validating CSV data
• Transforming and reshaping datasets
• Handling encoding and delimiter issues
• Merging or splitting CSV files
• Converting between tabular formats
• Querying CSV with SQL (DuckDB)

Quick Start

• Processing large CSV files efficiently
• Cleaning and validating CSV data
• Transforming and reshaping datasets
• Handling encoding and delimiter issues
• Querying CSV with SQL

• Building Excel files with formatting (use xlsx-skill)
• Statistical analysis of data (use data-analyst)
• Building data pipelines (use data-engineer)
• Database operations (use sql-pro)

Decision Framework

Tool Selection by File Size:
├── < 100MB → pandas
├── 100MB - 1GB → pandas with chunking or polars
├── 1GB - 10GB → DuckDB or polars
├── > 10GB → DuckDB, Spark, or streaming
└── Quick exploration → csvkit or xsv CLI

Processing Type:
├── SQL-like queries → DuckDB
├── Complex transforms → pandas/polars
├── Simple filtering → csvkit/xsv
└── Streaming → Python csv module

Core Workflows

1. Large CSV Processing

1. Profile file (size, encoding, delimiter)
2. Choose appropriate tool for scale
3. Process in chunks if memory-constrained
4. Handle encoding issues (UTF-8, Latin-1)
5. Validate data types per column
6. Write output with proper quoting

2. Data Cleaning Pipeline

1. Load sample to understand structure
2. Identify missing and malformed values
3. Define cleaning rules per column
4. Apply transformations
5. Validate output quality
6. Log cleaning statistics

3. CSV Query with DuckDB

1. Point DuckDB at CSV file(s)
2. Let DuckDB infer schema
3. Write SQL queries directly
4. Export results to new CSV
5. Optionally persist as Parquet

Best Practices

• Always specify encoding explicitly
• Use chunked reading for large files
• Profile before choosing tools
• Preserve original files, write to new
• Validate row counts before/after
• Handle quoted fields and escapes properly

Anti-Patterns