Sn Da Large File Analysis

This is a performance rescue kit for Excel files that choke pandas. It routes you through three tiers: under 10k rows, use normal read_excel; 10k to 100k, cache to Parquet; over 100k, stream with openpyxl's read_only mode and iter_rows to avoid loading everything into memory. The streaming converter writes chunks as Parquet with all columns cast to string to dodge PyArrow schema conflicts, then you convert types after. Also includes dtype downcasting that claims 50 to 80 percent memory savings and a write_only mode for exports over a million rows. The trigger logic is baked in, so it activates on row counts or keywords like OOM, streaming read, or big data. Good for when your data analysis suddenly becomes a memory management problem.

Install to Claude Code

npx -y skills add opensensenova/sensenova-skills --skill sn-da-large-file-analysis --agent claude-code