normalizing-text-for-measurement

wdm0006/python-skills

Strip, parse, and segment a document before something else measures it - close tokens that emit no separator and fuse two blocks into one word, character-based metrics inflated by leftover syntax while word-based metrics break on boundaries the strip never inserted, sibling consumers that skip the normalizer, parser presets whose rule set reclassifies a construct, sections keyed by a non-unique heading that collapse and double-count into an ancestor, deliberate lossiness mistaken for a bug, and regression tests that assert markdown-equals-plain-equivalent instead of a float from an unpinned library

GitHub stars

108

Sponsors

SnugThe clipboard manager you and your agent can use, safelyAgent CaffeineClose your lid, come back to finished workWaxBlock anything with supervision, without erasing your iPhoneOpen StepsSponsoredFree and open-source skills that make AI coding agents easier to understand, verify, and control.