Unique lines removes duplicates while preserving the first occurrence of each line. Whitespace differences count as distinct lines, which matches exact-list deduplication needs in agent pipelines, crawler output cleanup, and merge-conflict review.
How it works
The input is split on newlines and walked in order. Each line is tracked in a set the first time it appears; subsequent identical lines are skipped. The surviving lines are joined back with \n, keeping the relative order of their first appearances. Empty lines are deduplicated like any other line unless you trim them in a separate step first.
When to use it
- Deduplicate URL or email lists from crawlers and scrapers
- Clean merged agent outputs before sending to an LLM to save tokens
- Shrink log extracts while keeping chronological first-seen ordering
- Normalize allowlists exported from multiple sources before diffing
Example
apple\nbanana\napple\ncherry becomes apple\nbanana\ncherry. Pair with sort-lines if you also need alphabetical ordering after deduplication.
Limitations
Comparison is exact per line—leading or trailing spaces make lines distinct. Trim lines first if surrounding whitespace should be ignored. Case differences (Apple vs apple) are treated as separate entries. The tool operates line-wise, not on multi-line records.