Remove Duplicate Lines

Paste any list or text and remove repeated lines in one step. Choose case-sensitive or case-insensitive matching, ignore extra spaces, keep the first or last occurrence, or extract only the unique or only the duplicated lines, and see exactly what was removed.

  • 100% free
  • No sign-up
  • Private & local
  • Works on mobile
Drop a text or CSV fileor browse files · or paste below

Private by design: every calculation runs locally in your browser. Nothing you type is sent to a server or stored by us.

What Does Remove Duplicate Lines Do?

A remove duplicate lines tool deletes repeated lines from a block of text so that each line appears only once. It's a quick way to clean up email lists, keyword lists, URL lists, product SKUs, log files, CSV exports, tags, names, and any data copied from spreadsheets or databases that has picked up duplicates along the way. Doing this by hand is slow and error-prone. Scanning hundreds of lines for repeats is exactly the kind of job a computer does perfectly in milliseconds.

This free tool does much more than a simple de-duplication. You can decide whether "Apple" and "apple" count as the same line, ignore stray spaces, keep the first or the last occurrence, and switch to extracting only unique lines or only the lines that had duplicates. A summary tells you how many lines were removed, and you can review every removed line before copying the result.

How to Remove Duplicate Lines

  1. Paste your text into the input box, one item per line, or drop a TXT or CSV file.
  2. Choose your options: case sensitivity, whitespace handling and which occurrence to keep.
  3. Review the result and the count of removed lines.
  4. Copy or download the cleaned list.

Options Explained

OptionWhat it doesExample
Case sensitiveTreat different capitalisation as different lines"Apple" and "apple" are both kept
Ignore extra spacesCompare lines after trimming and collapsing spaces"red  car" matches "red car"
Keep first / lastWhich copy survives, which matters when order is meaningfulKeep the most recent log entry
Only unique linesRemove every line that appears more than onceFind items that occur exactly once
Only duplicatesShow one copy of each line that was repeatedAudit which entries were duplicated
Keep blank linesLeave empty lines in place instead of de-duplicating themPreserve paragraph spacing

Common Uses

  • Email marketing: de-duplicate subscriber lists before importing them into your email platform, to avoid double sends and inflated counts.
  • SEO: merge keyword lists from several research tools into one clean list.
  • Developers: clean up log output, dependency lists, IDs and configuration entries.
  • Data analysis: find distinct values in a column copied from a spreadsheet.
  • Everyday lists: shopping lists, guest lists, reading lists and inventories.

Features

  • Instant de-duplicationHandles tens of thousands of lines in the browser.
  • Order preservedRemaining lines stay in their original order, with optional sorting.
  • Flexible matchingCase sensitivity and whitespace normalisation.
  • Three output modesDe-duplicate, unique-only or duplicates-only.
  • TransparencySee the count and the exact lines that were removed.
  • PrivateYour data is processed locally and never uploaded.

De-duplicating in Other Tools

Spreadsheets can remove duplicates too. In Excel use Data → Remove Duplicates, and in Google Sheets use Data → Data cleanup → Remove duplicates or the =UNIQUE() function. On the command line, sort -u file.txt or awk '!seen[$0]++' (which keeps the original order) do the same. For quick jobs, or when you want case-insensitive and whitespace-tolerant matching without formulas, pasting into this tool is usually fastest.

Why Duplicates Appear in Lists

Duplicate lines usually sneak in when data is combined from several places: two exported spreadsheets, a sign-up form that was submitted twice, contact lists merged from different apps, or search results collected over several days. Small differences such as capitalisation ("John@Example.com" vs "john@example.com") or a stray trailing space make duplicates hard to spot by eye, and they're exactly the cases this tool's case-insensitive and whitespace-tolerant matching is designed to catch. Cleaning duplicates before you import data saves time later, keeps statistics accurate and avoids embarrassing double emails.

Frequently Asked Questions

What happens to blank lines?

By default they're removed. Tick "Keep blank lines" to leave them in place; they're never treated as duplicates of each other.

Does removing duplicates change the order of my lines?

No. The remaining lines stay in their original order unless you tick "Sort result A–Z".

How do I remove duplicates regardless of capitalisation?

Leave "Case sensitive" unticked. "Paris", "PARIS" and "paris" are then treated as the same line.

Can I see which lines were duplicates?

Yes. Choose "Only lines that were duplicated", or open "Removed lines" below the result.

Can it de-duplicate CSV rows?

Yes. Each row is one line, so identical rows are removed. To de-duplicate by a single column, copy just that column in.

Is there a size limit?

Hundreds of thousands of lines work fine on most devices, since everything runs locally in your browser.