AI Near-Duplicate Finder

A normal duplicate finder needs the text to match character for character (or close to it). This compares meaning using sentence embeddings, so it catches paraphrases like 'the meeting is at 3pm' and 'our 3 oclock catch-up' even though they share almost no words. It uses cosine similarity, not string matching.

No. The MiniLM embedding model runs entirely in your browser via WebAssembly. Your list is processed on your device and never sent to a server. Only the model is downloaded, once, then cached.

all-MiniLM-L6-v2, a compact sentence-transformer (about 23 MB) that maps each line to a 384-dimensional vector. It is fast, widely used for semantic similarity, and runs locally through transformers.js and ONNX.

It is the minimum cosine similarity for two lines to be flagged as near-duplicates. A higher threshold (closer to 98%) only catches very close paraphrases; a lower one (toward 50%) catches looser matches but may flag merely related lines. The default of 82% is a good starting point.

It comfortably handles hundreds to a few thousand lines. Because every pair is compared, the work grows with the square of the line count, so very large lists take longer; the model downloads once on first use, then is cached, and everything runs in your browser so nothing is uploaded.