Finding near-duplicates with Jaccard similarity and MinHash

Nelson Elhage · 3 Jul 2024

Nelson Elhage did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

Our reading
Document similarity scores across a corpus tend to cluster bimodally near identical and near unrelated, rather than spreading evenly.

Their wordsI suspect - but haven't verified - that in many corpora, we will encounter fairly bimodal Jaccard values - two clusters, near 1 and 0.

Keep reading

The other pieces are the ones whose claims are worded most like this one's. That measures language, not agreement: they may argue the opposite.