# How can I drop duplicates on datasets module?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-drop-duplicates-on-datasets-module%2F15369%3E%5C
**Category:** Beginners\
**Created:** [March 4, 2022, 6:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-drop-duplicates-on-datasets-module%2F15369 "2022-03-04T18:56:30Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![conceptofmind](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/conceptofmind/32/55565_2.png) [@conceptofmind](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fconceptofmind)%5C
**Post date:** [July 5, 2022, 1:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-drop-duplicates-on-datasets-module%2F15369%2F4 "2022-07-05T01:32:10Z")

</div>

Hi,

I made some adaptations to the deduplication scripts in the Code Parrot research repository. Here is what a barebones deduplication script would look like with MinHash and LSH: [GitHub - conceptofmind/Huggingface-deduplicate](https://github.com/conceptofmind/Huggingface-deduplicate)

I also opened up a new post regarding a general use case here: [Minhash Deduplication](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fminhash-deduplication%2F19992)

Best,

Enrico

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-drop-duplicates-on-datasets-module%2F15369)._