# How can I drop duplicates on datasets module?
**URL:**
**Author:**  [@ju-resplande](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fju-resplande)%5C
**Post date:** [March 4, 2022, 6:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-drop-duplicates-on-datasets-module%2F15369%2F1 "2022-03-04T18:56:30Z")
I have a very large dataset (about 200 GB) which a need to certify that are not duplicates, dropping them. How can I achieve that on datasets? Thanks!
---
**Author:**  [@cspartalis](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcspartalis)%5C
**Post date:** [July 4, 2022, 2:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-drop-duplicates-on-datasets-module%2F15369%2F2 "2022-07-04T14:13:02Z")
One possible solution -which doesn’t exactly answer the question- is to:
1. Convert the Dataset object to a Pandas DataFrame object (e.g. `pandas.DataFrame(your_dataset)`).
2. Employ [pandas.DataFrame.drop\_duplicates](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.drop_duplicates.html) function.
3. Finally, return to a Dataset object using [datasets.Dataset.from\_pandas](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Dataset.from_pandas) function.
I had the same question with you and I couldn’t find anything better.
---
**Author:**  [@conceptofmind](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fconceptofmind)%5C
**Post date:** [July 4, 2022, 3:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-drop-duplicates-on-datasets-module%2F15369%2F3 "2022-07-04T15:08:39Z")
Hi,
Papers such as OPT and PaLM detail dataset deduplication with MinHash and locality-sensitive hashing (LSH) with a Jaccard similarity of .9 or so. You would have to implement these algorithms or you can use Google’s tool for deduplicating text: [GitHub - google-research/deduplicate-text-datasets](https://github.com/google-research/deduplicate-text-datasets).
HuggingFace also has an example of text deduplication in this repository here: [transformers/preprocessing.py at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/main/examples/research_projects/codeparrot/scripts/preprocessing.py) and [transformers/minhash\_deduplication.py at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/main/examples/research_projects/codeparrot/scripts/minhash_deduplication.py).
Best,
Enrico
---
**Author:**  [@conceptofmind](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fconceptofmind)%5C
**Post date:** [July 5, 2022, 1:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-drop-duplicates-on-datasets-module%2F15369%2F4 "2022-07-05T01:32:10Z")
Hi,
I made some adaptations to the deduplication scripts in the Code Parrot research repository. Here is what a barebones deduplication script would look like with MinHash and LSH: [GitHub - conceptofmind/Huggingface-deduplicate](https://github.com/conceptofmind/Huggingface-deduplicate)
I also opened up a new post regarding a general use case here: [Minhash Deduplication](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fminhash-deduplication%2F19992)
Best,
Enrico