# Minhash Deduplication

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fminhash-deduplication%2F19992%3E%5C
**Category:** 🤗Datasets\
**Created:** [July 5, 2022, 4:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fminhash-deduplication%2F19992 "2022-07-05T04:35:17Z")\
**Posts on this page:** 1\
**Showing post:** 11

<div class="post-metadata">

**Author:** ![conceptofmind](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/conceptofmind/32/55565_2.png) [@conceptofmind](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fconceptofmind)%5C
**Post date:** [August 4, 2022, 2:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fminhash-deduplication%2F19992%2F11 "2022-08-04T02:18:58Z")

</div>

Hi @liyongsea ,

Additionally to the warning and error above, I have also been experiencing a second error which sometimes occurs on datasets around 300 GBs of size:

```auto
Time to filter dataset: 866.99| 5244089/5244135 [1:22:04<00:00, 1083.08ex/s]
Size of filtered dataset: 52435971| 5237337/5244135 [1:21:50<00:04, 1364.48ex/s]
2889863it [33:35, 1635.90it/s]
Process ForkPoolWorker-1:
Killed
Traceback (most recent call last):
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

```

I believe this error may be in relation to this function:

```python
def minhash_iter(dataset_iterator: Type[Dataset]):
    with mp.Pool() as pool:
        for data in pool.imap_unordered(
            _compute_min_hash,
            ThreadedIterator(dataset_iterator, max_queue_size=10000),
            chunksize=100,
        ):
            if data is not None:
                yield data

```

Thank you,

Enrico

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fminhash-deduplication%2F19992)._