# Caching a dataset processed with randomness

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-a-dataset-processed-with-randomness%2F65616%3E%5C
**Category:** 🤗Datasets\
**Created:** [December 13, 2023, 6:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-a-dataset-processed-with-randomness%2F65616 "2023-12-13T06:45:04Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ivnle](https://avatars.discourse-cdn.com/v4/letter/i/b5a626/32.png) [@ivnle](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fivnle)%5C
**Post date:** [December 13, 2023, 6:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-a-dataset-processed-with-randomness%2F65616%2F1 "2023-12-13T06:45:04Z")

</div>

Is it possible to cache datasets that have been processed with a “random” function?

For example, this tutorial [Process](https://huggingface.co/docs/datasets/process#data-augmentation) uses `random.randint` to pick words to mask. I believe this will break the hashing that Datasets relies upon to perform caching so I thought to use a generator like so.

```auto
        def sub_tok(batch, g):
            input_ids: List[List[int]] = batch["input_ids"]
            for i, example in enumerate(input_ids):
                token = example[
                    torch.randint(0, len(example), (1,), generator=g).item()
                ]
                input_ids[i] = [MASK if tok == token else tok for tok in example]
            batch["input_ids"] = input_ids

        g = torch.Generator(4444)
        lm_datasets["train"] = lm_datasets["train"].map(
            sub_tok,
            fn_kwargs={"generator": g},
            batched=True,
            num_proc=args.preprocessing_num_workers,
            load_from_cache_file=not args.overwrite_cache,
            desc=f"Enciphering XXX tokens per example",
        )

```

But I receive the error `TypeError: cannot pickle 'torch._C.Generator' object`. Removing the generator allows for the dataset to be processed but breaks caching as expected. Any advice?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [December 15, 2023, 2:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-a-dataset-processed-with-randomness%2F65616%2F2 "2023-12-15T14:01:02Z")

</div>

Hi! I opened a [PR](https://github.com/huggingface/datasets/pull/6502) that implements a serializer for `torch.Generator` to avoid the `pickle` error.

PS: `torch.Generator` expects a `device` as the first argument in the constructor, not a `seed`.