# One-to-many augmentations on the fly
**URL:**
**Author:**  [@pbelcak](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpbelcak)%5C
**Post date:** [March 27, 2023, 3:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fone-to-many-augmentations-on-the-fly%2F34804%2F1 "2023-03-27T15:24:16Z")
Hi,
I have a large dataset of long documents that I would like stream for training. The problem is that my model needs inputs of at most 512 tokens at a time, so I need to split/portion/partition each document into pieces of 512 tokens.
Is there a way to transform the rows of a HF Dataset on the fly such that one document may be augmented/transformed into several samples at once?
Both `.map()` and `.set_transform()` seem to account only for one-to-one transformations.
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [March 28, 2023, 3:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fone-to-many-augmentations-on-the-fly%2F34804%2F2 "2023-03-28T15:47:30Z")
Hi! You can use `.map` in the `batched` mode to transform a single example into multiple ones. You can find examples of such transforms in our [docs](https://huggingface.co/docs/datasets/process#batch-processing) and in the [course](https://huggingface.co/course/en/chapter5/3?fw=pt#the-map-methods-superpowers).
---
**Author:**  [@pbelcak](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpbelcak)%5C
**Post date:** [March 29, 2023, 8:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fone-to-many-augmentations-on-the-fly%2F34804%2F3 "2023-03-29T08:11:28Z")
But `.map()` does so ahead of time and not on the fly, correct?
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [March 29, 2023, 5:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fone-to-many-augmentations-on-the-fly%2F34804%2F4 "2023-03-29T17:17:50Z")
Correct!
Considering the number of input examples needs to match the number of output examples in `set_transform`, to do this on the fly, you would have to partition each document into a list of strings/tokens in the transform and then additionally do some postprocessing while iterating over this transformed dataset to turn these lists of strings/tokens into individual examples before finally passing them to the model.
---
**Author:**  [@pbelcak](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpbelcak)%5C
**Post date:** [March 30, 2023, 5:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fone-to-many-augmentations-on-the-fly%2F34804%2F5 "2023-03-30T17:45:02Z")
Okay, thanks for the response, so short answer – there is no way to do this with the huggingface Dataset’s library. I feel like, however, it should not be too hard to write a batched extension to `.set_transform()`, and I could perhaps give it a go. Any starting points or reasons not to try?
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [March 31, 2023, 1:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fone-to-many-augmentations-on-the-fly%2F34804%2F6 "2023-03-31T13:49:44Z")
You can define a transform that outputs more examples than it gets, but then indexing a single example (`dataset[i]`) will not behave as expected as we don’t know the “example offsets”, which shouldn’t be a problem if you only read examples in the batched manner (`dataset[i:j]`).
---
**Author:**  [@pbelcak](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpbelcak)%5C
**Post date:** [April 6, 2023, 2:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fone-to-many-augmentations-on-the-fly%2F34804%2F7 "2023-04-06T14:06:55Z")
But then I end up having un-even batch sizes, right? That would make the memory requirements of my training loop unpredictable.