# How to create a dataset for translation

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-a-dataset-for-translation%2F55879%3E%5C
**Category:** Beginners\
**Created:** [September 21, 2023, 1:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-a-dataset-for-translation%2F55879 "2023-09-21T13:24:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Altabus](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/altabus/32/21087_2.png) [@Altabus](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAltabus)%5C
**Post date:** [September 21, 2023, 1:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-a-dataset-for-translation%2F55879%2F1 "2023-09-21T13:24:32Z")

</div>

Hello, I need to create a translation dataset based on my text corpus. I only found this guide on how to create a dataset, [Create a dataset](https://huggingface.co/docs/datasets/v2.14.5/en/create_dataset)  
but it is negligible and there is no information on how to create a dataset of parallel texts

Thank you in advance

---

<div class="post-metadata">

**Author:** ![dblakely](https://avatars.discourse-cdn.com/v4/letter/d/54ee81/32.png) [@dblakely](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdblakely)%5C
**Post date:** [September 25, 2023, 10:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-a-dataset-for-translation%2F55879%2F2 "2023-09-25T10:44:58Z")

</div>

It varies depending on what exactly you need, but if you just create a folder like:

```auto
translation_dataset/
    train.jsonl
    eval.jsonl
    README.md

```

then you’ll be able to load the dataset and using Huggingface `datasets`. For example:

```auto
from datasets import load_dataset

path = "path/to/translation_dataset"
train_set = load_dataset(path, split="train")
eval_set = load_dataset(path, split="test")

```

Beyond that, there could be pre-processing you need to do depending on what your examples look like and what model you’re using.