# Correct way to create a Dataset from a csv file
**URL:**
**Author:**  [@Terence](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTerence)%5C
**Post date:** [March 14, 2022, 11:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-create-a-dataset-from-a-csv-file%2F15686%2F1 "2022-03-14T11:09:22Z")
Hi, Could somebody please point me to a beginner’s tutorial that
would enable to load a csv file in a dataset for a finetuning task. I completed such a task as a learning experience using the “opus\_books” dataset and my DatasetDict takes the following form:
> > > books
> > > DatasetDict({
> > > train: Dataset({
> > > features: [‘id’, ‘translation’],
> > > num\_rows: 127085
> > > })
> > > })
However, I’m struggling to get it right with a csv file. With the command
> > > luganda\_dataset = load\_dataset(‘csv’, data\_files=‘Luganda.csv’, cache\_dir=‘./’)
> > > I end up with a DatasetDict in the following form:
> > > luganda\_dataset
> > > DatasetDict({
> > > train: Dataset({
> > > features: [‘en’, ‘lg’, ‘Unnamed: 2’, ‘Unnamed: 3’],
> > > num\_rows: 16000
> > > })
> > > })
> > > This is clearly not right and I am at a loss as to what I need to do.
> > > I would welcome some guidance on what I have to do to create the Dataset correctly.
> > > Thank you.
> > > Terence
---
**Author:**  [@merve](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmerve)%5C
**Post date:** [March 14, 2022, 11:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-create-a-dataset-from-a-csv-file%2F15686%2F2 "2022-03-14T11:58:29Z")
Did you check your dataset’s features?
Maybe check first by reading it with a pandas dataframe, and if it works, do
```auto
> from datasets import Dataset
> import pandas as pd
> df = pd.DataFrame({"a": [1, 2, 3]})
> dataset = Dataset.from_pandas(df)
```
But this is just a workaround to debug.
---
**Author:**  [@Terence](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTerence)%5C
**Post date:** [March 15, 2022, 10:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-create-a-dataset-from-a-csv-file%2F15686%2F4 "2022-03-15T10:10:07Z")
I’m aware of the reason for ‘Unnamed:2’ and ‘Unnamed 3’ - each row of the csv file ended with “,”. However, I am still getting the column names “en” and “lg” as features when the features should be “id” and “translation”. I would really welcome some guidance on this, please.
---
**Author:**  [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [March 15, 2022, 10:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-create-a-dataset-from-a-csv-file%2F15686%2F5 "2022-03-15T10:17:10Z")
Well, what is the first line of your csv file? What’s the output of
> head -n 3 Luganda.csv
---
**Author:**  [@Terence](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTerence)%5C
**Post date:** [March 15, 2022, 10:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-create-a-dataset-from-a-csv-file%2F15686%2F6 "2022-03-15T10:37:31Z")
(venv) tel34@moses:~/nmtgateway/Helsinki-NLP/finetuning$ head -n 3 Luganda.csv
en,lg
All refugees were requested to register with the chairman.,Abanoonyiboobubudamu bonna baasabiddwa beewandiise ewa ssentebe.
They called for a refugees’ meeting yesterday.,Baayise olukungaana lw’abanoonyiboobubudamu eggulo.
---
**Author:**  [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [March 15, 2022, 11:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-create-a-dataset-from-a-csv-file%2F15686%2F7 "2022-03-15T11:07:13Z")
So as you can see, the titles of the columns in your csv are indeed en and lg… I don’t understand your problem. The feature names are the column names.
---
**Author:**  [@Terence](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTerence)%5C
**Post date:** [March 15, 2022, 11:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-create-a-dataset-from-a-csv-file%2F15686%2F8 "2022-03-15T11:26:34Z")
Well, when I run the fine-tuning script, which works perfectly well with fine-tuning t5-small with the “opus\_books” dataset, I get the following KeyError:
Traceback (most recent call last):
File “finetune\_luganda.py”, line 24, in
tokenized\_luganda = luganda\_dataset.map(preprocess\_function, batched=True)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/dataset\_dict.py”, line 512, in map
for k, dataset in self.items()
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/dataset\_dict.py”, line 512, in
for k, dataset in self.items()
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2120, in map
desc=desc,
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 518, in wrapper
out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 485, in wrapper
out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/fingerprint.py”, line 413, in wrapper
out = func(self, \*args, \*\*kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2485, in \_map\_single
offset=offset,
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2367, in apply\_function\_on\_filtered\_inputs
processed\_inputs = function(\*fn\_args, \*additional\_args, \*\*fn\_kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2062, in decorated
result = f(decorated\_item, \*args, \*\*kwargs)
File “finetune\_luganda.py”, line 16, in preprocess\_function
inputs = [prefix + example[source\_lang] for example in examples[“translation”]]
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 123, in **getitem**
values = super(). **getitem** (key)
File “/usr/lib/python3.6/collections/ **init**.py”, line 991, in **getitem**
raise KeyError(key)
KeyError: ‘translation’
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [March 21, 2022, 12:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-create-a-dataset-from-a-csv-file%2F15686%2F9 "2022-03-21T12:13:21Z")
Hi! These are the features of the `opus_books` dataset:
```python
features=datasets.Features(
{
"id": datasets.Value("string"),
"translation": datasets.Translation(languages=(self.config.lang1, self.config.lang2)),
},
),
```
You can align your dataset with it as follows (after loading):
```auto
ds.map(
lambda ex, i: {"id": i, "translation": dict(ex)},
remove_columns=["en", "lg"],
features=Features({"id": datasets.Value("string"), "translation": datasets.Translation(languages=["en", "lg"])}),
with_indices=True,
)
```
---
**Author:**  [@Terence](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTerence)%5C
**Post date:** [March 21, 2022, 7:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-create-a-dataset-from-a-csv-file%2F15686%2F10 "2022-03-21T19:34:58Z")
Thank you for taking the trouble to answer my query :-). I have incorporated your suggestion in my script yet I am still getting KeyError(‘translation’). See below with a Traceback:
import datasets
from transformers import AutoTokenizer
from datasets import load\_dataset
from transformers import DataCollatorForSeq2Seq
from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer
luganda\_dataset = load\_dataset(“csv”, data\_files=“Luganda.csv”)
luganda\_dataset.map(lambda ex, i: {“id”: i, “translation”: dict(ex)}, remove\_columns=[“en”, “lg”], features=datasets.Features({“id”: datasets.Value(“string”), “translation”: datasets
.Translation(languages=[“en”, “lg”])}), with\_indices=True,)
luganda\_dataset = luganda\_dataset[“train”].train\_test\_split(test\_size=0.2)
tokenizer = AutoTokenizer.from\_pretrained("./opus-mt-en-lg")
source\_lang = “en”
target\_lang = “lg”
prefix = "translate English to Luganda: "
def preprocess\_function(examples):
inputs = [prefix + example[source\_lang] for example in examples[“translation”]]
targets = [example[target\_lang] for example in examples[“translation”]]
model\_inputs = tokenizer(inputs, max\_length=128, truncation=True)
with tokenizer.as\_target\_tokenizer():
labels = tokenizer(targets, max\_length=128, truncation=True)
model\_inputs[“labels”] = labels[“input\_ids”]
return model\_inputs
tokenized\_luganda = luganda\_dataset.map(preprocess\_function, batched=True)
model = AutoModelForSeq2SeqLM.from\_pretrained(“t5-small”)
data\_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)
training\_args = Seq2SeqTrainingArguments(
output\_dir="./results",
evaluation\_strategy=“epoch”,
learning\_rate=2e-5,
per\_device\_train\_batch\_size=16,
per\_device\_eval\_batch\_size=16,
weight\_decay=0.01,
save\_total\_limit=3,
num\_train\_epochs=10,
fp16=True,
)
trainer = Seq2SeqTrainer(
model=model,
args=training\_args,
train\_dataset=tokenized\_luganda[“train”],
eval\_dataset=tokenized\_luganda[“test”],
tokenizer=tokenizer,
data\_collator=data\_collator,
)
trainer.train()
Traceback (most recent call last)
File “finetune\_luganda.py”, line 28, in
tokenized\_luganda = luganda\_dataset.map(preprocess\_function, batched=True)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/dataset\_dict.py”, line 512, in map
for k, dataset in self.items()
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/dataset\_dict.py”, line 512, in
for k, dataset in self.items()
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2120, in map
desc=desc,
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 518, in wrapper
out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 485, in wrapper
out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/fingerprint.py”, line 413, in wrapper
out = func(self, \*args, \*\*kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2485, in \_map\_single
offset=offset,
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2367, in apply\_function\_on\_filtered\_inputs
processed\_inputs = function(\*fn\_args, \*additional\_args, \*\*fn\_kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2062, in decorated
result = f(decorated\_item, \*args, \*\*kwargs)
File “finetune\_luganda.py”, line 20, in preprocess\_function
inputs = [prefix + example[source\_lang] for example in examples[“translation”]]
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 123, in **getitem**
values = super(). **getitem** (key)
File “/usr/lib/python3.6/collections/ **init**.py”, line 991, in **getitem**
raise KeyError(key)
KeyError: ‘translation’
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [March 21, 2022, 7:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-create-a-dataset-from-a-csv-file%2F15686%2F11 "2022-03-21T19:52:56Z")
You need to assign the actual value returned by `map`, so replace:
```python
luganda_dataset.map(lambda ex, i: {“id”: i, “translation”: dict(ex)}, remove_columns=[“en”, “lg”], features=datasets.Features({“id”: datasets.Value(“string”), “translation”: datasets
.Translation(languages=[“en”, “lg”])}), with_indices=True,)
```
with
```python
luganda_dataset = luganda_dataset.map(lambda ex, i: {“id”: i, “translation”: dict(ex)}, remove_columns=[“en”, “lg”], features=datasets.Features({“id”: datasets.Value(“string”), “translation”: datasets
.Translation(languages=[“en”, “lg”])}), with_indices=True,)
```
---
**Author:**  [@Terence](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTerence)%5C
**Post date:** [March 23, 2022, 11:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-create-a-dataset-from-a-csv-file%2F15686%2F12 "2022-03-23T11:01:25Z")
Yes, that was my silly error, sorry ☹ The script now goes further but throws the following: TypeError: must be str, not NoneType
at Line 16. The only significant difference with my script for the “opus books” dataset which worked perfectly is that here I am reading in a csv file.
Traceback (most recent call last):
File “finetune\_luganda.py”, line 25, in
tokenized\_luganda = luganda\_dataset.map(preprocess\_function, batched=True)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/dataset\_dict.py”, line 512, in map
for k, dataset in self.items()
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/dataset\_dict.py”, line 512, in
for k, dataset in self.items()
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2120, in map
desc=desc,
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 518, in wrapper
out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 485, in wrapper
out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/fingerprint.py”, line 413, in wrapper
out = func(self, \*args, \*\*kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2485, in \_map\_single
offset=offset,
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2367, in apply\_function\_on\_filtered\_inputs
processed\_inputs = function(\*fn\_args, \*additional\_args, \*\*fn\_kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2062, in decorated
result = f(decorated\_item, \*args, \*\*kwargs)
File “finetune\_luganda.py”, line 16, in preprocess\_function
inputs = [prefix + example[source\_lang] for example in examples[“translation”]]
File “finetune\_luganda.py”, line 16, in
inputs = [prefix + example[source\_lang] for example in examples[“translation”]]
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [March 23, 2022, 1:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-create-a-dataset-from-a-csv-file%2F15686%2F13 "2022-03-23T13:39:12Z")
Okay, this means you have `None` values under the `source_lang` key, which can happen if your CSV file is missing English translations in some lines, i.e., you have lines like this:
```auto
,
```
---
**Author:**  [@Terence](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTerence)%5C
**Post date:** [March 24, 2022, 1:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-create-a-dataset-from-a-csv-file%2F15686%2F14 "2022-03-24T13:26:14Z")
OK, that has been resolved. The script now moves on further to produce the following error:
“Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.”
I have no idea what input is invalid here. As mentioned at the outset the script worked with the “opus\_books” dataset.
Traceback (most recent call last):
File “finetune\_luganda.py”, line 25, in
tokenized\_luganda = luganda\_dataset.map(preprocess\_function, batched=True)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/dataset\_dict.py”, line 512, in map
for k, dataset in self.items()
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/dataset\_dict.py”, line 512, in
for k, dataset in self.items()
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2120, in map
desc=desc,
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 518, in wrapper
out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 485, in wrapper
out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/fingerprint.py”, line 413, in wrapper
out = func(self, \*args, \*\*kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2485, in \_map\_single
offset=offset,
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2367, in apply\_function\_on\_filtered\_inputs
processed\_inputs = function(\*fn\_args, \*additional\_args, \*\*fn\_kwargs)
File “/home/tel34/venv/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2062, in decorated
result = f(decorated\_item, \*args, \*\*kwargs)
File “finetune\_luganda.py”, line 21, in preprocess\_function
labels = tokenizer(targets, max\_length=128, truncation=True)
File “/home/tel34/venv/lib/python3.6/site-packages/transformers/tokenization\_utils\_base.py”, line 2420, in **call**
\*\*kwargs,
File “/home/tel34/venv/lib/python3.6/site-packages/transformers/tokenization\_utils\_base.py”, line 2605, in batch\_encode\_plus
\*\*kwargs,
File “/home/tel34/venv/lib/python3.6/site-packages/transformers/tokenization\_utils.py”, line 690, in \_batch\_encode\_plus
first\_ids = get\_input\_ids(ids)
File “/home/tel34/venv/lib/python3.6/site-packages/transformers/tokenization\_utils.py”, line 671, in get\_input\_ids
“Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.”
---
**Author:**  [@Terence](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTerence)%5C
**Post date:** [March 25, 2022, 9:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-create-a-dataset-from-a-csv-file%2F15686%2F15 "2022-03-25T09:06:27Z")
The script now works with a modification by Yasmin Moslem. I post it below in case others have a simil;ar issue:
```python
import datasets
from transformers import AutoTokenizer
from datasets import load_dataset
from transformers import DataCollatorForSeq2Seq
from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer
from random import randrange
luganda_dataset = load_dataset("csv", data_files="Luganda.csv")
luganda_dataset = luganda_dataset["train"].map(lambda ex, i: {"id": i, "translation": dict(ex)}, remove_columns=["en", "lg"], features=datasets.Features({"id": datasets.Value("string"), "translation": datasets
.Translation(languages=["en", "lg"])}), with_indices=True,)
luganda_dataset = luganda_dataset.train_test_split(test_size=0.2)
tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-lg")
source_lang = "en"
target_lang = "lg"
prefix = "translate English to Luganda: "
def preprocess_function(examples):
inputs = []
targets = []
for example in examples["translation"]:
if example[source_lang] is not None and example[target_lang] is not None and \
len(example[source_lang].strip()) > 3 and len(example[target_lang].strip()) > 3:
inputs.append(prefix + example[source_lang].strip())
targets.append(example[target_lang].strip())
else:
"There is an issue with this segment:"
print("Source:", example[source_lang])
print("Target:", example[target_lang])
random_num = randrange(10000)
print("Replaced with", random_num)
inputs.append(prefix + str(random_num))
targets.append(str(random_num))
model_inputs = tokenizer(inputs, max_length=128, truncation=True)
with tokenizer.as_target_tokenizer():
labels = tokenizer(targets, max_length=128, truncation=True)
model_inputs["labels"] = labels["input_ids"]
return model_inputs
print(luganda_dataset.map)
tokenized_luganda = luganda_dataset.map(preprocess_function, batched=True)
model = AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-en-lg")
data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)
training_args = Seq2SeqTrainingArguments(
output_dir="./results",
evaluation_strategy="epoch",
learning_rate=2e-5,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
weight_decay=0.01,
save_total_limit=3,
num_train_epochs=10,
fp16=True,
)
trainer = Seq2SeqTrainer(
model=model,
args=training_args,
train_dataset=tokenized_luganda["train"],
eval_dataset=tokenized_luganda["test"],
tokenizer=tokenizer,
data_collator=data_collator,
)
trainer.train()
```