# How do I add things (rows) to an already saved dataset?
**URL:**
**Author:**  [@shahbuland](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fshahbuland)%5C
**Post date:** [December 8, 2022, 12:14am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-add-things-rows-to-an-already-saved-dataset%2F27423%2F1 "2022-12-08T00:14:54Z")
I save a dataset to disk. Later, I load it from disk in a different script. I add some items. I try to save to disk again. I get the error “PermissionError: Tried to overwrite [path]/dataset.arrow but a dataset can’t overwrite itself.”. How do I solve this? What is the correct way to add new rows the dataset on disk?
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [December 13, 2022, 1:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-add-things-rows-to-an-already-saved-dataset%2F27423%2F2 "2022-12-13T13:59:44Z")
Hi! Our library relies on PyArrow as a storage backend. PyArrow tables are immutable, so adding new rows means creating a new table, and since `save_to_disk` saves the entire table, saving it over and over could occupy a lot of disk space. So a more optimal approach here would be to save each sub-dataset (new items equals one sub-dataset) individually and load + concatenate (`concatenate_datasets`) the sub-datasets later.
Regarding the PermissionError, this happens when you try to save a dataset to a location already used by some of its sub-tables (due to PyArrow being immutable, new rows are kept in memory as sub-tables before being saved to disk). You can easily avoid this by saving the dataset to some other location.
PS: In the future, we could optimize appending and saving new rows by storing these rows in a separate file, which we would then put inside the existing save directory.
---
**Author:**  [@shahbuland](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fshahbuland)%5C
**Post date:** [December 13, 2022, 9:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-add-things-rows-to-an-already-saved-dataset%2F27423%2F3 "2022-12-13T21:14:17Z")
Oh ok makes sense. I think in my use-case I may as well just use CSV files since there is realistically not going to be that much data (maybe 10 columns of text, max 100k rows)
---
**Author:**  [@GraemeSmith](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGraemeSmith)%5C
**Post date:** [June 11, 2023, 2:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-add-things-rows-to-an-already-saved-dataset%2F27423%2F4 "2023-06-11T14:22:06Z")
This feels embarrassingly stupid but what is the standard way to replace a saved to disk dataset with a modified/updated dataset. The obvious way would seem to be to save the modified dataset with a temp name, delete the original and then rename the temp to the original. But Python seems to keep hold of files in the original dataset and I can’t work out how to release them so that I can delete the dataset directory.
---
**Author:**  [@dataPlumber](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FdataPlumber)%5C
**Post date:** [April 11, 2024, 12:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-add-things-rows-to-an-already-saved-dataset%2F27423%2F5 "2024-04-11T12:43:14Z")
I would also like to know this.
---
**Author:**  [@Butanium](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FButanium)%5C
**Post date:** [August 5, 2024, 5:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-add-things-rows-to-an-already-saved-dataset%2F27423%2F6 "2024-08-05T17:15:42Z")
Bump
I guess I’ll just create bunch of datasets for now
My use case is to add a column when I make my experiment on a new model
---
**Author:**  [@severo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsevero)%5C
**Post date:** [August 8, 2024, 10:14am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-add-things-rows-to-an-already-saved-dataset%2F27423%2F7 "2024-08-08T10:14:19Z")
> add a column when I make my experiment on a new model
Interesting. Which format do you use to store your data?
---
**Author:**  [@Butanium](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FButanium)%5C
**Post date:** [August 8, 2024, 2:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-add-things-rows-to-an-already-saved-dataset%2F27423%2F8 "2024-08-08T14:59:05Z")
I just store the llm next token prediction as an extra label
---
**Author:**  [@severo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsevero)%5C
**Post date:** [August 8, 2024, 3:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-add-things-rows-to-an-already-saved-dataset%2F27423%2F9 "2024-08-08T15:14:26Z")
OK, but which file format? do you have an example dataset to share?
---
**Author:**  [@Butanium](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FButanium)%5C
**Post date:** [August 8, 2024, 3:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-add-things-rows-to-an-already-saved-dataset%2F27423%2F10 "2024-08-08T15:32:14Z")
I just use hf datasets `save_to_disk`.
My dataset is some texts from fineweb and then for each model I add a last token id column and a last token predicted column. E.g.
`text="I| love| Ber|kley", gt_gpt2={kley token id}, pred_gpt2={lin token id}` (using | to show how the model would tokenize the string)