# Saving a dataset to disk after select copies the data
**URL:**
**Author:**  [@PaulLerner](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPaulLerner)%5C
**Post date:** [March 30, 2022, 2:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-a-dataset-to-disk-after-select-copies-the-data%2F16262%2F1 "2022-03-30T14:02:48Z")
Hi,
As you can see in [datasets/arrow\_dataset.py at 2.0.0 Β· huggingface/datasets Β· GitHub](https://github.com/huggingface/datasets/blob/2.0.0/src/datasets/arrow_dataset.py#L2541) when selecting indices from dataset A for dataset B, it keeps the same data as A. I guess this is the expected behavior so I did not open an issue.
The problem is when saving the dataset B to disk, since the data of A was not filtered, the whole data is saved to disk.
This is problematic in my use case: training and test splits (I am aware of the train\_test\_split method but I need some specific sampling). Indeed, if I have my original dataset A and I split it in TRAIN and TEST using select, then save them using save\_to\_disk, my data will be duplicated: in both TRAIN and TEST.
Any idea on how to fix this/what am I supposed to do to save a dataset after selection without copying the whole data ?
Bests,
Paul
---
**Author:**  [@PaulLerner](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPaulLerner)%5C
**Post date:** [March 30, 2022, 2:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-a-dataset-to-disk-after-select-copies-the-data%2F16262%2F2 "2022-03-30T14:24:20Z")
Actually it seems that `train_test_split` also uses `select` [datasets/arrow\_dataset.py at 2.0.0 Β· huggingface/datasets Β· GitHub](https://github.com/huggingface/datasets/blob/2.0.0/src/datasets/arrow_dataset.py#L2980) so it must have the same problem?
---
**Author:**  [@PaulLerner](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPaulLerner)%5C
**Post date:** [March 30, 2022, 2:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-a-dataset-to-disk-after-select-copies-the-data%2F16262%2F3 "2022-03-30T14:41:22Z")
Found a (not so satisfying) work-around: `d = d.filter(lambda x: True)` before `d.save_to_disk`
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [March 30, 2022, 4:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-a-dataset-to-disk-after-select-copies-the-data%2F16262%2F4 "2022-03-30T16:02:38Z")
Hi! The same data shouldnβt be saved twice because `save_to_disk` calls `flatten_indices` to save only the selected rows (referenced by the `_indices` mapping) from the table. Why do you think thatβs not the case?
---
**Author:**  [@PaulLerner](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPaulLerner)%5C
**Post date:** [March 30, 2022, 5:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-a-dataset-to-disk-after-select-copies-the-data%2F16262%2F5 "2022-03-30T17:19:10Z")
No idea π€
```auto
$ datasets-cli env
- `datasets` version: 1.8.0
- Platform: Linux-4.18.0-305.40.2.el8_4.x86_64-x86_64-with-redhat-8.4-Ootpa
- Python version: 3.7.11
- PyArrow version: 3.0.0
```
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [March 30, 2022, 5:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-a-dataset-to-disk-after-select-copies-the-data%2F16262%2F6 "2022-03-30T17:22:42Z")
Could you please try with the newest version of `datasets` and report back? It can be installed as follows:
```bash
pip install -U datasets
```
---
**Author:**  [@PaulLerner](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPaulLerner)%5C
**Post date:** [April 6, 2022, 12:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-a-dataset-to-disk-after-select-copies-the-data%2F16262%2F7 "2022-04-06T12:43:02Z")
Even worse! With `datasets 2.0.0`, if I load the previously saved subset, it loads the whole dataset instead of the selected indices.
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [April 7, 2022, 12:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-a-dataset-to-disk-after-select-copies-the-data%2F16262%2F8 "2022-04-07T12:10:52Z")
Can you share the reproducer? Feel free to replace the original data with dummy data (to keep it private).
---
**Author:**  [@PaulLerner](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPaulLerner)%5C
**Post date:** [April 7, 2022, 1:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsaving-a-dataset-to-disk-after-select-copies-the-data%2F16262%2F9 "2022-04-07T13:23:18Z")
Never mind, I did not want to update because of the issue on the dataset that I had already saved, but updating to `datasets 1.18.3` solves the issue (probably future versions as well, I tried `1.18.3` because it was already installed on another machine).
Thank you for your help!
See for instance:
```nohighlight
In [2]: from datasets import Dataset
In [3]: d = Dataset.from_dict({'foo':[1]*10000})
In [4]: d
Out[4]:
Dataset({
features: ['foo'],
num_rows: 10000
})
In [5]: d.save_to_disk('foo')
In [7]: ls -lh foo
total 88K
-rw-rw-r-- 1 lerner lerner 79K avril 7 15:18 dataset.arrow
-rw-rw-r-- 1 lerner lerner 480 avril 7 15:18 dataset_info.json
-rw-rw-r-- 1 lerner lerner 253 avril 7 15:18 state.json
In [8]: d=d.select([0,1])
In [9]: d
Out[9]:
Dataset({
features: ['foo'],
num_rows: 2
})
In [10]: d.save_to_disk('bar')
Flattening the indices: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1/1 [00:00<00:00, 25.38ba/s]
In [11]: ls -lsh bar
total 12K
4,0K -rw-rw-r-- 1 lerner lerner 440 avril 7 15:19 dataset.arrow
4,0K -rw-rw-r-- 1 lerner lerner 480 avril 7 15:19 dataset_info.json
4,0K -rw-rw-r-- 1 lerner lerner 253 avril 7 15:19 state.json
```