# Loading multiple serialized datasets with \`multiprocessing\`
**URL:**
**Author:**  [@carted-ml](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcarted-ml)%5C
**Post date:** [March 25, 2022, 10:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-multiple-serialized-datasets-with-multiprocessing%2F16065%2F1 "2022-03-25T10:18:07Z")
If I have multiple serialized datasets then is it a good practice to use `multiprocessing` along with `datasets.load_dataset()`?
Something like so:
```nohighlight
import multiprocessing
import datasets
import gcsfs
BUCKET_NAME = "my-bucket"
GCS_FS = gcsfs.GCSFileSystem()
def load_ds(ds_path):
return datasets.load_from_disk(ds_path, fs=GCS_FS)
ds_dirs = GCS_FS.listdir(f"{BUCKET_NAME}/saved_datasets")
ds_dirs = list(
{
f"{dd['name']}"
for dd in ds_dirs
if "tokenized" in dd["name"]
}
)
with multiprocessing.Pool() as pool:
ds_list = pool.starmap_async(load_ds, ds_dirs).get()
ds_list = [ds for ds in ds_list]
train_ds = datasets.concatenate_datasets(ds_list)
print(train_ds)
```
Running this snippet does not execute as expected though.
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [March 31, 2022, 12:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-multiple-serialized-datasets-with-multiprocessing%2F16065%2F2 "2022-03-31T12:22:11Z")
Hi! Could you please explain why this snippet “does not execute as expected” or provide the error stack trace if there is one?
---
**Author:**  [@carted-ml](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcarted-ml)%5C
**Post date:** [April 2, 2022, 5:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-multiple-serialized-datasets-with-multiprocessing%2F16065%2F3 "2022-04-02T05:30:30Z")
By "does not execute as expected”, I meant it does not execute at all.