return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
File "/home/ubuntu/venv/lib/python3.10/site-packages/datasets/table.py", line 2176, in embed_array_storage
return feature.embed_storage(array)
File "/home/ubuntu/venv/lib/python3.10/site-packages/datasets/features/image.py", line 276, in embed_storage
storage = pa.StructArray.from_arrays([bytes_array, path_array], ["bytes", "path"], mask=bytes_array.is_null())
File "pyarrow/array.pxi", line 2850, in pyarrow.lib.StructArray.from_arrays
File "pyarrow/array.pxi", line 3290, in pyarrow.lib.c_mask_inverted_from_obj
TypeError: Mask must be a pyarrow.Array of type boolean
```
Bonus:
If I save the dataset generated with the generator to local disk. Then load it and upload it like:
```auto
dataset = Dataset.from_generator(dataset_gen, features=features)
dataset.save_to_disk("/path/to/db")
dataset = datasets.load_from_disk("/path/to/db")
dataset.push_to_hub(dataset_id)
```
It uploads shards of ~1.2GB but again I reach the git commit quota
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [October 24, 2023, 9:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fany-workaround-for-push-to-hub-limits%2F59274%2F4 "2023-10-24T09:01:14Z")
Good news, we improved how push\_to\_hub works : it does only one commit every 50 files instead of every file. The improvement is available if you install `datasets` from source, and it will be available in 2.15 as well
Feel free to try it out
---
**Author:**  [@ocg2347](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Focg2347)%5C
**Post date:** [October 25, 2023, 9:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fany-workaround-for-push-to-hub-limits%2F59274%2F5 "2023-10-25T09:09:03Z")
This is great news! Thank you all for that. It seems I managed to solve this π
Feedback for additional improvement can be:
- adaptive upload frequency that makes sure commit quota limit is not reached.
- and of course, better implementation of sharding so that user can upload large shards without any problem
Again, thanks a lot π
---
**Author:**  [@zeio](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzeio)%5C
**Post date:** [December 10, 2023, 1:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fany-workaround-for-push-to-hub-limits%2F59274%2F6 "2023-12-10T13:19:36Z")
Have the same problem. One commit per 50 files is still too frequent for me. Canβt customize shard size or number of shards due to the error. Using the datasets version β2.15.0β.
Most likely it is caused by around 1% of entires in the beginning of the dataset being much larger than the rest of the entries. As a result, in the beginning of pushing to the hub, the commits are relatively rare and the shard size is reasonably high (around 400-500 MB), for most of the entries the commits are too frequent and the shard size is extremely small (around 500 KB - 2 MB).
Apparently, the pusher somehow infers the optimal number of entries per shard in the beginning and doesnβt adjust it in the process, which is quite a hard assumption in my case.
---
**Author:**  [@zeio](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzeio)%5C
**Post date:** [December 10, 2023, 8:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fany-workaround-for-push-to-hub-limits%2F59274%2F7 "2023-12-10T20:03:40Z")
Solved for now by setting max shard size to 700 MB:
`batch.push_to_hub('zeio/auto-batch', config_name = 'spoken', max_shard_size = '700MB')`
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [December 12, 2023, 1:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fany-workaround-for-push-to-hub-limits%2F59274%2F8 "2023-12-12T13:36:13Z")
> [@zeio](#):
>
> Apparently, the pusher somehow infers the optimal number of entries per shard in the beginning and doesnβt adjust it in the process, which is quite a hard assumption in my case.
Thatβs correct. And indeed in this case having extremely small (or big) examples at the beginning of the dataset can cause the shards to not have the same sizes. This can indeed be improved though
---
**Author:**  [@buaa42wxy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbuaa42wxy)%5C
**Post date:** [May 2, 2024, 5:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fany-workaround-for-push-to-hub-limits%2F59274%2F9 "2024-05-02T05:15:21Z")
any other solution? same problem, but 700MB not work
---
**Author:**  [@buaa42wxy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbuaa42wxy)%5C
**Post date:** [May 2, 2024, 5:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fany-workaround-for-push-to-hub-limits%2F59274%2F10 "2024-05-02T05:16:04Z")
```auto
Mask must be a pyarrow.Array of type boolean
File "/home/huangxijie/MedMLLM_attack/hf_dataset.py", line 14, in
dataset.push_to_hub("MedMLLM-attack/3MAD-24K", max_shard_size="300MB")
TypeError: Mask must be a pyarrow.Array of type boolean
```
```auto
import pandas as pd
from datasets import Dataset,Image
# Read the CSV file
df = pd.read_csv("MedMQ-2k/metadata.csv")
# Create a Hugging Face Dataset
dataset = Dataset.from_pandas(df)
dataset = dataset.map(lambda example: {"image": example["file_name"]}, batched=True)
# Convert the file_name column to Image type
dataset = dataset.cast_column("image", Image())
# Upload to Hugging Face Hub (make sure authentication is set up)
dataset.push_to_hub("MedMLLM-attack/3MAD-24K", max_shard_size="300MB")
```