# Iterating on dataset extremely slow **URL:** **Author:** ![zhh210](https://avatars.discourse-cdn.com/v4/letter/z/ce73a5/32.png) [@zhh210](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzhh210)%5C **Post date:** [June 18, 2024, 5:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fiterating-on-dataset-extremely-slow%2F92813%2F1 "2024-06-18T17:00:59Z") I’ve tried a few different approaches to iterate on 1M records with datasets but all seem quite [slow](https://github.com/huggingface/datasets/issues/6279). So this time I’m trying the recommended streaming method from the recent [post](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspeeding-up-streaming-of-large-datasets-fineweb%2F90714) but still it’s extremely slow. It seems for dataset there will always be multiple ways to iterate but most (or all) of them will not yield satisfiable speed. ```auto import s3fs s3 = s3fs.S3FileSystem() import datasets as hfdt from torch.utils.data import DataLoader SEED = 0 SAMPLE_BUFFER_SIZE=5_000 RECORDS_TO_KEEP= 100_000 TAKE_SIZE = 1_000_000 dt_train = hfdt.load_dataset('parquet', data_files = ['s3://' + i for i in s3.glob('s3://my_s3_bucket/*')], streaming=True) dt_train = dt_train['train'].shuffle(seed=SEED, buffer_size=SAMPLE_BUFFER_SIZE) import torch import pandas as pd def custom_collate_fn(batch): # batch = list(filter(lambda x: x is not None, batch)) # Remove None values batch = [0 for i in batch] return torch.utils.data.dataloader.default_collate(batch) # Use the default collate function dl_train = DataLoader(dt_train.take(TAKE_SIZE), num_workers=8, prefetch_factor=2, batch_size=128, collate_fn=custom_collate_fn) from tqdm import tqdm for sample in tqdm(dl_train): pass ``` The s3 folder has 200 parquet files and in total it has 1M rows and 28 columns. However, for such a small dataset, it takes forever to iterate on the data: ```auto Resolving data files: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 200/200 [00:07<00:00, 27.81it/s] 5381it [1:24:45, 1.25it/s] ``` It took almost 2 hours to finish: ```auto real 118m42.879s user 421m2.584s sys 0m55.513s ``` I’m wondering if HF dataset is really feasible for moderate sized dataset? ---
**Author:** ![FelixLabelle](https://avatars.discourse-cdn.com/v4/letter/f/49beb7/32.png) [@FelixLabelle](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FFelixLabelle)%5C **Post date:** [June 20, 2024, 11:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fiterating-on-dataset-extremely-slow%2F92813%2F2 "2024-06-20T11:57:47Z")
@zhh210 The post you referenced was mine and I’m able to stream 900+ records per second from the FineWeb dataset while inferencing over the text read in with a linear model. What type of data are you loading (images, text, tabular) and how big are these files? If increasing the number of workers didn’t increase speed that might be a sign that the bottleneck is the download (or upload on the other end) speed ---
**Author:** ![zhh210](https://avatars.discourse-cdn.com/v4/letter/z/ce73a5/32.png) [@zhh210](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzhh210)%5C **Post date:** [June 20, 2024, 5:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fiterating-on-dataset-extremely-slow%2F92813%2F3 "2024-06-20T17:36:18Z")
@FelixLabelle It was a surprise to me as well. My parquet files (.snappy.parquet) are tabular data with only numeric values and each of the 200 parquet files is just 1.3M. ---
**Author:** ![FelixLabelle](https://avatars.discourse-cdn.com/v4/letter/f/49beb7/32.png) [@FelixLabelle](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FFelixLabelle)%5C **Post date:** [June 20, 2024, 6:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fiterating-on-dataset-extremely-slow%2F92813%2F4 "2024-06-20T18:44:10Z")
Ok, in that case I would just try substitutions to debug potential sources of the issue. My guesses of causes in order of likelihood are 1. Upload speed limitations (the servers you are downloading from) 2. (tied for first) Download speed limitations 3. Something specific to your data 4. Something in your code which is adding time unbeknownst to you For 1. you can try downloading similar datasets from HF directly and see if you still have the issue. If you don’t I would check upload speed. I don’t use AWS, but certain storage methods can be slower to get data from and would be a bottleneck. This test could also be indicative for cause 3. If it fails and upload looks OK, I would try streaming a dummy dataset that’s much smaller and seeing if that’s slow as well. If it, then I would start stripping away your dataset to see if it’s a specific column or number of columns causing an issue. For 2. if you are using a computer just run a download speed test in your browser. I would also turn off any VPNs if you can, those can add time Cause 4 can be profiled, I would try that if you haven’t already. That would likely be required anyway if you’ve gotten this far debugging. If that doesn’t narrow down the problem LMK, we can look at the other causes ---
**Author:** ![zhh210](https://avatars.discourse-cdn.com/v4/letter/z/ce73a5/32.png) [@zhh210](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzhh210)%5C **Post date:** [June 21, 2024, 4:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fiterating-on-dataset-extremely-slow%2F92813%2F5 "2024-06-21T16:54:48Z")
Tried loading the dataset from local instead of from s3, it does reduce the running time from 2 hours to 1 hour: ```auto CPU times: user 9.21 s, sys: 2.87 s, total: 12.1 s Wall time: 1h 51s ``` profiling shows that most of the time were spent on `select.poll()`, so i guess this means it’s slow reading .snappy.parquet file: ```auto 7811it [2:33:58, 1.18s/it] 3624330 function calls (3624329 primitive calls) in 9238.469 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.899 0.899 9238.469 9238.469 415229234.py:3(run_local) 7871 0.022 0.000 0.030 0.000 :404(parent) 8357 0.012 0.000 0.019 0.000 :1() 1 0.000 0.000 9238.469 9238.469 :1() 7812 0.003 0.000 0.003 0.000 __init__.py:127(annotate) 8 0.000 0.000 0.000 0.000 __init__.py:219(_acquireLock) 8 0.000 0.000 0.000 0.000 __init__.py:228(_releaseLock) 1 0.000 0.000 0.000 0.000 __init__.py:9(is_available) 7812 0.004 0.000 0.004 0.000 _jit_internal.py:1120(is_scripting) 1 0.000 0.000 0.000 0.000 _monitor.py:94(report) 7812 0.015 0.000 1.027 0.000 _ops.py:591( __call__ ) 7812 0.018 0.000 0.479 0.000 _ops.py:846( __call__ ) 7811 0.048 0.000 0.409 0.000 _utils.py:178(_rebuild_tensor) 1 0.000 0.000 0.000 0.000 _weakrefset.py:111(remove) 2 0.000 0.000 0.000 0.000 _weakrefset.py:17( __init__ ) 2 0.000 0.000 0.000 0.000 _weakrefset.py:21( __enter__ ) 2 0.000 0.000 0.000 0.000 _weakrefset.py:27( __exit__ ) 8 0.000 0.000 0.000 0.000 _weakrefset.py:39(_remove) 2 0.000 0.000 0.000 0.000 _weakrefset.py:53(_commit_removals) 3 0.000 0.000 0.000 0.000 _weakrefset.py:63( __iter__ ) 17 0.000 0.000 0.000 0.000 _weakrefset.py:86(add) 4 0.000 0.000 0.000 0.000 abc.py:117( __instancecheck__ ) 7829 0.025 0.000 0.025 0.000 connection.py:117( __init__ ) 7813 0.005 0.000 0.005 0.000 connection.py:130( __del__ ) 87005 0.050 0.000 0.050 0.000 connection.py:134(_check_closed) 39601 0.017 0.000 0.017 0.000 connection.py:138(_check_readable) 31244 0.017 0.000 0.017 0.000 connection.py:142(_check_writable) 16160 0.019 0.000 0.026 0.000 connection.py:168(fileno) 7811 0.020 0.000 0.105 0.000 connection.py:173(close) 23433 0.150 0.000 0.580 0.000 connection.py:181(send_bytes) 7811 0.041 0.000 0.455 0.000 connection.py:202(send) 31252 0.118 0.000 96.117 0.003 connection.py:208(recv_bytes) 8349 0.038 0.000 9091.500 1.089 connection.py:253(poll) 7811 0.003 0.000 0.003 0.000 connection.py:259( __enter__ ) 7811 0.014 0.000 0.118 0.000 connection.py:262( __exit__ ) 7813 0.011 0.000 0.085 0.000 connection.py:360(_close) 31244 0.052 0.000 0.355 0.000 connection.py:365(_send) 62504 0.251 0.000 95.719 0.002 connection.py:374(_recv) 31244 0.096 0.000 0.484 0.000 connection.py:390(_send_bytes) 31252 0.168 0.000 95.958 0.003 connection.py:413(_recv_bytes) 8349 0.045 0.000 9091.452 1.089 connection.py:423(_poll) 7811 0.074 0.000 97.772 0.013 connection.py:493(Client) 9 0.000 0.000 0.001 0.000 connection.py:516(Pipe) 7811 0.108 0.000 0.597 0.000 connection.py:623(SocketClient) 7811 0.096 0.000 31.872 0.004 connection.py:732(deliver_challenge) 7811 0.128 0.000 65.169 0.008 connection.py:747(answer_challenge) 7811 0.017 0.000 0.017 0.000 connection.py:83(_validate_family) 8357 0.136 0.000 9091.431 1.088 connection.py:917(wait) 7827 0.011 0.000 0.011 0.000 connection.py:933() 15622 0.035 0.000 0.050 0.000 connection.py:95(address_type) 9 0.001 0.000 0.033 0.004 context.py:100(Queue) 33 0.000 0.000 0.000 0.000 context.py:187(get_context) 32 0.000 0.000 0.000 0.000 context.py:197(get_start_method) 8 0.000 0.000 0.076 0.009 context.py:222(_Popen) 18 0.000 0.000 0.000 0.000 context.py:237(get_context) 8 0.001 0.000 0.076 0.009 context.py:278(_Popen) 19 0.000 0.000 0.023 0.001 context.py:65(Lock) 1 0.000 0.000 0.000 0.000 context.py:75(Condition) 4 0.000 0.000 0.000 0.000 context.py:80(Semaphore) 9 0.000 0.000 0.001 0.000 context.py:85(BoundedSemaphore) 1 0.000 0.000 0.000 0.000 context.py:90(Event) 9 0.000 0.000 0.000 0.000 dataloader.py:1082() 1 0.000 0.000 0.066 0.066 dataloader.py:1087(_reset) 1 0.000 0.000 0.000 0.000 dataloader.py:1103() 8349 0.047 0.000 9230.746 1.106 dataloader.py:1120(_try_get_data) 7819 0.029 0.000 9230.775 1.181 dataloader.py:1266(_get_data) 7812 0.119 0.000 9231.913 1.182 dataloader.py:1299(_next_data) 7835 0.066 0.000 1.019 0.000 dataloader.py:1348(_try_put_index) 7811 0.019 0.000 0.976 0.000 dataloader.py:1368(_process_data) 8 0.000 0.000 0.000 0.000 dataloader.py:1375(_mark_worker_as_unavailable) 2 0.000 0.000 0.027 0.014 dataloader.py:1401(_shutdown_workers) 1 0.000 0.000 0.000 0.000 dataloader.py:1478( __del__ ) 1 0.000 0.000 0.180 0.180 dataloader.py:382(_get_iterator) 1 0.000 0.000 0.000 0.000 dataloader.py:389(multiprocessing_context) 1 0.000 0.000 0.180 0.180 dataloader.py:426( __iter__ ) 2 0.000 0.000 0.000 0.000 dataloader.py:441(_auto_collation) 1 0.000 0.000 0.000 0.000 dataloader.py:445(_index_sampler) 1 0.000 0.000 0.000 0.000 dataloader.py:457( __len__ ) 1 0.000 0.000 0.000 0.000 dataloader.py:486(check_worker_number_rationality) 1 0.000 0.000 0.001 0.001 dataloader.py:565( __init__ ) 1 0.000 0.000 0.000 0.000 dataloader.py:610(_reset) 7835 0.009 0.000 0.614 0.000 dataloader.py:620(_next_index) 7812 0.107 0.000 9233.713 1.182 dataloader.py:626( __next__ ) 1002881 0.153 0.000 0.153 0.000 dataloader.py:88( __iter__ ) 1 0.000 0.000 0.000 0.000 dataloader.py:93(_get_distributed_settings) 1 0.002 0.002 0.180 0.180 dataloader.py:991( __init__ ) 1 0.000 0.000 0.000 0.000 distributed_c10d.py:460(default_pg) 1 0.000 0.000 0.000 0.000 distributed_c10d.py:588(WORLD) 1 0.000 0.000 0.000 0.000 distributed_c10d.py:973(is_initialized) 1 0.000 0.000 0.000 0.000 functools.py:393( __get__ ) 15622 0.012 0.000 0.012 0.000 hmac.py:139(_current) 15622 0.028 0.000 0.089 0.000 hmac.py:151(digest) 15622 0.035 0.000 0.260 0.000 hmac.py:167(new) 15622 0.050 0.000 0.225 0.000 hmac.py:38( __init__ ) 15622 0.058 0.000 0.167 0.000 hmac.py:66(_init_hmac) 6160 0.010 0.000 0.010 0.000 iostream.py:138(_event_pipe) 6160 0.030 0.000 0.233 0.000 iostream.py:259(schedule) 1 0.000 0.000 0.000 0.000 iostream.py:364(fileno) 2043 0.005 0.000 0.006 0.000 iostream.py:505(parent_header) 2043 0.004 0.000 0.008 0.000 iostream.py:550(_is_master_process) 2043 0.008 0.000 0.136 0.000 iostream.py:577(_schedule_flush) 2059 0.029 0.000 3.042 0.001 iostream.py:592(flush) 2043 0.043 0.000 0.198 0.000 iostream.py:655(write) 8 0.000 0.000 0.002 0.000 ipkernel.py:768(init_closure) 2 0.000 0.000 0.000 0.000 ipkernel.py:775(_clean_thread_parent_frames) 1 0.000 0.000 0.000 0.000 ipkernel.py:790() 1 0.000 0.000 0.000 0.000 os.py:675( __getitem__ ) 1 0.000 0.000 0.000 0.000 os.py:755(encode) 8 0.000 0.000 0.075 0.009 popen_fork.py:15( __init__ ) 4281 0.006 0.000 0.024 0.000 popen_fork.py:24(poll) 8 0.000 0.000 0.027 0.003 popen_fork.py:36(wait) 8 0.001 0.000 0.049 0.006 popen_fork.py:62(_launch) 8 0.001 0.000 0.077 0.010 process.py:110(start) 8 0.000 0.000 0.027 0.003 process.py:142(join) 4245 0.013 0.000 0.041 0.000 process.py:153(is_alive) 8 0.000 0.000 0.000 0.000 process.py:189(name) 8 0.000 0.000 0.000 0.000 process.py:205(daemon) 7811 0.010 0.000 0.010 0.000 process.py:213(authkey) 8 0.000 0.000 0.000 0.000 process.py:234(ident) 7843 0.006 0.000 0.006 0.000 process.py:37(current_process) 8 0.000 0.000 0.000 0.000 process.py:61(_cleanup) 8 0.001 0.000 0.001 0.000 process.py:80( __init__ ) 16 0.000 0.000 0.000 0.000 process.py:94() 4269 0.002 0.000 0.002 0.000 process.py:99(_check_closed) 7812 0.037 0.000 0.054 0.000 profiler.py:593( __init__ ) 7812 0.038 0.000 0.517 0.000 profiler.py:604( __enter__ ) 7812 0.086 0.000 1.122 0.000 profiler.py:610( __exit__ ) 8 0.000 0.000 0.000 0.000 queues.py:140(close) 16 0.000 0.000 0.000 0.000 queues.py:153(cancel_join_thread) 8 0.001 0.000 0.058 0.007 queues.py:161(_start_thread) 8 0.000 0.000 0.000 0.000 queues.py:204(_finalize_close) 9 0.001 0.000 0.031 0.003 queues.py:37( __init__ ) 9 0.001 0.000 0.005 0.001 queues.py:71(_reset) 7842 0.060 0.000 0.335 0.000 queues.py:86(put) 8349 0.135 0.000 9230.658 1.106 queues.py:98(get) 32 0.001 0.000 0.010 0.000 random.py:506(choices) 32 0.009 0.000 0.009 0.000 random.py:519() 7811 0.129 0.000 38.008 0.005 reduction.py:153(recvfds) 7811 0.079 0.000 38.264 0.005 reduction.py:186(recv_handle) 7811 0.112 0.000 0.233 0.000 reduction.py:38( __init__ ) 7811 0.052 0.000 0.317 0.000 reduction.py:48(dumps) 7811 0.076 0.000 0.484 0.000 reductions.py:109(rebuild_tensor) 7811 0.038 0.000 0.043 0.000 reductions.py:32( __init__ ) 8256 0.007 0.000 0.014 0.000 reductions.py:45(expired) 15622 0.028 0.000 0.116 0.000 reductions.py:479(fd_id) 7764 0.005 0.000 0.012 0.000 reductions.py:48( __del__ ) 7811 0.017 0.000 0.063 0.000 reductions.py:487(storage_from_cache) 7811 0.135 0.000 137.586 0.018 reductions.py:494(rebuild_storage_fd) 7811 0.069 0.000 0.147 0.000 reductions.py:531(rebuild_typed_storage) 7811 0.037 0.000 0.046 0.000 reductions.py:76(get) 7811 0.030 0.000 0.075 0.000 reductions.py:80( __setitem__ ) 64 0.013 0.000 0.039 0.001 reductions.py:86(free_dead_references) 7811 0.072 0.000 136.913 0.018 resource_sharer.py:55(detach) 7811 0.167 0.000 98.456 0.013 resource_sharer.py:81(get_connection) 7837 0.443 0.000 0.596 0.000 sampler.py:274( __iter__ ) 8357 0.003 0.000 0.003 0.000 selectors.py:200( __enter__ ) 8357 0.015 0.000 0.053 0.000 selectors.py:203( __exit__ ) 8357 0.026 0.000 0.043 0.000 selectors.py:21(_fileobj_to_fd) 8357 0.028 0.000 0.037 0.000 selectors.py:210( __init__ ) 8357 0.013 0.000 0.055 0.000 selectors.py:216(_fileobj_lookup) 8357 0.057 0.000 0.131 0.000 selectors.py:235(register) 8357 0.032 0.000 0.038 0.000 selectors.py:269(close) 7827 0.015 0.000 0.015 0.000 selectors.py:276(_key_from_fd) 8357 0.035 0.000 0.080 0.000 selectors.py:348( __init__ ) 8357 0.041 0.000 0.179 0.000 selectors.py:352(register) 8357 0.150 0.000 9090.966 1.088 selectors.py:403(select) 8357 0.009 0.000 0.009 0.000 selectors.py:64( __init__ ) 1 0.000 0.000 0.000 0.000 signal_handling.py:47(_set_SIGCHLD_handler) 8 0.000 0.000 0.000 0.000 signal_handling.py:63(handler) 15622 0.193 0.000 0.193 0.000 socket.py:220( __init__ ) 15622 0.006 0.000 0.006 0.000 socket.py:236( __enter__ ) 15622 0.019 0.000 0.068 0.000 socket.py:239( __exit__ ) 7811 0.014 0.000 0.031 0.000 socket.py:494(_real_close) 7811 0.018 0.000 0.048 0.000 socket.py:498(close) 7811 0.027 0.000 0.032 0.000 socket.py:504(detach) 7811 0.023 0.000 0.098 0.000 socket.py:539(fromfd) 6160 0.159 0.000 0.159 0.000 socket.py:545(send) 2044 0.012 0.000 0.016 0.000 std.py:102(acquire) 2044 0.011 0.000 0.013 0.000 std.py:106(release) 4 0.000 0.000 0.000 0.000 std.py:110( __enter__ ) 4 0.000 0.000 0.000 0.000 std.py:113( __exit__ ) 1 0.000 0.000 0.000 0.000 std.py:1147( __del__ ) 2041 0.017 0.000 0.150 0.000 std.py:1150( __str__ ) 2 0.000 0.000 0.000 0.000 std.py:1153(_comparable) 2 0.000 0.000 0.000 0.000 std.py:1157( __hash__ ) 7812 0.050 0.000 9237.568 1.182 std.py:1160( __iter__ ) 2039 0.035 0.000 3.617 0.002 std.py:1198(update) 2 0.000 0.000 0.001 0.000 std.py:1265(close) 2 0.000 0.000 0.000 0.000 std.py:1286(fp_write) 1 0.000 0.000 0.000 0.000 std.py:1301() 2040 0.017 0.000 3.553 0.002 std.py:1325(refresh) 2041 0.021 0.000 0.028 0.000 std.py:1446(format_dict) 2041 0.014 0.000 3.508 0.002 std.py:1464(display) 3 0.000 0.000 0.000 0.000 std.py:226( __init__ ) 12204 0.034 0.000 0.034 0.000 std.py:231( __call__ ) 4081 0.028 0.000 0.031 0.000 std.py:400(format_interval) 1 0.000 0.000 0.002 0.002 std.py:438(status_printer) 2041 0.013 0.000 3.241 0.002 std.py:451(fp_write) 2041 0.013 0.000 3.341 0.002 std.py:457(print_status) 2041 0.061 0.000 0.106 0.000 std.py:464(format_meter) 1 0.000 0.000 0.000 0.000 std.py:663( __new__ ) 1 0.000 0.000 0.000 0.000 std.py:679(_get_free_pos) 1 0.000 0.000 0.000 0.000 std.py:682() 1 0.000 0.000 0.000 0.000 std.py:686(_decr_instances) 1 0.000 0.000 0.000 0.000 std.py:760(get_lock) 1 0.000 0.000 0.002 0.002 std.py:952( __init__ ) 8256 0.004 0.000 0.007 0.000 storage.py:1131(_expired) 7811 0.030 0.000 0.034 0.000 storage.py:509( __new__ ) 7811 0.039 0.000 0.044 0.000 storage.py:581( __init__ ) 7764 0.004 0.000 0.007 0.000 storage.py:974(_free_weak_ref) 32 0.000 0.000 0.011 0.000 synchronize.py:114(_make_name) 4 0.000 0.000 0.000 0.000 synchronize.py:125( __init__ ) 9 0.000 0.000 0.001 0.000 synchronize.py:144( __init__ ) 19 0.000 0.000 0.022 0.001 synchronize.py:161( __init__ ) 1 0.000 0.000 0.000 0.000 synchronize.py:212( __init__ ) 9 0.000 0.000 0.000 0.000 synchronize.py:229( __enter__ ) 9 0.000 0.000 0.000 0.000 synchronize.py:232( __exit__ ) 1 0.000 0.000 0.000 0.000 synchronize.py:235(_make_methods) 1 0.000 0.000 0.000 0.000 synchronize.py:270(notify) 1 0.000 0.000 0.000 0.000 synchronize.py:296(notify_all) 1 0.000 0.000 0.000 0.000 synchronize.py:323( __init__ ) 8 0.000 0.000 0.000 0.000 synchronize.py:327(is_set) 1 0.000 0.000 0.000 0.000 synchronize.py:334(set) 32 0.010 0.000 0.023 0.001 synchronize.py:50( __init__ ) 32 0.000 0.000 0.000 0.000 synchronize.py:90(_make_methods) 9 0.000 0.000 0.000 0.000 synchronize.py:94( __enter__ ) 9 0.000 0.000 0.000 0.000 synchronize.py:97( __exit__ ) 32 0.000 0.000 0.000 0.000 tempfile.py:142(rng) 32 0.001 0.000 0.011 0.000 tempfile.py:153( __next__ ) 8219 0.010 0.000 0.018 0.000 threading.py:1102(_wait_for_tstate_lock) 4143 0.005 0.000 0.005 0.000 threading.py:1145(ident) 8219 0.021 0.000 0.041 0.000 threading.py:1169(is_alive) 8 0.000 0.000 0.000 0.000 threading.py:1183(daemon) 8 0.000 0.000 0.000 0.000 threading.py:1198(daemon) 8 0.000 0.000 0.000 0.000 threading.py:1301(_make_invoke_excepthook) 2068 0.005 0.000 0.006 0.000 threading.py:1430(current_thread) 1 0.000 0.000 0.000 0.000 threading.py:1478(enumerate) 2076 0.025 0.000 0.025 0.000 threading.py:236( __init__ ) 9917 0.014 0.000 0.021 0.000 threading.py:264( __enter__ ) 9917 0.024 0.000 0.027 0.000 threading.py:267( __exit__ ) 2053 0.002 0.000 0.003 0.000 threading.py:273(_release_save) 2053 0.003 0.000 0.006 0.000 threading.py:276(_acquire_restore) 9903 0.011 0.000 0.018 0.000 threading.py:279(_is_owned) 2053 0.019 0.000 2.874 0.001 threading.py:288(wait) 7850 0.039 0.000 0.177 0.000 threading.py:359(notify) 2067 0.007 0.000 0.034 0.000 threading.py:545( __init__ ) 8236 0.003 0.000 0.003 0.000 threading.py:553(is_set) 2067 0.014 0.000 2.906 0.001 threading.py:589(wait) 8 0.001 0.000 0.002 0.000 threading.py:827( __init__ ) 8 0.000 0.000 0.056 0.007 threading.py:916(start) 7812 0.014 0.000 0.014 0.000 typing.py:306(inner) 1 0.000 0.000 0.000 0.000 tz.py:218(utcoffset) 1 0.000 0.000 0.000 0.000 tz.py:262(_isdst) 41 0.000 0.000 0.001 0.000 util.py:171(register_after_fork) 16 0.001 0.000 0.001 0.000 util.py:186( __init__ ) 16 0.000 0.000 0.002 0.000 util.py:205( __call__ ) 8 0.000 0.000 0.025 0.003 util.py:433(_flush_std_streams) 16 0.000 0.000 0.000 0.000 util.py:44(sub_debug) 8 0.000 0.000 0.002 0.000 util.py:461(close_fds) 80 0.000 0.000 0.000 0.000 util.py:48(debug) 1 0.000 0.000 0.000 0.000 utils.py:125( __eq__ ) 2 0.000 0.000 0.000 0.000 utils.py:139( __getattr__ ) 3 0.000 0.000 0.000 0.000 utils.py:152(wrapper_setattr) 1 0.000 0.000 0.000 0.000 utils.py:156( __init__ ) 2 0.000 0.000 0.000 0.000 utils.py:187(disable_on_exception) 4084 0.015 0.000 3.228 0.001 utils.py:194(inner) 1 0.000 0.000 0.000 0.000 utils.py:213( __init__ ) 2 0.000 0.000 0.000 0.000 utils.py:222( __eq__ ) 1 0.000 0.000 0.000 0.000 utils.py:252(_is_utf) 1 0.000 0.000 0.000 0.000 utils.py:266(_supports_unicode) 1 0.000 0.000 0.000 0.000 utils.py:282(_screen_shape_wrapper) 1 0.000 0.000 0.000 0.000 utils.py:333(_screen_shape_linux) 1 0.000 0.000 0.000 0.000 utils.py:347() 2041 0.008 0.000 0.071 0.000 utils.py:374(_text_width) 53902 0.032 0.000 0.049 0.000 utils.py:375() 2041 0.007 0.000 0.084 0.000 utils.py:378(disp_len) 25 0.000 0.000 0.000 0.000 weakref.py:106(remove) 41 0.000 0.000 0.001 0.000 weakref.py:165( __setitem__ ) 41 0.000 0.000 0.000 0.000 weakref.py:348( __new__ ) 41 0.000 0.000 0.000 0.000 weakref.py:353( __init__ ) 16210 0.011 0.000 0.011 0.000 {built-in method __new__ of type object at 0x55868a8fe5e0} 4 0.000 0.000 0.000 0.000 {built-in method _abc._abc_instancecheck} 8256 0.003 0.000 0.003 0.000 {built-in method _expired} 7764 0.003 0.000 0.003 0.000 {built-in method _free_weak_ref} 15622 0.108 0.000 0.108 0.000 {built-in method _hashlib.hmac_new} 7811 0.224 0.000 0.224 0.000 {built-in method _new_shared_fd_cpu} 7819 0.490 0.000 138.707 0.018 {built-in method _pickle.loads} 7811 0.008 0.000 0.008 0.000 {built-in method _socket.CMSG_SPACE} 7811 0.029 0.000 0.029 0.000 {built-in method _socket.dup} 31244 0.028 0.000 0.028 0.000 {built-in method _struct.pack} 31252 0.058 0.000 0.058 0.000 {built-in method _struct.unpack} 4129 0.007 0.000 0.007 0.000 {built-in method _thread.allocate_lock} 2076 0.001 0.000 0.001 0.000 {built-in method _thread.get_ident} 8 0.002 0.000 0.002 0.000 {built-in method _thread.start_new_thread} 25 0.000 0.000 0.000 0.000 {built-in method _weakref._remove_dead_weakref} 2 0.000 0.000 0.000 0.000 {built-in method _weakref.proxy} 2043 0.001 0.000 0.001 0.000 {built-in method builtins.abs} 8162 0.003 0.000 0.003 0.000 {built-in method builtins.divmod} 1 0.000 0.000 9238.469 9238.469 {built-in method builtins.exec} 7855 0.006 0.000 0.006 0.000 {built-in method builtins.getattr} 4089 0.002 0.000 0.002 0.000 {built-in method builtins.hasattr} 45 0.000 0.000 0.000 0.000 {built-in method builtins.id} 96874 0.034 0.000 0.034 0.000 {built-in method builtins.isinstance} 2 0.000 0.000 0.000 0.000 {built-in method builtins.iter} 232852/232851 0.071 0.000 0.071 0.000 {built-in method builtins.len} 2105 0.003 0.000 0.003 0.000 {built-in method builtins.max} 1 0.000 0.000 0.000 0.000 {built-in method builtins.min} 15818 0.014 0.000 0.620 0.000 {built-in method builtins.next} 2041 0.015 0.000 0.064 0.000 {built-in method builtins.sum} 1 0.000 0.000 0.000 0.000 {built-in method fcntl.ioctl} 2041 0.014 0.000 0.014 0.000 {built-in method fromtimestamp} 8357 0.010 0.000 0.010 0.000 {built-in method math.ceil} 256 0.000 0.000 0.000 0.000 {built-in method math.floor} 15656 0.088 0.000 0.088 0.000 {built-in method posix.close} 8 0.046 0.006 0.047 0.006 {built-in method posix.fork} 15622 0.088 0.000 0.088 0.000 {built-in method posix.fstat} 14196 0.026 0.000 0.026 0.000 {built-in method posix.getpid} 25 0.000 0.000 0.000 0.000 {built-in method posix.pipe} 62504 95.406 0.002 95.406 0.002 {built-in method posix.read} 1 0.000 0.000 0.000 0.000 {built-in method posix.sched_getaffinity} 7811 0.047 0.000 0.047 0.000 {built-in method posix.urandom} 4273 0.018 0.000 0.018 0.000 {built-in method posix.waitpid} 8 0.000 0.000 0.000 0.000 {built-in method posix.waitstatus_to_exitcode} 31244 0.294 0.000 0.294 0.000 {built-in method posix.write} 8357 0.008 0.000 0.008 0.000 {built-in method select.poll} 25585 0.012 0.000 0.012 0.000 {built-in method time.monotonic} 13913 0.007 0.000 0.007 0.000 {built-in method time.time} 8 0.000 0.000 0.000 0.000 {built-in method torch._C._error_if_any_worker_fails} 1 0.000 0.000 0.000 0.000 {built-in method torch._C._remove_worker_pids} 1 0.000 0.000 0.000 0.000 {built-in method torch._C._set_worker_pids} 7812 0.461 0.000 0.461 0.000 {built-in method torch._ops.profiler._record_function_enter_new} 7812 1.012 0.000 1.012 0.000 {built-in method torch._ops.profiler.} 7812 0.225 0.000 0.225 0.000 {built-in method torch.empty} 51861 0.017 0.000 0.017 0.000 {built-in method unicodedata.east_asian_width} 7811 0.017 0.000 0.017 0.000 {function socket.close at 0x7fdf404f64d0} 7811 0.004 0.000 0.004 0.000 {function socket.detach at 0x7fdf404f6560} 9 0.000 0.000 0.000 0.000 {method ' __enter__' of '_multiprocessing.SemLock' objects} 9917 0.007 0.000 0.007 0.000 {method ' __enter__' of '_thread.lock' objects} 9 0.000 0.000 0.000 0.000 {method ' __exit__' of '_multiprocessing.SemLock' objects} 2052 0.001 0.000 0.001 0.000 {method ' __exit__' of '_thread.RLock' objects} 25539 0.009 0.000 0.009 0.000 {method ' __exit__' of '_thread.lock' objects} 7812 0.005 0.000 0.005 0.000 {method ' __exit__' of 'torch._C.DisableTorchFunctionSubclass' objects} 1 0.000 0.000 0.000 0.000 {method '_is_mine' of '_multiprocessing.SemLock' objects} 7811 0.005 0.000 0.005 0.000 {method '_weak_ref' of 'torch._C.StorageBase' objects} 18247 0.021 0.000 0.021 0.000 {method 'acquire' of '_multiprocessing.SemLock' objects} 2052 0.002 0.000 0.002 0.000 {method 'acquire' of '_thread.RLock' objects} 24281 2.858 0.000 2.858 0.000 {method 'acquire' of '_thread.lock' objects} 27 0.000 0.000 0.000 0.000 {method 'add' of 'set' objects} 16063 0.005 0.000 0.005 0.000 {method 'append' of 'collections.deque' objects} 7843 0.003 0.000 0.003 0.000 {method 'append' of 'list' objects} 8 0.000 0.000 0.000 0.000 {method 'clear' of 'collections.deque' objects} 8357 0.006 0.000 0.006 0.000 {method 'clear' of 'dict' objects} 7811 0.235 0.000 0.235 0.000 {method 'connect' of '_socket.socket' objects} 7819 0.013 0.000 0.013 0.000 {method 'copy' of 'dict' objects} 1 0.000 0.000 0.000 0.000 {method 'difference' of 'set' objects} 15622 0.049 0.000 0.049 0.000 {method 'digest' of '_hashlib.HMAC' objects} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 24 0.000 0.000 0.000 0.000 {method 'discard' of 'set' objects} 7811 0.026 0.000 0.026 0.000 {method 'dump' of '_pickle.Pickler' objects} 2 0.000 0.000 0.000 0.000 {method 'encode' of 'str' objects} 7811 0.008 0.000 0.008 0.000 {method 'frombytes' of 'array.array' objects} 2043 0.001 0.000 0.001 0.000 {method 'get' of '_contextvars.ContextVar' objects} 7819 0.006 0.000 0.006 0.000 {method 'get' of 'dict' objects} 7811 0.006 0.000 0.006 0.000 {method 'getbuffer' of '_io.BytesIO' objects} 62504 0.022 0.000 0.022 0.000 {method 'getvalue' of '_io.BytesIO' objects} 1 0.000 0.000 0.000 0.000 {method 'item' of 'torch._C.TensorBase' objects} 2107 0.001 0.000 0.001 0.000 {method 'items' of 'dict' objects} 40 0.000 0.000 0.000 0.000 {method 'join' of 'str' objects} 4 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects} 8357 9090.788 1.088 9090.788 1.088 {method 'poll' of 'select.poll' objects} 5739 0.005 0.000 0.005 0.000 {method 'pop' of 'dict' objects} 2 0.000 0.000 0.000 0.000 {method 'pop' of 'list' objects} 256 0.000 0.000 0.000 0.000 {method 'random' of '_random.Random' objects} 1 0.000 0.000 0.000 0.000 {method 'random_' of 'torch._C.TensorBase' objects} 7811 37.854 0.005 37.854 0.005 {method 'recvmsg' of '_socket.socket' objects} 8357 0.007 0.000 0.007 0.000 {method 'register' of 'select.poll' objects} 18213 0.014 0.000 0.014 0.000 {method 'release' of '_multiprocessing.SemLock' objects} 2052 0.001 0.000 0.001 0.000 {method 'release' of '_thread.RLock' objects} 9893 0.120 0.000 0.120 0.000 {method 'release' of '_thread.lock' objects} 7840 0.003 0.000 0.003 0.000 {method 'remove' of 'collections.deque' objects} 3 0.000 0.000 0.000 0.000 {method 'remove' of 'set' objects} 7871 0.008 0.000 0.008 0.000 {method 'rpartition' of 'str' objects} 7811 0.136 0.000 0.136 0.000 {method 'set_' of 'torch._C.TensorBase' objects} 7811 0.025 0.000 0.025 0.000 {method 'setblocking' of '_socket.socket' objects} 15622 0.015 0.000 0.015 0.000 {method 'startswith' of 'str' objects} 2041 0.005 0.000 0.005 0.000 {method 'sub' of 're.Pattern' objects} 7811 0.108 0.000 0.108 0.000 {method 'update' of 'dict' objects} 2 0.000 0.000 0.000 0.000 {method 'values' of 'dict' objects} 62504 0.037 0.000 0.037 0.000 {method 'write' of '_io.BytesIO' objects} 2043 0.001 0.000 0.001 0.000 {method 'write' of '_io.StringIO' objects} ``` ---
**Author:** ![zhh210](https://avatars.discourse-cdn.com/v4/letter/z/ce73a5/32.png) [@zhh210](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzhh210)%5C **Post date:** [July 16, 2024, 9:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fiterating-on-dataset-extremely-slow%2F92813%2F6 "2024-07-16T21:23:29Z")
Switched to Ray which works much faster on first shot without any optimization but just taking the example code from Ray doc. If speed is the priority, I would say Ray is the way to go. ---
**Author:** ![Manu101](https://avatars.discourse-cdn.com/v4/letter/m/e47c2d/32.png) [@Manu101](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FManu101)%5C **Post date:** [September 27, 2024, 4:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fiterating-on-dataset-extremely-slow%2F92813%2F7 "2024-09-27T16:06:02Z")
Hi @zhh210 , can you please provide the exact code you substituted for Ray ? And how much the difference is between the two approaches? Thanks ---
**Author:** ![rohanahluwalia7](https://avatars.discourse-cdn.com/v4/letter/r/919ad9/32.png) [@rohanahluwalia7](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frohanahluwalia7)%5C **Post date:** [October 25, 2024, 9:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fiterating-on-dataset-extremely-slow%2F92813%2F8 "2024-10-25T21:06:27Z")
Id also like to see the Ray code implementation. Would love to see if I could get a speed up with my data loading as well. ---
**Author:** ![zhh210](https://avatars.discourse-cdn.com/v4/letter/z/ce73a5/32.png) [@zhh210](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzhh210)%5C **Post date:** [November 6, 2024, 7:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fiterating-on-dataset-extremely-slow%2F92813%2F9 "2024-11-06T19:33:08Z")
Ray’s implementation is quite straightforward and well documented [here](https://docs.ray.io/en/latest/data/api/doc/ray.data.read_parquet.html). In my usecase, I just use the default one-liner with no tweak at all and it runs much faster than heavily tuned HF dataset: ```auto import ray import torch ds = ray.data.read_parquet(['s3://' + i for i in s3.glob('s3://my_s3_bucket/*')]) dl_train = ds.iter_torch_batches(batch_size=1600, dtypes=torch.float32) ```