# Datasets.load\_datasets fails
**URL:**
**Author:**  [@yashara](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyashara)%5C
**Post date:** [October 4, 2024, 10:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%2F1 "2024-10-04T22:46:00Z")
I am trying to load a dataset from the openx datasets using a command like this:
dataset = datasets.load\_dataset(
âjxu124/OpenX-Embodimentâ,
âberkeley\_gnm\_cory\_hallâ,
streaming=False,
split=âtrainâ,
cache\_dir=ds\_root,
trust\_remote\_code=True,
)
But itâs failing on any argument configurations. The summarized error stack is:
File â~/.cache/huggingface/modules/datasets\_modules/datasets/jxu124âOpenX-Embodiment/317e9044a9bb97bb1db9ea5aebf1c15f5cc3e1e071e5da025e97892e96dae22b/OpenX-Embodiment.pyâ, line 29, in decode\_image
data = data.decode()
UnicodeDecodeError: âutf-8â codec canât decode byte 0xff in position 0: invalid start byte
The above exception was the direct cause of the following exception:
.
.
.
File ââŚ/lib/python3.10/site-packages/datasets/builder.pyâ, line 1642, in \_prepare\_split\_single
raise DatasetGenerationError(âAn error occurred while generating the datasetâ) from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
Looks like the API is buggy for this specific dataset. Does anyone know how to successfully load data from this dataset suit? Thanks.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 4, 2024, 11:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%2F2 "2024-10-04T23:47:22Z")
Character code errors still occur in 2024âŚ
Apparently there are cases where it can be avoided by explicitly specifying it at load time.
If this does not work, there may be another cause.
> [@Random utf-8 errors from dataset](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frandom-utf-8-errors-from-dataset%2F22134%2F9)%3A
>
> Iâve moved on from that project, at this point, so unfortunately I canât give a stack trace. But I will say that I think it was more a data problem than a datasets problem. I still see the error with some of the data Iâm using now, but Iâve started including chardet in my data pipeline, which seems to fix it (though itâs a bit pokey).
> [@'utf-8' codec can't decode byte 0xff in position 0: invalid start byte](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Futf-8-codec-cant-decode-byte-0xff-in-position-0-invalid-start-byte%2F51825%2F4)%3A
>
> Thank you very much!!! Problem solved! Your answer really helps me a lot!
> [@UTF-16 for datasets?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Futf-16-for-datasets%2F43828)%3A
>
> Working on a translator, hoping to do fine-tuning with a utf-16 dataset so I can get all the French accents etc. Datasets load\_dataset() doesnât seem to like non-utf-8 Is there a way to specify or does it HAVE to be utf-8? If it has to be utf-8, any suggestions for special characters?
```py
dataset = datasets.load_dataset(
âjxu124/OpenX-Embodimentâ,
âberkeley_gnm_cory_hallâ,
streaming=False,
split=âtrainâ,
cache_dir=ds_root,
trust_remote_code=True,
encoding="utf-16",
)
```
or
```py
dataset = datasets.load_dataset(
âjxu124/OpenX-Embodimentâ,
âberkeley_gnm_cory_hallâ,
streaming=False,
split=âtrainâ,
cache_dir=ds_root,
trust_remote_code=True,
encoding="utf-8",
)
```
---
**Author:**  [@yashara](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyashara)%5C
**Post date:** [October 5, 2024, 12:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%2F3 "2024-10-05T00:43:37Z")
Thanks for the reply, @John6666.
Checking the dataset.load\_dataset(), I donât see any âencodingâ argument option. Setting it triggers such errors:
`ValueError: BuilderConfig OpenXConfig (name='berkeley_gnm_cory_hall', version=0.0.0, data_dir=None, data_files=None, description=None, features=None) doesn't have a 'encoding' key`
Other problem when tracing the error stack at `OpenX-Embodiment.py", line 30: data = data.decode()`, I can see the data is not an image, but some bogus values like
```auto
b'/home/jonathan/dev/gnm_dataset/cory_hall/cory5_aug17_00_0546'
```
So, make me thinking if the API is not downloading the data properly.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 5, 2024, 12:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%2F4 "2024-10-05T00:56:18Z")
The referenced article is from 2023, so maybe the argument is obsoleteâŚ
Anyway, I agree with you that just because the error message is UTF-8 related does not mean that it is a character code issue.
I hope itâs a problem that can be avoided by modifying the options or code without changing the dataset.
It could even be some kind of bug that has never been fixed because it can only be avoided by changing the dataset.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 5, 2024, 1:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%2F5 "2024-10-05T01:03:29Z")
Iâm sure youâve done this long ago, but just a spell.
```auto
pip uninstall datasets
pip install git+https://github.com/huggingface/datasets.git
```
This is an article from April of this year, but it seems that some people are using an unusually old datasets library, perhaps pulled in by some library dependency.
>
>
> \### Describe the bug
>
> \*\*emotions = load\_dataset('emotion')\*\*
>
> \_UnicodeDecodeErâŚror: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte\_
>
> \### Steps to reproduce the bug
>
> load\_dataset('emotion')
>
> \### Expected behavior
>
> succese
>
> \### Environment info
>
> py3.10
> transformers 4.41.0.dev0
> datasets 2.19.0
---
**Author:**  [@yashara](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyashara)%5C
**Post date:** [October 5, 2024, 5:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%2F6 "2024-10-05T17:16:17Z")
Thanks for the suggestions, @John6666 . I tried changing versions, but doesnât look like working. Maybe itâs because the code is not well-maintained &/ dataset formats have become incompatible.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 5, 2024, 9:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%2F7 "2024-10-05T21:06:47Z")
@lhoestq There is an occasional problem with some datasets not loading properly in the datasets library. On the surface this appears as a UTF-8 error, but the actual cause is unknown. Is this fixable?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [October 7, 2024, 12:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%2F8 "2024-10-07T12:54:50Z")
Weâre using `PyArrow` to load the data, maybe we can see with them how to improve it. Do you have a reproducible example ?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 7, 2024, 11:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%2F9 "2024-10-07T23:02:07Z")
> **[jxu124/OpenX-Embodiment ¡ Datasets at Hugging Face](https://huggingface.co/datasets/jxu124/OpenX-Embodiment)**
>
> Weâre on a journey to advance and democratize artificial intelligence through open source and open science.
I would have to ask yashara to know exactly, but perhaps this data set can reproduce it. Also, I think Iâve seen some of these in past forums and github posts, so Iâll look for them later.
```py
import datasets
dataset = datasets.load_dataset(
âjxu124/OpenX-Embodimentâ,
âberkeley_gnm_cory_hallâ,
streaming=False,
split=âtrainâ,
cache_dir=ds_root,
trust_remote_code=True,
)
```
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [October 11, 2024, 1:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%2F10 "2024-10-11T13:56:22Z")
Do you have an example that doesnât use `trust_remote_code` ? We stopped developing that option because itâs not great for obvious security reasons.
Also note that datasets in WebDataset format are supported out-of-the-box so `trust_remote_code` / having a loading script is not needed in that case.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 11, 2024, 2:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%2F11 "2024-10-11T14:00:03Z")
> Do you have an example that doesnât use `trust_remote_code` ?
2023, but his code doesnât seem to trust\_remote\_code.
> [@'utf-8' codec can't decode byte 0xff in position 0: invalid start byte](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Futf-8-codec-cant-decode-byte-0xff-in-position-0-invalid-start-byte%2F51825)%3A
>
> Hello, there. I downloaded a dataset in Hub and saved it to a local folder. However I could not reload it. Here is my code from datasets import load\_dataset raw\_datasets = load\_dataset("roneneldan/TinyStories") raw\_datasets.save\_to\_disk("Tiny\_Stories") raw\_datasets = load\_dataset('text', data\_dir = "Tiny\_Stories") Here is my error. UnicodeDecodeError Traceback (most recent call last) File c:\Users\Tom W\AppData\Local\Programs\Python\Python310\lib\site-packages\datasets\âŚ
```py
from datasets import load_dataset
raw_datasets = load_dataset("roneneldan/TinyStories")
raw_datasets.save_to_disk("Tiny_Stories")
raw_datasets = load_dataset('text', data_dir = "Tiny_Stories")
```
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [October 11, 2024, 2:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%2F12 "2024-10-11T14:16:06Z")
You should use load\_form\_disk after save\_to\_disk (itâs a serialized form for local disk - not optimized for online sharing but faster to reload locally):
```python
from datasets import load_dataset, load_from_disk
raw_datasets = load_dataset("roneneldan/TinyStories")
raw_datasets.save_to_disk("Tiny_Stories")
raw_datasets = load_from_disk("Tiny_Stories")
```
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 11, 2024, 2:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%2F13 "2024-10-11T14:31:19Z")
Ah. That was a completely different error.
Thatâs all the errors Iâve seen in my searches. Here is an example that was addressed by an option added in a library upgrade. The rest of the problems donât seem to exist.
>
>
> \### Describe the bug
>
> \*\*emotions = load\_dataset('emotion')\*\*
>
> \_UnicodeDecodeErâŚror: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte\_
>
> \### Steps to reproduce the bug
>
> load\_dataset('emotion')
>
> \### Expected behavior
>
> succese
>
> \### Environment info
>
> py3.10
> transformers 4.41.0.dev0
> datasets 2.19.0
>
>
> \### Describe the bug
>
> unicodedecodeerror: 'utf-8' codec can't decode byte 0xac⌠in position 25: invalid start byte
>
> \### Steps to reproduce the bug
> \`\`\`
> import sys
> sys.getdefaultencoding()
> 'utf-8'
>
>
>
> from datasets import load\_dataset
>
> print(f"Train dataset size: {len(dataset\['train'\])}")
> print(f"Test dataset size: {len(dataset\['test'\])}")
>
> Resolving data files: 100%
> 159/159 \[00:00\<00:00, 9909.28it/s\]
> Using custom data configuration samsum-0b1209637541c9e6
> Downloading and preparing dataset json/samsum to C:/Users/Administrator/.cache/huggingface/datasets/json/samsum-0b1209637541c9e6/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51...
> Downloading data files: 100%
> 3/3 \[00:00\<00:00, 119.99it/s\]
> Extracting data files: 100%
> 3/3 \[00:00\<00:00, 9.54it/s\]
> Generating train split:
> 88392/0 \[00:15\<00:00, 86848.17 examples/s\]
> Generating test split:
> 0/0 \[00:00\ \---------------------------------------------------------------------------
> ArrowInvalid Traceback (most recent call last)
> File ~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\datasets\\packaged\_modules\\json\\json.py:132, in Json.\_generate\_tables(self, files)
> 131 try:
> \--\> 132 pa\_table = paj.read\_json(
> 133 io.BytesIO(batch), read\_options=paj.ReadOptions(block\_size=block\_size)
> 134 )
> 135 break
>
> File ~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\pyarrow\\\_json.pyx:290, in pyarrow.\_json.read\_json()
>
> File ~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\pyarrow\\error.pxi:144, in pyarrow.lib.pyarrow\_internal\_check\_status()
>
> File ~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\pyarrow\\error.pxi:100, in pyarrow.lib.check\_status()
>
> ArrowInvalid: JSON parse error: Invalid value. in row 0
>
> During handling of the above exception, another exception occurred:
>
> UnicodeDecodeError Traceback (most recent call last)
> File ~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\datasets\\builder.py:1819, in ArrowBasedBuilder.\_prepare\_split\_single(self, gen\_kwargs, fpath, file\_format, max\_shard\_size, job\_id)
> 1818 \_time = time.time()
> \-\> 1819 for \_, table in generator:
> 1820 if max\_shard\_size is not None and writer.\_num\_bytes \> max\_shard\_size:
>
> File ~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\datasets\\packaged\_modules\\json\\json.py:153, in Json.\_generate\_tables(self, files)
> 152 with open(file, encoding="utf-8") as f:
> \--\> 153 dataset = json.load(f)
> 154 except json.JSONDecodeError:
>
> File ~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\json\\\_\_init\_\_.py:293, in load(fp, cls, object\_hook, parse\_float, parse\_int, parse\_constant, object\_pairs\_hook, \*\*kw)
> 276 """Deserialize \`\`fp\`\` (a \`\`.read()\`\`-supporting file-like object containing
> 277 a JSON document) to a Python object.
> 278
> (...)
> 291 kwarg; otherwise \`\`JSONDecoder\`\` is used.
> 292 """
> \--\> 293 return loads(fp.read(),
> 294 cls=cls, object\_hook=object\_hook,
> 295 parse\_float=parse\_float, parse\_int=parse\_int,
> 296 parse\_constant=parse\_constant, object\_pairs\_hook=object\_pairs\_hook, \*\*kw)
>
> File ~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\codecs.py:322, in BufferedIncrementalDecoder.decode(self, input, final)
> 321 data = self.buffer + input
> \--\> 322 (result, consumed) = self.\_buffer\_decode(data, self.errors, final)
> 323 # keep undecoded input until the next call
>
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xac in position 25: invalid start byte
>
> The above exception was the direct cause of the following exception:
>
> DatasetGenerationError Traceback (most recent call last)
> Cell In\[81\], line 5
> 1 from datasets import load\_dataset
> 3 # Load dataset from the hub
> 4 #dataset = load\_dataset("json",data\_files="C:/Users/Administrator/Desktop/samsum/samsum/data/corpus/train.json",field="data")
> \----\> 5 dataset = load\_dataset('json',"samsum")
> 6 #dataset = load\_dataset("samsum")
> 7 print(f"Train dataset size: {len(dataset\['train'\])}")
>
> File ~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\datasets\\load.py:1758, in load\_dataset(path, name, data\_dir, data\_files, split, cache\_dir, features, download\_config, download\_mode, ignore\_verifications, keep\_in\_memory, save\_infos, revision, use\_auth\_token, task, streaming, num\_proc, \*\*config\_kwargs)
> 1755 try\_from\_hf\_gcs = path not in \_PACKAGED\_DATASETS\_MODULES
> 1757 # Download and prepare data
> \-\> 1758 builder\_instance.download\_and\_prepare(
> 1759 download\_config=download\_config,
> 1760 download\_mode=download\_mode,
> 1761 ignore\_verifications=ignore\_verifications,
> 1762 try\_from\_hf\_gcs=try\_from\_hf\_gcs,
> 1763 num\_proc=num\_proc,
> 1764 )
> 1766 # Build dataset for splits
> 1767 keep\_in\_memory = (
> 1768 keep\_in\_memory if keep\_in\_memory is not None else is\_small\_dataset(builder\_instance.info.dataset\_size)
> 1769 )
>
> File ~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\datasets\\builder.py:860, in DatasetBuilder.download\_and\_prepare(self, output\_dir, download\_config, download\_mode, ignore\_verifications, try\_from\_hf\_gcs, dl\_manager, base\_path, use\_auth\_token, file\_format, max\_shard\_size, num\_proc, storage\_options, \*\*download\_and\_prepare\_kwargs)
> 858 if num\_proc is not None:
> 859 prepare\_split\_kwargs\["num\_proc"\] = num\_proc
> \--\> 860 self.\_download\_and\_prepare(
> 861 dl\_manager=dl\_manager,
> 862 verify\_infos=verify\_infos,
> 863 \*\*prepare\_split\_kwargs,
> 864 \*\*download\_and\_prepare\_kwargs,
> 865 )
> 866 # Sync info
> 867 self.info.dataset\_size = sum(split.num\_bytes for split in self.info.splits.values())
>
> File ~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\datasets\\builder.py:953, in DatasetBuilder.\_download\_and\_prepare(self, dl\_manager, verify\_infos, \*\*prepare\_split\_kwargs)
> 949 split\_dict.add(split\_generator.split\_info)
> 951 try:
> 952 # Prepare split will record examples associated to the split
> \--\> 953 self.\_prepare\_split(split\_generator, \*\*prepare\_split\_kwargs)
> 954 except OSError as e:
> 955 raise OSError(
> 956 "Cannot find data file. "
> 957 + (self.manual\_download\_instructions or "")
> 958 + "\\nOriginal error:\\n"
> 959 + str(e)
> 960 ) from None
>
> File ~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\datasets\\builder.py:1708, in ArrowBasedBuilder.\_prepare\_split(self, split\_generator, file\_format, num\_proc, max\_shard\_size)
> 1706 gen\_kwargs = split\_generator.gen\_kwargs
> 1707 job\_id = 0
> \-\> 1708 for job\_id, done, content in self.\_prepare\_split\_single(
> 1709 gen\_kwargs=gen\_kwargs, job\_id=job\_id, \*\*\_prepare\_split\_args
> 1710 ):
> 1711 if done:
> 1712 result = content
>
> File ~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\datasets\\builder.py:1851, in ArrowBasedBuilder.\_prepare\_split\_single(self, gen\_kwargs, fpath, file\_format, max\_shard\_size, job\_id)
> 1849 if isinstance(e, SchemaInferenceError) and e.\_\_context\_\_ is not None:
> 1850 e = e.\_\_context\_\_
> \-\> 1851 raise DatasetGenerationError("An error occurred while generating the dataset") from e
> 1853 yield job\_id, True, (total\_num\_examples, total\_num\_bytes, writer.\_features, num\_shards, shard\_lengths)
>
> DatasetGenerationError: An error occurred while generating the dataset
> \`\`\`
>
> \### Expected behavior
>
> can't load dataset
>
> \### Environment info
>
> dataset:samsum
> system :win10
> gpu:m40 24G