# Datasets.load\_datasets fails

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%3E%5C
**Category:** 🤗Datasets\
**Created:** [October 4, 2024, 10:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172 "2024-10-04T22:46:00Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 4, 2024, 11:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172%2F2 "2024-10-04T23:47:22Z")

</div>

Character code errors still occur in 2024…  
Apparently there are cases where it can be avoided by explicitly specifying it at load time.  
If this does not work, there may be another cause.

> [@Random utf-8 errors from dataset](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frandom-utf-8-errors-from-dataset%2F22134%2F9)%3A
>
> I’ve moved on from that project, at this point, so unfortunately I can’t give a stack trace. But I will say that I think it was more a data problem than a datasets problem. I still see the error with some of the data I’m using now, but I’ve started including chardet in my data pipeline, which seems to fix it (though it’s a bit pokey).

> [@'utf-8' codec can't decode byte 0xff in position 0: invalid start byte](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Futf-8-codec-cant-decode-byte-0xff-in-position-0-invalid-start-byte%2F51825%2F4)%3A
>
> Thank you very much!!! Problem solved! Your answer really helps me a lot!

> [@UTF-16 for datasets?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Futf-16-for-datasets%2F43828)%3A
>
> Working on a translator, hoping to do fine-tuning with a utf-16 dataset so I can get all the French accents etc. Datasets load\_dataset() doesn’t seem to like non-utf-8 Is there a way to specify or does it HAVE to be utf-8? If it has to be utf-8, any suggestions for special characters?

```py
dataset = datasets.load_dataset(
“jxu124/OpenX-Embodiment”,
“berkeley_gnm_cory_hall”,
streaming=False,
split=“train”,
cache_dir=ds_root,
trust_remote_code=True,
encoding="utf-16",
)

```

or

```py
dataset = datasets.load_dataset(
“jxu124/OpenX-Embodiment”,
“berkeley_gnm_cory_hall”,
streaming=False,
split=“train”,
cache_dir=ds_root,
trust_remote_code=True,
encoding="utf-8",
)

```

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-load-datasets-fails%2F110172)._