# Loading natural\_questions
**URL:**
**Author:**  [@Arij](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FArij)%5C
**Post date:** [January 20, 2023, 5:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-natural-questions%2F30089%2F1 "2023-01-20T17:59:54Z")
using just
from datasets import load\_dataset
dataset = load\_dataset(βnatural\_questionsβ)
gives me the following error
File ~/anaconda3/lib/python3.9/site-packages/datasets/builder.py:1879, in BeamBasedBuilder.\_download\_and\_prepare(self, dl\_manager, verify\_infos, \*\*prepare\_splits\_kwargs)
1877 if not beam\_runner and not beam\_options:
1878 usage\_example = f"load\_dataset(β{self.name}β, β{self.config.name}β, beam\_runner=βDirectRunnerβ)"
β 1879 raise MissingBeamOptions(
1880 "Trying to generate a dataset using Apache Beam, yet no Beam Runner "
1881 "or PipelineOptions() has been provided in `load_dataset` or in the "
1882 "builder arguments. For big datasets it has to run on large-scale data "
1883 "processing tools like Dataflow, Spark, etc. More information about "
1884 "Apache Beam runners at "
1885 β[Apache Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/)β
1886 "\nIf you really want to run it locally because you feel like the "
1887 βDataset is small enough, you can use the local beam runner called "
1888 β`DirectRunner` (you may run out of memory). \nExample of usage: "
1889 fβ\n\t`{usage_example}`β
1890 )
1892 # Beam type checking assumes transforms multiple outputs are of same type,
1893 # which is not our case. Plus it doesnβt handle correctly all types, so we
1894 # are better without it.
1895 pipeline\_options = {βpipeline\_type\_checkβ: False}
MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at [Apache Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/)
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
`load_dataset('natural_questions', 'default', beam_runner='DirectRunner')`
using load\_dataset(βnatural\_questionsβ, βdefaultβ, beam\_runner=βDirectRunnerβ) Gives this error
> > > load\_dataset(βnatural\_questionsβ, βdefaultβ, beam\_runner=βDirectRunnerβ)
> > > Downloading and preparing dataset natural\_questions/default to /.cache/huggingface/datasets/natural\_questions/default/0.0.4/da8124c83e3394df62c0f9bbc6c07652bbe9288ad833053134d5f0e978bb4ee5β¦
> > > Downloading: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 17.4k/17.4k [00:00\<00:00, 1.03MB/s]
> > > Downloading data files: 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 2/2 [00:00\<00:00, 182.84it/s]
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1/1 [00:00\<00:00, 79.02parquet files/s]
0%| | 0/1 [03:38\
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [January 23, 2023, 1:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-natural-questions%2F30089%2F2 "2023-01-23T13:27:22Z")
Hi ! What version of `datasets` are you using ? Can you try updating ? We fixed a bug like this one in `datasets` 2.8.0
---
**Author:**  [@Arij](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FArij)%5C
**Post date:** [January 23, 2023, 1:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-natural-questions%2F30089%2F3 "2023-01-23T13:57:15Z")
Acually the last error was from a new conda environment that I created it especially to solve this issue. In this environment I had installed just datsets using conda install datasets.
This is the result again
dataset = load\_dataset(βnatural\_questionsβ)
Resolving data files: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 51/51 [00:00\<00:00, 281459.87it/s]
Using custom data configuration natural\_questions-4f5f22b23f27c846
Downloading and preparing dataset json/natural\_questions to /home/arij/.cache/huggingface/datasets/json/natural\_questions-4f5f22b23f27c846/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aabβ¦
Downloading data files: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 2/2 [00:00\<00:00, 2597.09it/s]
Extracting data files: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 2/2 [12:37\<00:00, 378.77s/it]
Traceback (most recent call last):
File ββ, line 1, in
File βanaconda3/envs/lib/python3.10/site-packages/datasets/load.pyβ, line 1742, in load\_dataset
builder\_instance.download\_and\_prepare(
File βanaconda3/envs/lib/python3.10/site-packages/datasets/builder.pyβ, line 814, in download\_and\_prepare
self.\_download\_and\_prepare(
File βanaconda3/envs/lib/python3.10/site-packages/datasets/builder.pyβ, line 905, in \_download\_and\_prepare
self.\_prepare\_split(split\_generator, \*\*prepare\_split\_kwargs)
File βanaconda3/envs/lib/python3.10/site-packages/datasets/builder.pyβ, line 1520, in \_prepare\_split
writer.write\_table(table)
File βanaconda3/envs/lib/python3.10/site-packages/datasets/arrow\_writer.pyβ, line 540, in write\_table
pa\_table = table\_cast(pa\_table, self.\_schema)
File βanaconda3/envslib/python3.10/site-packages/datasets/table.pyβ, line 2068, in table\_cast
return cast\_table\_to\_schema(table, schema)
File βanaconda3/envs/lib/python3.10/site-packages/datasets/table.pyβ, line 2030, in cast\_table\_to\_schema
arrays = [cast\_array\_to\_feature(table[name], feature) for name, feature in features.items()]
File βanaconda3/envs/lib/python3.10/site-packages/datasets/table.pyβ, line 2030, in
arrays = [cast\_array\_to\_feature(table[name], feature) for name, feature in features.items()]
File βanaconda3/envs/lib/python3.10/site-packages/datasets/table.pyβ, line 1740, in wrapper
return pa.chunked\_array([func(chunk, \*args, \*\*kwargs) for chunk in array.chunks])
File βanaconda3/envs/lib/python3.10/site-packages/datasets/table.pyβ, line 1740, in
return pa.chunked\_array([func(chunk, \*args, \*\*kwargs) for chunk in array.chunks])
File βanaconda3/envs/lib/python3.10/site-packages/datasets/table.pyβ, line 1867, in cast\_array\_to\_feature
casted\_values = \_c(array.values, feature[0])
File βanaconda3/envs/lib/python3.10/site-packages/datasets/table.pyβ, line 1742, in wrapper
return func(array, \*args, \*\*kwargs)
File βanaconda3/envs/lib/python3.10/site-packages/datasets/table.pyβ, line 1862, in cast\_array\_to\_feature
arrays = [\_c(array.field(name), subfeature) for name, subfeature in feature.items()]
File βanaconda3/envs/lib/python3.10/site-packages/datasets/table.pyβ, line 1862, in
arrays = [\_c(array.field(name), subfeature) for name, subfeature in feature.items()]
File βanaconda3/envs/lib/python3.10/site-packages/datasets/table.pyβ, line 1742, in wrapper
return func(array, \*args, \*\*kwargs)
File βanaconda3/envs/lib/python3.10/site-packages/datasets/table.pyβ, line 1912, in cast\_array\_to\_feature
return array\_cast(array, feature(), allow\_number\_to\_str=allow\_number\_to\_str)
File βanaconda3/envs/lib/python3.10/site-packages/datasets/table.pyβ, line 1742, in wrapper
return func(array, \*args, \*\*kwargs)
File βanaconda3/envs/lib/python3.10/site-packages/datasets/table.pyβ, line 1821, in array\_cast
return array.cast(pa\_type)
File βpyarrow/array.pxiβ, line 915, in pyarrow.lib.Array.cast
File βanaconda3/envs/lib/python3.10/site-packages/pyarrow/compute.pyβ, line 376, in cast
return call\_function(βcastβ, [arr], options)
File βpyarrow/\_compute.pyxβ, line 542, in pyarrow.\_compute.call\_function
File βpyarrow/\_compute.pyxβ, line 341, in pyarrow.\_compute.Function.call
File βpyarrow/error.pxiβ, line 144, in pyarrow.lib.pyarrow\_internal\_check\_status
File βpyarrow/error.pxiβ, line 100, in pyarrow.lib.check\_status
pyarrow.lib.ArrowInvalid: Integer value 3868261029123585616 not in range: -9007199254740992 to 9007199254740992
conda install datasets
conda list | grep datasets
datasets 2.6.1 pypi\_0 pypi
pip install datasets
pip list | grep datasets
datasets 2.6.1
what I did as a final try is unistall the datasets using both conda and pip the reinstall the datasets using pip. Finally latest version of datasets was installed succesfully and the nq dataset was not installed again error:
File βpyarrow/array.pxiβ, line 915, in pyarrow.lib.Array.cast
File β/home/arij/anaconda3/envs/QA/lib/python3.10/site-packages/pyarrow/compute.pyβ, line 376, in cast
return call\_function(βcastβ, [arr], options)
File βpyarrow/\_compute.pyxβ, line 542, in pyarrow.\_compute.call\_function
File βpyarrow/\_compute.pyxβ, line 341, in pyarrow.\_compute.Function.call
File βpyarrow/error.pxiβ, line 144, in pyarrow.lib.pyarrow\_internal\_check\_status
File βpyarrow/error.pxiβ, line 100, in pyarrow.lib.check\_status
pyarrow.lib.ArrowInvalid: Integer value 3868261029123585616 not in range: -9007199254740992 to 9007199254740992
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File ββ, line 1, in
File βanaconda3/envs/lib/python3.10/site-packages/datasets/load.pyβ, line 1757, in load\_dataset
builder\_instance.download\_and\_prepare(
File βanaconda3/envs/lib/python3.10/site-packages/datasets/builder.pyβ, line 860, in download\_and\_prepare
self.\_download\_and\_prepare(
File βanaconda3/envs/lib/python3.10/site-packages/datasets/builder.pyβ, line 953, in \_download\_and\_prepare
self.\_prepare\_split(split\_generator, \*\*prepare\_split\_kwargs)
File βanaconda3/envs/lib/python3.10/site-packages/datasets/builder.pyβ, line 1706, in \_prepare\_split
for job\_id, done, content in self.\_prepare\_split\_single(
File βanaconda3/envs/lib/python3.10/site-packages/datasets/builder.pyβ, line 1849, in \_prepare\_split\_single
raise DatasetGenerationError(βAn error occurred while generating the datasetβ) from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
Anyway yesterday I have installed the dataset from the offficial cite. Hope this issue will be fixed soon for future users
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [January 26, 2023, 10:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-natural-questions%2F30089%2F4 "2023-01-26T10:00:34Z")
Can you try installing `datasets` using the official Hugging Face conda channel ?
```auto
conda install -c huggingface datasets
```
In your env you seem to have `datasets` 2.6.1 while this was fixed in `2.8.0`
---
**Author:**  [@anon11030295](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanon11030295)%5C
**Post date:** [April 2, 2023, 2:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-natural-questions%2F30089%2F5 "2023-04-02T14:02:05Z")
I am facing the same issue with a dataset of +170GB:
> File β/cluster/apps/nss/gcc-8.2.0/python/3.10.4/x86\_64/lib64/python3.10/site-packages/datasets/builder.pyβ, line 1495, in \_prepare\_split
> for key, table in logging.tqdm(
> File β/cluster/apps/nss/gcc-8.2.0/python/3.10.4/x86\_64/lib64/python3.10/site-packages/tqdm/std.pyβ, line 1195, in **iter**
> for obj in iterable:
> File β/cluster/apps/nss/gcc-8.2.0/python/3.10.4/x86\_64/lib64/python3.10/site-packages/datasets/packaged\_modules/parquet/parquet.pyβ, line 69, in \_generate\_tables
> for batch\_idx, record\_batch in enumerate(
> File βpyarrow/\_parquet.pyxβ, line 1323, in iter\_batches
> File βpyarrow/error.pxiβ, line 121, in pyarrow.lib.check\_status
> **pyarrow.lib.ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs**
The code throws this at the following line:
> current\_dataset = load\_dataset(
> dataset\_info.key,
> dataset\_info.subset,
> split=βtrainβ,
> use\_auth\_token=True,
> )
I am using datasets `2.11.0` installed with `conda install -c huggingface datasets`
---
**Author:**  [@crownor](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcrownor)%5C
**Post date:** [May 12, 2023, 3:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-natural-questions%2F30089%2F6 "2023-05-12T03:57:58Z")
I found this [https://github.com/huggingface/datasets/issues/5695](https://github.com/huggingface/datasets/issues/5695) , maybe each shard is too large
---
**Author:**  [@wangskyone](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwangskyone)%5C
**Post date:** [December 12, 2023, 6:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-natural-questions%2F30089%2F7 "2023-12-12T06:07:25Z")
I solve the problem by this [https://github.com/huggingface/datasets/issues/2181](https://github.com/huggingface/datasets/issues/2181)