# Can't generate my own dataset using load\_dataset **URL:** **Author:** ![HelenProkonina](https://avatars.discourse-cdn.com/v4/letter/h/e9a140/32.png) [@HelenProkonina](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHelenProkonina)%5C **Post date:** [February 13, 2024, 11:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-generate-my-own-dataset-using-load-dataset%2F72985%2F1 "2024-02-13T11:19:21Z") Hi! I am trying to create my own dataset for image classification and use the following code: `dataset = load_dataset("imagefolder", data_dir="Vials_all_cases")` The structure of folders is the following: ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/e/a/eaeb18c62698ee152db76ade4b0333e3aa081550.png) And first I got this: `Generating train split: 0 examples [00:00, ? examples/s]` And then I get an error: ```auto --------------------------------------------------------------------------- ImportError Traceback (most recent call last) File c:\Projects\Vials_Transformers\.venv\lib\site-packages\datasets\builder.py:1646, in GeneratorBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, split_info, check_duplicate_keys, job_id) 1637 writer = writer_class( 1638 features=writer._features, 1639 path=fpath.replace("SSSSS", f"{shard_id:05d}").replace("JJJJJ", f"{job_id:05d}"), (...) 1644 embed_local_files=embed_local_files, 1645 ) -> 1646 example = self.info.features.encode_example(record) if self.info.features is not None else record 1647 writer.write(example, key) File c:\Projects\Vials_Transformers\.venv\lib\site-packages\datasets\features\features.py:1851, in Features.encode_example(self, example) 1850 example = cast_to_python_objects(example) -> 1851 return encode_nested_example(self, example) File c:\Projects\Vials_Transformers\.venv\lib\site-packages\datasets\features\features.py:1229, in encode_nested_example(schema, obj, level) 1227 raise ValueError("Got None but expected a dictionary instead") 1228 return ( -> 1229 { 1230 k: encode_nested_example(sub_schema, sub_obj, level=level + 1) 1231 for k, (sub_schema, sub_obj) in zip_dict(schema, obj) 1232 } 1233 if obj is not None 1234 else None ... 1664 e = e. __context__ -> 1665 raise DatasetGenerationError("An error occurred while generating the dataset") from e 1667 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths) DatasetGenerationError: An error occurred while generating the dataset ``` ---
**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C **Post date:** [February 15, 2024, 5:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-generate-my-own-dataset-using-load-dataset%2F72985%2F2 "2024-02-15T17:43:11Z")
Hi! The stack trace seems to be cropped. Can you please share the full one so we can see the actual error? ---
**Author:** ![HelenProkonina](https://avatars.discourse-cdn.com/v4/letter/h/e9a140/32.png) [@HelenProkonina](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHelenProkonina)%5C **Post date:** [February 18, 2024, 1:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-generate-my-own-dataset-using-load-dataset%2F72985%2F3 "2024-02-18T13:11:24Z")
![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/7/9/798f439dc1805047955ddc87fd5b3feacae42ad8.png) And then the error: ```auto --------------------------------------------------------------------------- ImportError Traceback (most recent call last) File c:\Projects\Vials_Transformers\.venv\lib\site-packages\datasets\builder.py:1646, in GeneratorBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, split_info, check_duplicate_keys, job_id) 1637 writer = writer_class( 1638 features=writer._features, 1639 path=fpath.replace("SSSSS", f"{shard_id:05d}").replace("JJJJJ", f"{job_id:05d}"), (...) 1644 embed_local_files=embed_local_files, 1645 ) -> 1646 example = self.info.features.encode_example(record) if self.info.features is not None else record 1647 writer.write(example, key) File c:\Projects\Vials_Transformers\.venv\lib\site-packages\datasets\features\features.py:1851, in Features.encode_example(self, example) 1850 example = cast_to_python_objects(example) -> 1851 return encode_nested_example(self, example) File c:\Projects\Vials_Transformers\.venv\lib\site-packages\datasets\features\features.py:1229, in encode_nested_example(schema, obj, level) 1227 raise ValueError("Got None but expected a dictionary instead") 1228 return ( -> 1229 { 1230 k: encode_nested_example(sub_schema, sub_obj, level=level + 1) 1231 for k, (sub_schema, sub_obj) in zip_dict(schema, obj) 1232 } 1233 if obj is not None 1234 else None ... 1664 e = e. __context__ -> 1665 raise DatasetGenerationError("An error occurred while generating the dataset") from e 1667 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths) DatasetGenerationError: An error occurred while generating the dataset ``` ---
**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C **Post date:** [February 19, 2024, 4:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-generate-my-own-dataset-using-load-dataset%2F72985%2F4 "2024-02-19T16:31:48Z")
That stack trace cannot be expanded? Based on the `ImportError` at the start of the trace, maybe running `pip install "datasets[vision]"` (to install the missing dependencies) before the script can resolve the issue.