# Pre-training a language model on a large dataset **URL:** **Author:** ![go-inoue](https://avatars.discourse-cdn.com/v4/letter/g/ecd19e/32.png) [@go-inoue](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgo-inoue)%5C **Post date:** [August 20, 2020, 4:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-a-language-model-on-a-large-dataset%2F790%2F1 "2020-08-20T16:15:26Z") Hi, I’m getting a memory error when I run the example code for language modeling. I’m interested in pre-training a RoBERTa model using a 25GB text data on a virtual machine with a `v3-8` TPU on Google Cloud Platform. I’m using the following command with `transformers/examples/xla_spawn.py` and `transformers/examples/run_language_modeling.py`. ```auto python xla_spawn.py --num_cores 8 \ run_language_modeling.py \ --output_dir=[*****] \ --config_name=[*****] \ --tokenizer_name=[*****] \ --do_train \ --per_device_train_batch_size 8 \ --gradient_accumulation_steps 128 \ --learning_rate 6e-4 \ --weight_decay 0.01 \ --adam_epsilon 1e-6 \ --adam_beta1 0.9 \ --adam_beta2 0.98 \ --max_steps 500_000 \ --warmup_steps 24_000 \ --save_total_limit 5 \ --save_steps=100_000 \ --block_size=512 \ --train_data_file=[*****] \ --mlm \ --line_by_line ``` When I run this, I get the following error. ```auto 08/20/2020 15:21:07 - INFO - transformers.data.datasets.language_modeling - Creating features from dataset file at [*****] Traceback (most recent call last): File "xla_spawn.py", line 72, in main() File "xla_spawn.py", line 68, in main xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores) File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 395, in spawn start_method=start_method) File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes while not context.join(): File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 108, in join (error_index, name) Exception: process 0 terminated with signal SIGKILL ``` It looks like the script gets killed while it’s loading the training data [here](https://github.com/huggingface/transformers/blob/573bdb0a5d2897ff6c7520ebb38693c7acfbf17e/src/transformers/data/datasets/language_modeling.py#L89-L92). ```python with open(file_path, encoding="utf-8") as f: lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())] ``` When I run the above block of code separately with `transformers/examples/xla_spawn.py`, I get an error. ```auto Traceback (most recent call last): File "xla_spawn.py", line 72, in main() File "xla_spawn.py", line 68, in main xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores) File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 395, in spawn start_method=start_method) File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes while not context.join(): File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 108, in join (error_index, name) Exception: process 0 terminated with signal SIGKILL ``` When I run the above block of code separately using `n1-highmem-16 (16 vCPUs, 104 GB memory)` without TPU, I still get an error. ```auto Traceback (most recent call last): File "debug_load.py", line 7, in lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())] File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/codecs.py", line 321, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) MemoryError ``` Has anyone successfully reproduced the original RoBERTa model or pretrained a language model with a large dataset using Huggingface’s transformers (with TPU)? If so, what are the specifications of your machine? Has this code (`transformers/examples/run_language_modeling.py`) tested on a large dataset? ---
**Author:** ![user123](https://avatars.discourse-cdn.com/v4/letter/u/ed655f/32.png) [@user123](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fuser123)%5C **Post date:** [October 15, 2020, 5:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-a-language-model-on-a-large-dataset%2F790%2F2 "2020-10-15T17:07:12Z")
Same problem here. It seems that `run_language_modeling.py` is not able to deal with very large files. Any help?! @valhalla @lhoestq Thanks ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C **Post date:** [October 16, 2020, 5:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-a-language-model-on-a-large-dataset%2F790%2F3 "2020-10-16T17:27:31Z")
Hi @user123 If you have large dataset, you’ll need to write your own dataset to lazy load examples. Also consider using [datasets](https://github.com/huggingface/datasets) library. It allows you to memory map dataset and cache the processed data, by memory mapping it won’t take too much RAM and by caching you can reuse the processed dataset. ---
**Author:** ![user123](https://avatars.discourse-cdn.com/v4/letter/u/ed655f/32.png) [@user123](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fuser123)%5C **Post date:** [October 21, 2020, 5:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-a-language-model-on-a-large-dataset%2F790%2F4 "2020-10-21T17:00:11Z")
Hi @valhalla Thanks for your suggestion. I modified the `get_dataset` function in `run_language_modeling.py` using `datasets` as explained [here](https://github.com/huggingface/datasets/issues/532) ``` dataset = load_dataset('text', data_files=file_path, split='train') dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True, truncation=True, max_length=args.block_size, ), batched=True) dataset.set_format(type='torch', columns=['input_ids']) ``` it runs without error on TPU but it is superslow. I used `wikitext103` to test the code. Each training step takes less than 1 sec in the original code, but with `datasets` it takes more than 60 sec. Am I missing something or this is expected? ---
**Author:** ![clk](https://avatars.discourse-cdn.com/v4/letter/c/d2c977/32.png) [@clk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fclk)%5C **Post date:** [January 14, 2021, 5:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-a-language-model-on-a-large-dataset%2F790%2F5 "2021-01-14T17:35:53Z")
have you solved this problem? I think i met the same problem as yours… ---
**Author:** ![deathcrush](https://avatars.discourse-cdn.com/v4/letter/d/58f4c7/32.png) [@deathcrush](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdeathcrush)%5C **Post date:** [March 15, 2022, 8:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-a-language-model-on-a-large-dataset%2F790%2F6 "2022-03-15T08:13:10Z")
@valhalla is this expected? It seems to be a performance issue? What is the go to method for training a model on a large amount of data over multiple GPUs? Will the huggingface datasets work with DDP “out of the box”?