# Tokenizer.train() running out of memory

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-train-running-out-of-memory%2F31355%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [February 9, 2023, 1:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-train-running-out-of-memory%2F31355 "2023-02-09T01:15:32Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![5GreatApes](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/5greatapes/32/14044_2.png) [@5GreatApes](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2F5GreatApes)%5C
**Post date:** [February 9, 2023, 1:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-train-running-out-of-memory%2F31355%2F1 "2023-02-09T01:15:32Z")

</div>

I am trying to train a tokenizer using the following code:

```
tokenizer= ByteLevelBPETokenizer()

tokenizer.train(files=paths, vocab_size=52_000, min_frequency=2, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])

# Save files to disk
tokenizer.save_model("tokenizer")

```

It works fine when using a small dataset but when using my full dataset it reads the following error:

[00:00:00] Pre-processing files (485 Mo) ████████████████████████████████████████████████ 100%  
memory allocation of 21474836480 bytes failed

My system has 16gb of RAM. Is there a way around this issue that isn’t upgrading RAM? I’m not finding solutions online. Thanks