# Tokenizer taking lot of memory
**URL:**
**Author:**  [@siva1777](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsiva1777)%5C
**Post date:** [July 22, 2021, 4:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-taking-lot-of-memory%2F8597%2F1 "2021-07-22T16:17:30Z")
I am using the BertTokenizerFast from transformers library to encode my text data. I am using pre-trained model for that “bert-base-uncased”. I have a data set of 7 Million rows which is around 2GB. I am using 64GB RAM.
When I am trying to convert into vectors using the model BertModel with same pre-trained model"bert-base-uncased", It cannot convert even 10,000 encoded vectors. It is allocating by saying very high memory needed (100s of GB).
I am using the reference code from here [reference blog](https://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/)
---
**Author:**  [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [July 22, 2021, 8:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-taking-lot-of-memory%2F8597%2F2 "2021-07-22T20:14:11Z")
You are probably trying to do “all data at once” which of course will not work. Try chunking your data into batches and process them one by one.
---
**Author:**  [@siva1777](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsiva1777)%5C
**Post date:** [August 30, 2021, 12:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-taking-lot-of-memory%2F8597%2F3 "2021-08-30T12:03:15Z")
@BramVanroy Thank you. It worked for me.
---
**Author:**  [@sr5434](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsr5434)%5C
**Post date:** [April 16, 2023, 9:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-taking-lot-of-memory%2F8597%2F4 "2023-04-16T21:22:05Z")
What code did you use? I am using the TrainDataset function.