# Train a large transformer with Custom Tokenizer/Data

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-a-large-transformer-with-custom-tokenizer-data%2F28350%3E%5C
**Category:** 🤗Transformers\
**Created:** [December 23, 2022, 9:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-a-large-transformer-with-custom-tokenizer-data%2F28350 "2022-12-23T21:44:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![htlerdo](https://avatars.discourse-cdn.com/v4/letter/h/e9c0ed/32.png) [@htlerdo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhtlerdo)%5C
**Post date:** [December 23, 2022, 9:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-a-large-transformer-with-custom-tokenizer-data%2F28350%2F1 "2022-12-23T21:44:51Z")

</div>

I’d like to train large transformer model with my own data and tokenizer. I have billions of examples, so I’d like to avoid using anything that’s pre-trained, I just want to train from scratch. Ideally I’d like to use tf/keras (hoping to upload to Google Cloud and train with TPUs there since I already have TPUs).

What’s the easiest way to do this?

I can think of several options:

1. I dump a dataset where each document is just sequence of integers and the model doesn’t care about tokenization etc… It just learns to predict the next integer given a list of integers. Is there any TF code where I can do this easily? (I expect I will have a few thousand documents where each document have ~million integers/tokens).

2. I implement my tokenizer and the model allows me to plug-in a custom tokenizer. Is there any TF code where I can do this easily?