# LM from Scratch for Tensorflow

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flm-from-scratch-for-tensorflow%2F3219%3E%5C
**Category:** 🤗Transformers\
**Created:** [January 15, 2021, 5:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flm-from-scratch-for-tensorflow%2F3219 "2021-01-15T17:50:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jojo2k](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jojo2k](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjojo2k)%5C
**Post date:** [January 15, 2021, 5:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flm-from-scratch-for-tensorflow%2F3219%2F1 "2021-01-15T17:50:57Z")

</div>

Hi,  
I was following this tutorial to train a LM from scratch: [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train)  
The result is a pytorch model. Though I need one for Tensorflow. Is there an easy way to convert it?

I tried to modify the training code by using TFTrainer, TFBertForModelLM instead. but TFTrainer is causing trouble with the Data\_collator and LineByLineTextDataset objects.  
When intitializing the trainer with the data collator i get the error: **init** () got an unexpected keyword argument ‘data\_collator’  
When calling trainer.train() (without collator) I receive the error: LineByLineTextDataset object has no attribute ‘\_variant\_tensor’ .

---

<div class="post-metadata">

**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJung)%5C
**Post date:** [January 16, 2021, 3:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flm-from-scratch-for-tensorflow%2F3219%2F2 "2021-01-16T03:05:01Z")

</div>

Hi! Here’s a nice example of custom TF MLM learning on XLM-Roberta with Kaggle TPU

[https://www.kaggle.com/riblidezso/finetune-xlm-roberta-on-jigsaw-test-data-with-mlm](https://www.kaggle.com/riblidezso/finetune-xlm-roberta-on-jigsaw-test-data-with-mlm)

---

<div class="post-metadata">

**Author:** ![jojo2k](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jojo2k](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjojo2k)%5C
**Post date:** [January 18, 2021, 1:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flm-from-scratch-for-tensorflow%2F3219%2F3 "2021-01-18T13:30:13Z")

</div>

Thank you! Will check that link later. Sometimes I wonder why I don’t generally start my research on kaggle 😃 Has been a helpful source quite a few times.