# How to train causal language model

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-train-causal-language-model%2F69586%3E%5C
**Category:** 🤗Transformers\
**Created:** [January 18, 2024, 7:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-train-causal-language-model%2F69586 "2024-01-18T07:01:27Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![MarkWuICT](https://avatars.discourse-cdn.com/v4/letter/m/d78d45/32.png) [@MarkWuICT](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMarkWuICT)%5C
**Post date:** [January 18, 2024, 7:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-train-causal-language-model%2F69586%2F1 "2024-01-18T07:01:27Z")

</div>

I found an example in [transformers/examples/pytorch/language-modeling/run\_clm.py]  
According to the script, the trainer use dafault\_data\_collator to model causal language modelling.([transformers/examples/pytorch/language-modeling/run\_clm.py at 98dda8ed03ac3f4af5733bdddaa1dab6a81e15c1 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/98dda8ed03ac3f4af5733bdddaa1dab6a81e15c1/examples/pytorch/language-modeling/run_clm.py#L604))  
Shouldn’t we use DataCollatorForLanguageModeling to shift input and output by 1 token instead? It seems that dafault\_data\_collator can’t achieve this goal.