# Gemma3 - shift labels to the right

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgemma3-shift-labels-to-the-right%2F149403%3E%5C
**Category:** 🤗Transformers\
**Created:** [April 8, 2025, 8:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgemma3-shift-labels-to-the-right%2F149403 "2025-04-08T08:57:07Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [April 8, 2025, 11:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgemma3-shift-labels-to-the-right%2F149403%2F2 "2025-04-08T11:42:11Z")

</div>

Hmm…?

> <https://github.com/huggingface/transformers/issues/32944>
>
> \### Feature request
> 
> i believe \`labels\` in the training of causal LMs means the …value to predict at time \`n\`, i.e., the next token. in other words, i'd assume, if \`labels\` is given, it should be already shifted by one in the data loader w.r.t. the \`input\_ids\`.
> 
> 
> however, in \`LlamaForCausalLM.forward()\`, i found the labels are always shifted, silently.
> 
> https://github.com/huggingface/transformers/blob/f1d822ba337499d429f832855622b97d90ac1406/src/transformers/models/llama/modeling\_llama.py#L1205-L1210
> 
> \`\`\`python
> 
> Args:
> labels (\`torch.LongTensor\` of shape \`(batch\_size, sequence\_length)\`, \*optional\*):
> Labels for computing the masked language modeling loss. Indices should either be in \`\[0, ...,
> config.vocab\_size\]\` or -100 (see \`input\_ids\` docstring). Tokens with indices set to \`-100\` are ignored
> (masked), the loss is only computed for the tokens with labels in \`\[0, ..., config.vocab\_size\]\`.
> 
> \`\`\`
> 
> ...
> 
> \`\`\`python
> 
> 
> if labels is not None:
> # Shift so that tokens \< n predict n
> shift\_logits = logits\[..., :-1, :\].contiguous()
> shift\_labels = labels\[..., 1:\].contiguous()
> # Flatten the tokens
> loss\_fct = CrossEntropyLoss()
> shift\_logits = shift\_logits.view(-1, self.config.vocab\_size)
> shift\_labels = shift\_labels.view(-1)
> # Enable model parallelism
> shift\_labels = shift\_labels.to(shift\_logits.device)
> loss = loss\_fct(shift\_logits, shift\_labels)
> 
> \`\`\`
> 
> i found it quite unexpected hence calling it "silently". as this is for a causal LM, shouldn't it be not shifting the labels by default? in modeling GPT2, this is at least documented explicitly.
> 
> https://github.com/huggingface/transformers/blob/f1d822ba337499d429f832855622b97d90ac1406/src/transformers/models/gpt2/modeling\_gpt2.py#L1309-1314
> 
> in gemma2, it has the same behavior and no explicit mentioning in the docstring.
> 
> https://github.com/huggingface/transformers/blob/f1d822ba337499d429f832855622b97d90ac1406/src/transformers/models/gemma2/modeling\_gemma2.py#L978-L982
> 
> i think at least we should force the docstring to mention this, if making a change is too dangerous at this point. 
> 
> \### Motivation
> 
> i didn't expect this behavior and used my data loader, which does the shifting already, as i believe that is what \`labels\` should mean. as a result, i ended up finetuning a model to predict the next next token, which outputted gibberish.
> 
> 
> 
> \### Your contribution
> 
> \- hopefully leaving this issue helps communication across users
> \- i can make a one line change in the docstring.
> \- not sure how exactly, but if this potential misunderstanding could be checked, it'd be great. technically, we can check if the labels are already shifted. though i don't know where is the best place for this.

> [@GPT-2 shift logits and labels](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-2-shift-logits-and-labels%2F812)%3A
>
> I am working with GPT-2 and I was looking at the LM head and how it performs the forward pass when labels are provided: [https://huggingface.co/transformers/\_modules/transformers/modeling\_gpt2.html#GPT2LMHeadModel](https://huggingface.co/transformers/_modules/transformers/modeling_gpt2.html#GPT2LMHeadModel) It looks like the logits are shifted right (last value is ignored) and the labels are shifted left (first value is ignored). Why are the logits and labels shifted in different directions?

> [@Shifting ids to the right when training GPT-2 on text generation?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshifting-ids-to-the-right-when-training-gpt-2-on-text-generation%2F5308)%3A
>
> Hello, I am slightly confused how to exactly prepare the training data for GPT-2 text generation. In order to train you have to provide input\_ids (inputs) and labels (outputs). Both are supposed to be lists of token indices. This is the easy part. Question: Are inputs\_ids and labels supposed to be absolutely identical, or are the labels supposed to be input\_ids shifted one element to the right? Best, Tristan

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgemma3-shift-labels-to-the-right%2F149403)._