# How to output loss from model.generate()?
**URL:**
**Author:**  [@mittu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmittu)%5C
**Post date:** [April 20, 2022, 10:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F1 "2022-04-20T10:56:18Z")
I need the probability distribution of word generation to calculate the loss in my original loss function.
In particular, the model differs from a normal loss function in that it generates sentences in the same way as when testing.
Therefore, I have tried to calculate the loss using model.generate(), but this method does not leave me with the calculated graph needed to calculate the gradient. Could this be easily solved by passing a special argument to the method? Or is there an equally simple solution? Or do I have to implement my own function that generates the text in a way that leaves the computed graph?
---
**Author:**  [@tomroth1001](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftomroth1001)%5C
**Post date:** [April 22, 2022, 1:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F2 "2022-04-22T01:36:17Z")
I canât answer many of your questions, but I did find this code snippet useful to get a computational graph with `generate()`:
```auto
from undecorated import undecorated
from types import MethodType
generate_with_grad = undecorated(model.generate)
model.generate_with_grad = MethodType(generate_with_grad, model)
```
The `generate()` function has a `no_grad` decorator that stops the computational graph being returned, and this code just removes the decorator and leaves the rest of the generate function unchanged.
---
**Author:**  [@mittu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmittu)%5C
**Post date:** [April 22, 2022, 5:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F3 "2022-04-22T05:37:36Z")
Thanks for sharing your simple solution! đ
Iâll give this method a try!
---
**Author:**  [@mittu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmittu)%5C
**Post date:** [April 22, 2022, 6:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F4 "2022-04-22T06:03:56Z")
thanks! [tomroth1001](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftomroth1001)!
I got scores with computational graph!
---
**Author:**  [@halamvac](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhalamvac)%5C
**Post date:** [October 1, 2022, 7:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F5 "2022-10-01T19:38:05Z")
Hello, I tried this method to retain the computational graph and it works. However when I try to backpropagate some loss computed from the generation scores I get an error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
Which is triggered somewhere in generation\_logits\_process.py.
Last few lines of the trace:
next\_token\_scores\_processed = logits\_processor(
File â/home/halamvac/venvs/venv39/lib/python3.9/site-packages/transformers/generation\_logits\_process.pyâ, line 92, in **call**
scores = processor(input\_ids, scores)
File â/home/halamvac/venvs/venv39/lib/python3.9/site-packages/transformers/generation\_logits\_process.pyâ, line 161, in **call**
score = torch.gather(scores, 1, input\_ids)
@mittu @tomroth1001 Did any of you make it work with backpropagating the gradient?
---
**Author:**  [@tomroth1001](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftomroth1001)%5C
**Post date:** [October 1, 2022, 11:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F6 "2022-10-01T23:30:40Z")
Not sure, sorry. It worked for my case.
The clichĂ© advice is to make sure all your packages are up to date and then try againâmight be a bug.
---
**Author:**  [@halamvac](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhalamvac)%5C
**Post date:** [October 3, 2022, 11:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F7 "2022-10-03T11:50:05Z")
Thanks for the tip but I already have the latest versions. Might I ask what versions of pytorch and transformers are you using?
---
**Author:**  [@mittu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmittu)%5C
**Post date:** [October 4, 2022, 7:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F8 "2022-10-04T19:58:12Z")
Iâm using transformers 4.19.2 and torch 1.11.0+cu113, and it still work with out error
And my minimum code is here.
```auto
from transformers import BartForConditionalGeneration
from undecorated import undecorated
model = BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn")
generate_with_grad = undecorated(model.generate)
model.generate_with_grad = MethodType(generate_with_grad, model)
output=model.generate_with_grad(
input_ids=input_ids,
output_scores=True,return_dict_in_generate=True,
output_hidden_states = True,
)
```
According to error message, there is an in-place operation at somewhere, so It would be a good idea to do a backward look, commenting out each calculation process line by line from the end to the beginning, to find which calculation process has the problem. Note backward() is possible only for scalars, so for multidimensional tensors, it is better to sum or average as appropriate.
---
**Author:**  [@Hottuck](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHottuck)%5C
**Post date:** [January 26, 2024, 4:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F9 "2024-01-26T04:22:42Z")
I have a question.
Are there concerns that temporarily removing @torch.no\_grad() could lead to learning during the test phase?
---
**Author:**  [@jhuang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjhuang)%5C
**Post date:** [April 18, 2024, 1:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F10 "2024-04-18T01:01:16Z")
I got the following error while trying the [un-decorating method](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F8).
```auto
TypeError: _DecoratorContextManager.clone() got an unexpected keyword argument 'input_ids'
```
Package versions
```auto
torch == 2.1.0+cu121
transformers == 4.33.2
undecorated == 0.3.0
```
---
**Author:**  [@liyongkang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fliyongkang)%5C
**Post date:** [April 18, 2024, 6:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F11 "2024-04-18T18:30:22Z")
@jhuang Hi Jiaji, I have the same problem as you, did you fix it? Thank you~
---
**Author:**  [@jhuang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjhuang)%5C
**Post date:** [April 19, 2024, 1:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F12 "2024-04-19T01:16:53Z")
Yes, this works for me
```auto
output = model.generate. __wrapped__ (model, input_ids, output_scores=True, return_dict_in_generate=True)
```
You should see `output.scores[0].requires_grad=True`.
Note: `model.generate. __wrapped__ ` is the actual function `generate()`, getting rid of the `@no_grad()` decorator. But as the function is a method of a python class, weâd have to input something for its first argument `self`. So here I simply put `model` as first argument.
The [undecorate method](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F8) doesnât work, as it gets the decorator `@no_grad` itself rather than the actual `generate()` function.
---
**Author:**  [@liyongkang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fliyongkang)%5C
**Post date:** [April 25, 2024, 9:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F13 "2024-04-25T09:24:08Z")
Hi JiajiïŒ Sorry for the late reply. I have been testing according to the method you provided these days, thank you very much!
Thanks again for your quick response.
---
**Author:**  [@jhuang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjhuang)%5C
**Post date:** [April 25, 2024, 5:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F14 "2024-04-25T17:22:11Z")
@liyongkang No problem! However, I do observe some notable difference in gradients against the following approach:
1. generate as usual with no\_grad
2. forward pass of prompt concatenated with generated tokens.
3. calculate some loss function and call loss.backward()
I am curious about your test outcome. Let me know what you find?
---
**Author:**  [@liyongkang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fliyongkang)%5C
**Post date:** [April 25, 2024, 9:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F15 "2024-04-25T21:27:35Z")
Hi Jiaji, Maybe it is because output.scores are not the raw logits, you can [check this](https://github.com/huggingface/transformers/pull/28667) . And set `output_logits=True` then it will return the raw logits.
I originally hoped that this gradient would help me have more control over the generated content. HoweverïŒ I found that I couldnât even control the length of the generated text (I can only specify the maximum length, but not the generated text to a certain length. If the length is uncertain, I cannot calculate my loss. ), so I switched to other strategies.
Thanks again for your help.
---
**Author:**  [@jhuang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjhuang)%5C
**Post date:** [April 26, 2024, 5:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F16 "2024-04-26T17:29:07Z")
Iâm pretty sure in both cases, they are raw logits. In fact, the difference between the logits in these two cases are tiny. But the backward() results in notable difference in gradient. Iâd say the steps 1-3 list above is a safer choice.
---
**Author:**  [@KKKe3](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKKKe3)%5C
**Post date:** [January 7, 2025, 11:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-output-loss-from-model-generate%2F16999%2F17 "2025-01-07T11:06:11Z")
Thank you for the suggested methods. Regarding the differences in output\_logits between the two approaches, I believe itâs due to randomness in the inference process. When comparing the outputs between model.generate (without gradient computation) and the unwrapped model.generate. **wrapped** (with grad), I found that their decoded results are almost identical. Interestingly, the token distributions are nearly the same for approximately the first half of the sequence, but gradually diverge as the sequence length increases. This behavior is expected and can be attributed to the randomness introduced by the sampling strategy during generation.
More specifically:
- The initial token distributions are highly consistent between both methods
- Divergence increases with sequence length
- Final outputs remain semantically similar despite minor differences
- This pattern aligns with the expected behavior of stochastic sampling methods
```auto
logits_with_grad: tensor([[[-14.6484, -10.0078, -12.7109, ..., 11.3516, 11.3516, 11.3516]],
[[-5.3594, -3.2188, -5.5977, ..., 7.5000, 7.5000, 7.5000]],
[[-21.5312, -16.6562, -18.5938, ..., 12.9766, 12.9688, 12.9766]],
...,
[[3.4531, 2.1445, -1.0059, ..., -0.3789, -0.3787, -0.3787]],
[[14.2656, 3.1895, 0.5615, ..., -1.7119, -1.7119, -1.7119]],
[[1.0234, -0.8833, -1.7861, ..., -0.6836, -0.6831, -0.6831]]],
device='cuda:0', grad_fn=)
logits_no_grad: tensor([[[-14.6484, -10.0078, -12.7109, ..., 11.3516, 11.3516, 11.3516]],
[[-5.3594, -3.2188, -5.5977, ..., 7.5000, 7.5000, 7.5000]],
[[-21.5312, -16.6562, -18.5938, ..., 12.9766, 12.9688, 12.9766]],
...,
[[3.2891, 4.6953, 1.4385, ..., -1.8242, -1.8242, -1.8242]],
[[2.5020, 2.7227, -0.5269, ..., -1.3711, -1.3711, -1.3711]],
[[3.3125, 4.7188, 0.6821, ..., -1.4043, -1.4043, -1.4043]]],
device='cuda:0')
```