# Gradient Issue after using torch.no\_grad
**URL:**
**Author:**  [@sasuke-ss1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsasuke-ss1)%5C
**Post date:** [July 27, 2025, 4:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-issue-after-using-torch-no-grad%2F164699%2F1 "2025-07-27T16:25:01Z")
Hi, I am training a diffusion model. I am facing a weird issue where the gradient tracking is not working.
If I run this code segement
```auto
noise_pred = self.unet(
latent_model_input,
t,
encoder_hidden_states=prompt_embeds,
cross_attention_kwargs=cross_attention_kwargs,
return_dict=False,
)[0]
```
I get the output for `noise_pred.requires_grad` as True. But if I run this instead
```auto
with torch.no_grad():
noise_pred = self.unet(
latent_model_input,
t,
encoder_hidden_states=prompt_embeds,
cross_attention_kwargs=cross_attention_kwargs,
return_dict=False,
)[0]
noise_pred = self.unet(
latent_model_input,
t,
encoder_hidden_states=prompt_embeds,
cross_attention_kwargs=cross_attention_kwargs,
return_dict=False,
)[0]
print(noise_pred.requires_grad)
```
I get the result to be False. It seems that using the Unet one time under the torch.no\_grad context turns of gradient tracking somehow. I checked the `requires_grad` for parameters in the Unet and it was True.
Thanks in advance
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [July 27, 2025, 10:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-issue-after-using-torch-no-grad%2F164699%2F2 "2025-07-27T22:25:22Z")
> `with torch.no_grad():`
It seems that tensors originating from the block with this `with` statement may continue to be affected by `no_grad`. In addition, there appear to be cases where the effect remains due to caching…
[https://stackoverflow.com/questions/63785319/pytorch-torch-no-grad-versus-requires-grad-false](https://stackoverflow.com/questions/63785319/pytorch-torch-no-grad-versus-requires-grad-false)
> **[Torch.no\_grad vs requires\_grad = false](https://discuss.pytorch.org/t/torch-no-grad-vs-requires-grad-false/175810)**
>
> I understand that using torch.no\_grad has the benefit of saving memory and computation because the network won’t backpropagate gradients to layers before torch.no\_grad. So, is the following statement correct: If I have a network and only want to...
> **[Autocast and torch.no\_grad() unexpected behaviour](https://discuss.pytorch.org/t/autocast-and-torch-no-grad-unexpected-behaviour/93475)**
>
> import torch from torch.cuda.amp import autocast net = torch.nn.Conv2d(3,3,3,3).to('cuda') input = torch.rand(\[3,3,5,5\],device='cuda') with autocast(): with torch.no\_grad(): y = net(input) z = net(input) print('z...
>
>
> Hmm, I actually think your example is unexpected, can you file a separate bug fo…r it.
>
> \_Originally posted by @ezyang in https://github.com/pytorch/pytorch/issues/105211#issuecomment-1787772959\_
>
>
> If I'm in an autocast context and I do a forward pass of a model in no\_grad, then do another forward pass outside of that no\_grad context but still in the autocast context, arbitrary nodes in the graph will be lost. See example:
>
> \`\`\`Python
> import torch
> import torch.nn as nn
>
> l1 = nn.Sequential(
> nn.Linear(2, 2),
> ).cuda()
>
> l2 = nn.Sequential(
> nn.Linear(2, 2),
> nn.LayerNorm(2),
> ).cuda()
>
> x = torch.randn(2, 2).cuda()
>
> \#################################
> \# Just linear
> \################################
> with torch.cuda.amp.autocast():
> with torch.no\_grad():
> y1 = l1(x)
>
> y1\_2 = l1(x)
> print(y1\_2.grad\_fn) # None
>
> \# Remove autocast
> with torch.no\_grad():
> y1 = l1(x)
>
> y1\_2 = l1(x)
> print(y1\_2.grad\_fn) # AddmmBackward
>
> \#################################
> \# Linear -\> LayerNorm makes output
> \# have grad\_fn
> \################################
> with torch.cuda.amp.autocast():
> with torch.no\_grad():
> y2 = l2(x)
>
> y2\_2 = l2(x)
> print(y2\_2.grad\_fn) # LayerNormBackward
>
> with torch.no\_grad():
> y2 = l2(x)
>
> y2\_2 = l2(x)
> print(y2\_2.grad\_fn) # Still LayerNormBackward
>
> with torch.cuda.amp.autocast():
> with torch.no\_grad():
> y2 = l2(x)
>
> y2\_2 = l2(x)
> print(y2\_2.grad\_fn) # LayerNormBackward
>
> y2\_2.sum().backward()
> print(l2\[0\].weight.grad) # None
> print(l2\[1\].weight.grad) # Not none
> \`\`\`
>
> PyTorch version: 2.1.0
>
> cc @ezyang @gchanan @zou3519 @kadeng @albanD @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @mcarilli @ptrblck @leslie-fang-intel @jgong5