# Gpt-oss training on A100 - OOM error
**URL:**
**Author:**  [@clementlr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fclementlr)%5C
**Post date:** [November 20, 2025, 8:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-oss-training-on-a100-oom-error%2F170658%2F1 "2025-11-20T08:26:53Z")
Dear community,
I am following OpenAI/HF cookbook to fine-tune gpt-oss:
[Fine-tuning with gpt-oss and Hugging Face Transformers](https://cookbook.openai.com/articles/gpt-oss/fine-tune-transfomers)
I tried training gpt-oss with A100 GPU that comes with 80 GB vRAM but got the below OOM error.
I already set my trl config as below to try to reduce memory pressure.
I appreciate your feedback.
training\_args = SFTConfig(
```
learning_rate=2e-4,
```
gradient\_checkpointing=True,
num\_train\_epochs=1,
logging\_steps=1,
per\_device\_train\_batch\_size=4,
gradient\_accumulation\_steps=4,
max\_length=1024,
warmup\_ratio=0.03,
lr\_scheduler\_type=“cosine\_with\_min\_lr”,
lr\_scheduler\_kwargs={“min\_lr\_rate”: 0.1},
output\_dir=“gpt-oss-20b-multilingual-reasoner”,
push\_to\_hub=True,
)
from trl import SFTTrainer
trainer = SFTTrainer(
model=peft\_model,
args=training\_args,
train\_dataset=dataset,
processing\_class=tokenizer,
)
trainer.train()
> > > trainer.train()
0%| | 0/63 [00:43\
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 20, 2025, 6:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-oss-training-on-a100-oom-error%2F170658%2F2 "2025-11-20T18:05:35Z")
Perhaps [real OOM](https://huggingface.co/datasets/John6666/forum2/blob/main/gpt_oss_ft_a100_oom_1.md)?
---
**Author:**  [@DhineshR](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDhineshR)%5C
**Post date:** [November 20, 2025, 6:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-oss-training-on-a100-oom-error%2F170658%2F3 "2025-11-20T18:31:11Z")
Do you want to train in fp32 precision? If not required, try bf16 precision. I see ` convert_to_fp32` in the stack trace and hence this suggestion.
Also, make sure the training data shapes are as expected.
---
**Author:**  [@clementlr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fclementlr)%5C
**Post date:** [November 20, 2025, 9:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-oss-training-on-a100-oom-error%2F170658%2F4 "2025-11-20T21:41:26Z")
> [@DhineshR](#):
>
> convert\_to\_fp32
Dear DhineshR, thanks for your reply.
The convert\_to\_fp32 you mentioned is part of error message but I did not set such parameter.
The torch dtype of the model config is the one recommended in openAI cookbook:
import torch
from transformers import AutoModelForCausalLM, Mxfp4Config
quantization\_config = Mxfp4Config(dequantize=True)
model\_kwargs = dict(
attn\_implementation=“eager”,
torch\_dtype=torch.bfloat16,
quantization\_config=quantization\_config,
use\_cache=False,
device\_map=“auto”,
)
I appreciate your feedback.
model = AutoModelForCausalLM.from\_pretrained(“openai/gpt-oss-20b”, \*\*model\_kwargs)
---
**Author:**  [@DhineshR](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDhineshR)%5C
**Post date:** [November 21, 2025, 4:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-oss-training-on-a100-oom-error%2F170658%2F5 "2025-11-21T04:30:37Z")
Though not set explicitly, the stack trace shows at some point the tensors are converted to fp32.
You could try increasing the batch size by 1 starting from 1 and find the breaking point.
You could also calculate size of activations for your batch\_size, max\_length, and precision of training. And, check if there is enough space left in the GPU for activations after allocating required amount of memory for fine-tuning.
---
**Author:**  [@clementlr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fclementlr)%5C
**Post date:** [November 21, 2025, 1:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-oss-training-on-a100-oom-error%2F170658%2F6 "2025-11-21T13:43:03Z")
Dear **[DhineshR](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdhineshr)%2C** thanks for your feedback.
I try downsizing:
_per\_device\_train\_batch\_size=1, gradient\_accumulation\_steps=1,_
But still results in OOM error message.
I also tried to:
_model\_kwargs = dict(_
attn\_implementation=“eager”,
_torch\_dtype=torch.bfloat16,_
_quantization\_config=quantization\_config,_
_use\_cache=False,_
_#device\_map=“auto”,_
_)_
as I noticed warning info about model beeing on multiple device (although runnning on A100 single GPU) after:
_trainer = SFTTrainer(_
_… model=peft\_model,_
_… args=training\_args,_
_… train\_dataset=dataset,_
_… processing\_class=tokenizer,_
_… )_
_The model is already on multiple devices. Skipping the move to device specified in `args`._
But still result in OOM error.
Do you have other thought ?
Right now I am considering testing on H100 but not sure this will make it as VRAM is equal to 80 GB as well.
Kind regards
---
**Author:**  [@DhineshR](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDhineshR)%5C
**Post date:** [November 22, 2025, 8:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-oss-training-on-a100-oom-error%2F170658%2F7 "2025-11-22T08:00:23Z")
I’d suggest checking the number of GPUs you have and make sure they are as expected by running `nvidia-smi`.
If you want to only use one GPU, try setting CUDA\_VISIBLE\_DEVICES=0 environment variable when running the training script.
However, I believe 80GB is not enough to fientune gpt-oss 20B at bf16 precision. There are many guides available online that would help you estimate the amount of memory required to fine-tune. I’d suggest performing those calculations for your use case.
---
**Author:**  [@clementlr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fclementlr)%5C
**Post date:** [November 26, 2025, 5:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-oss-training-on-a100-oom-error%2F170658%2F8 "2025-11-26T17:35:14Z")
> [@DhineshR](#):
>
> CUDA\_VISIBLE\_DEVICES
Dear DhineshR, thanks for your feedback.
In [OpenAI cookbook](https://cookbook.openai.com/articles/gpt-oss/fine-tune-transfomers) documentation they state that:
\*\*\*Note: This notebook is designed to be run on a single H100 GPU with 80GB of memory. If you have access to a smaller GPU, you can reduce the batch size and sequence length in the hyperparameters below.
* * *
I tried additional HW set up testing with **Nvidia H100** and **Nvidia 4xA10G large (which comes with 92 GB VRAM).** In both cases I still get OOM error.
Any other idea ? Tx
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 26, 2025, 11:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-oss-training-on-a100-oom-error%2F170658%2F9 "2025-11-26T23:18:24Z")
When considering the difference compared to the H100 GPU environment shown in OpenAI’s Notebook, the overhead of executing `torch.compile` might be the culprit. Next in line are differences in PyTorch, CUDA, or Transformers versions.
Either way, I think VRAM is quite tight and failures are likely to occur from even minor differences, so personally, I recommend using QLoRA.
* * *
## Main causes of OOM on a single H100
1. **You’re really doing BF16 LoRA on a 20B MoE, not FP4 training**
2. **Attention uses full L×L “eager” kernels**
3. **80GB H100 is “just enough”, so any extra overhead breaks it**
4. **This is a known pattern, not just your setup**
* * *
## Practical solutions / workarounds on a single H100
### A. Best fix: switch to an optimized QLoRA stack (Unsloth)
- Unsloth’s GPT-OSS guide: **gpt-oss-20B fine-tuning fits in ~14GB VRAM** ; they recommend ≥16GB for stable runs. ([Unsloth](https://unsloth.ai/blog/gpt-oss "Fine-tune gpt-oss with Unsloth"))
- Their key point: Unsloth keeps GPT-OSS in a quantized / optimized form and uses Flex Attention, cutting VRAM by ≈70–80% compared to BF16 LoRA.
If you can change stack, this is by far the cleanest fix: your H100 then has _massive_ headroom instead of being at the cliff edge.
### B. If you must stay on HF/TRL BF16 LoRA notebook
1. **Shrink “tokens per step” hard**
2. **Remove avoidable overhead**
3. **Consider better attention kernels on H100**
Even with all of this, BF16 LoRA on gpt-oss-20B will always be near the 80GB limit; QLoRA / Unsloth is the only way to make it “comfortable”.
* * *
## Key links
- OpenAI Cookbook – Fine-tuning gpt-oss with Transformers (original notebook, H100-80GB target) ([OpenAI Cookbook](https://cookbook.openai.com/articles/gpt-oss/fine-tune-transfomers "Fine-tuning with gpt-oss and Hugging Face Transformers"))
- Unsloth guide – “gpt-oss: How to run and fine-tune” (explains 65GB vs 14GB VRAM for 20B) ([Unsloth Docs](https://docs.unsloth.ai/models/gpt-oss-how-to-run-and-fine-tune "gpt-oss: How to Run & Fine-tune"))
- Zenn hardware outline for GPT-OSS fine-tuning (20B LoRA BF16 ≈ 60GB+ VRAM, QLoRA ≈ 14GB+) ([Zenn](https://zenn.dev/hiraoku/scraps/126ba194607f5e "OpenAI GPT-OSSファインチューニング実践手順書のアウトライン"))
---
**Author:**  [@clementlr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fclementlr)%5C
**Post date:** [November 27, 2025, 6:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-oss-training-on-a100-oom-error%2F170658%2F10 "2025-11-27T06:00:44Z")
Dear John6666,
thanks for the heads up analysis your provided.
I was reading unsloth documentation yesterday so this is probably where I am going to look next.
I’ll update the thread while I will be able to run new tests.
Kind regards