# Overflow when using DeepSpeed for GPT-J (training aborts)
**URL:**
**Author:**  [@Eichhof](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FEichhof)%5C
**Post date:** [August 30, 2022, 9:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Foverflow-when-using-deepspeed-for-gpt-j-training-aborts%2F22320%2F1 "2022-08-30T21:32:34Z")
Hello
I’m fine-tuning EleutherAI/gpt-j-6B on a conversational dataset (TriviaQA with 80’000 rows) using DeepSpeed on a 24 GB GPU and 150 GB of RAM. The DeepSpeed config file and the HF TrainingArguments are shown below.
During training I’m getting often the message `OVERFLOW! Rank 0 Skipping step.`. The loss scale is then reduced. After short time training then also aborts with `Current loss scale already at minimum - cannot decrease scale anymore. Exiting run.` When I lock at the loss it is also wiggling around.
Is there an error in my DeepSpeed config or the TrainingArguments or how should I change the hyperparameters? Hyperparameter tuning is pretty difficult due to the large model size and long runtime.
Second, I’m also getting the warning `FP16 params for CPUAdam may not work on AMD CPUs`. I’m using an AMD CPU and FP16. Is this a problem?
```auto
training_args = TrainingArguments(
disable_tqdm=True,
output_dir='/checkpoints',
save_total_limit=5,
logging_dir='/logs',
num_train_epochs=3,
evaluation_strategy='epoch',
save_strategy='epoch',
logging_steps=10,
overwrite_output_dir=True,
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
gradient_accumulation_steps=4,
eval_accumulation_steps=4,
gradient_checkpointing=True,
max_grad_norm=0.5,
lr_scheduler_type="constant_with_warmup",
learning_rate=7e-5,
warmup_ratio=0.05,
weight_decay=0.1,
fp16_full_eval=True,
fp16=True,
fp16_opt_level='O1',
deepspeed="configs/ds_config.json",
report_to=['tensorboard']
)
```
configs/ds\_config.json:
```auto
{
"fp16": {
"enabled": "auto",
"loss_scale": 0,
"loss_scale_window": 1000,
"initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 1,
"opt_level": "O1"
},
"zero_optimization": {
"stage": 2,
"offload_optimizer": {
"device": "cpu"
},
"allgather_partitions": true,
"allgather_bucket_size": 2e8 ,
"reduce_bucket_size": 2e8,
"overlap_comm": true,
"reduce_scatter": true,
"contiguous_gradients": true,
"sub_group_size": 1e7,
"stage3_prefetch_bucket_size": "auto",
"stage3_param_persistence_threshold": "auto",
"stage3_max_live_parameters": 1e7,
"stage3_max_reuse_distance": 1e7,
"stage3_gather_16bit_weights_on_model_save": true
},
"optimizer": {
"type": "AdamW",
"params": {
"lr": "auto",
"betas": "auto",
"eps": "auto",
"weight_decay": "auto"
}
},
"scheduler": {
"type": "WarmupLR",
"params": {
"warmup_min_lr": "auto",
"warmup_max_lr": "auto",
"warmup_num_steps": "auto"
}
},
"activation_checkpointing": {
"partitioned_activations":true,
"number_checkpoints": 100,
"contiguous_memory_optimization": true,
"cpu_checkpointing": true,
"profile": true,
"synchronize_checkpoint_boundary": true
},
"tensorboard": {
"enabled": true,
"output_path": "/cluster/scratch/wrafael/cache/models/logs",
"job_name": "finetune_gpt_j_6b"
},
"steps_per_print": 100,
"zero_allow_untested_optimizer": true,
"gradient_accumulation_steps": "auto",
"gradient_clipping": "auto",
"train_batch_size": "auto",
"train_micro_batch_size_per_gpu": "auto",
"wall_clock_breakdown": false,
"memory_breakdown": false
}
```
---
**Author:**  [@Boman](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBoman)%5C
**Post date:** [December 5, 2022, 1:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Foverflow-when-using-deepspeed-for-gpt-j-training-aborts%2F22320%2F2 "2022-12-05T01:21:31Z")
Hi Eichhof,
I’ve met the same problem as yours, yet I cannot figure it out.
I just found a suggestion from [@myleott](https://github.com/facebookresearch/fairseq/issues/1529#issuecomment-567955547) and heading to try it.
And if you have fixed this problem in another way, I sincerely look forward to your sharing and help.
---
**Author:**  [@maveriq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmaveriq)%5C
**Post date:** [February 14, 2023, 8:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Foverflow-when-using-deepspeed-for-gpt-j-training-aborts%2F22320%2F3 "2023-02-14T20:44:29Z")
Hi. I am running into similar issue. Did anyone find a reason? @Boman
---
**Author:**  [@ticoneva](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fticoneva)%5C
**Post date:** [March 8, 2023, 8:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Foverflow-when-using-deepspeed-for-gpt-j-training-aborts%2F22320%2F4 "2023-03-08T08:47:56Z")
I believe you have to fine-tune it in bf16 instead of fp16.
In `TrainingArguments`:
```python
training_args = TrainingArguments(
...
bf16 = True,
...
)
```
In DeepSpeed config:
```json
{
...
"bf16": { "enabled": auto }
...
}
```
---
**Author:**  [@kastan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkastan)%5C
**Post date:** [March 9, 2023, 5:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Foverflow-when-using-deepspeed-for-gpt-j-training-aborts%2F22320%2F5 "2023-03-09T17:19:18Z")
See also this comment about root cause, implications and 4 next steps: [FloatingPointError: Minimum loss scale reached (0.0001). · Issue #1529 · facebookresearch/fairseq · GitHub](https://github.com/facebookresearch/fairseq/issues/1529#issuecomment-567955547)