# FREQUENT LOSS SPIKING in CONTINUE TRAINING LLM

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffrequent-loss-spiking-in-continue-training-llm%2F55080%3E%5C
**Category:** 🤗Transformers\
**Created:** [September 15, 2023, 2:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffrequent-loss-spiking-in-continue-training-llm%2F55080 "2023-09-15T02:41:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![hunggggg](https://avatars.discourse-cdn.com/v4/letter/h/9dc877/32.png) [@hunggggg](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhunggggg)%5C
**Post date:** [September 15, 2023, 2:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffrequent-loss-spiking-in-continue-training-llm%2F55080%2F1 "2023-09-15T02:41:01Z")

</div>

Hi everyone, I am trying to continue training Llama-2-hf both chat and non chat version with a custom dataset crawled on the internet using Transformer together with PEFT and QLORA. However with different configs, I experienced a very weird loss curve as such:

- Chat version: ![Chat_version](https://us1.discourse-cdn.com/hellohellohello/original/3X/e/b/ebc0ca0c9db7754daaf9d9494980f9d9eaf07c28.png).
- Non-chat version: Similar pattern but can’t add it due to being a new member.

The configs that I use in each version:

- Chat:

```auto
train_name: baseline
model_source: NousResearch
model_name: Llama-2-7b-chat-hf
bnb_cfg:
  load_in_4bit: True
  bnb_4bit_use_double_quant: True
  bnb_4bit_quant_type: nf4
  bnb_4bit_compute_dtype: torch.bfloat16
lora_cfg:
  peft_type: null
  auto_mapping: null
  base_model_name_or_path: null
  revision: null
  task_type: CAUSAL_LM
  inference_mode: False
  r: 64
  target_modules: null
  lora_alpha: 32
  lora_dropout: 0.05
  fan_in_fan_out: False
  bias: none
  modules_to_save: null
  init_lora_weights: True
  layers_to_transform: null
  layers_pattern: null
train_cfg:
  per_device_train_batch_size: 8
  gradient_accumulation_steps: 16
  warmup_ratio: 0.03
  max_steps: -1
  learning_rate: 8.e-5
  weight_decay: 0.0001
  fp16: True
  logging_steps: 1
  num_train_epochs: 5
  optim: paged_adamw_32bit
  evaluation_strategy: steps
  lr_scheduler_type : constant
  do_train: True
  do_eval: True
  eval_steps: 200
  save_strategy: steps
  save_steps: 100
  group_by_length: True
  dataloader_num_workers: 0
  dataloader_drop_last: True
  ddp_find_unused_parameters: False
max_seq_length: 512

```

- Non-chat:

```auto
train_name: baseline
model_source: NousResearch
model_name: Llama-2-7b-hf
bnb_cfg:
  load_in_4bit: True
  bnb_4bit_use_double_quant: True
  bnb_4bit_quant_type: nf4
  bnb_4bit_compute_dtype: torch.bfloat16
lora_cfg:
  peft_type: null
  auto_mapping: null
  base_model_name_or_path: null
  revision: null
  task_type: CAUSAL_LM
  inference_mode: False
  r: 16 #experience with rank = 8 but similar loss pattern
  target_modules: null
  lora_alpha: 32
  lora_dropout: 0.05
  fan_in_fan_out: False
  bias: none
  modules_to_save: null
  init_lora_weights: True
  layers_to_transform: null
  layers_pattern: null
train_cfg:
  per_device_train_batch_size: 8
  gradient_accumulation_steps: 16
  warmup_ratio: 0.03
  max_steps: -1
  learning_rate: 8.e-5
  weight_decay: 0.0001
  fp16: True
  logging_steps: 1
  num_train_epochs: 5
  optim: paged_adamw_32bit
  evaluation_strategy: steps
  lr_scheduler_type : constant
  do_train: True
  do_eval: True
  eval_steps: 200
  save_strategy: steps
  save_steps: 100
  group_by_length: True
  dataloader_num_workers: 0
  dataloader_drop_last: True
  ddp_find_unused_parameters: False
max_seq_length: 512

```

Can anyone explain why this is happening and provide me with any suggestion to improve the continue training process? Thank everyone a lot! I am looking forward to your responses!

---

<div class="post-metadata">

**Author:** ![cvamsi815](https://avatars.discourse-cdn.com/v4/letter/c/43a26b/32.png) [@cvamsi815](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcvamsi815)%5C
**Post date:** [April 3, 2024, 11:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffrequent-loss-spiking-in-continue-training-llm%2F55080%2F2 "2024-04-03T23:14:56Z")

</div>

@hunggggg were you able to solve this? I am facing similar issue, can you please help

---

<div class="post-metadata">

**Author:** ![swtb](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@swtb](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fswtb)%5C
**Post date:** [April 4, 2024, 9:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffrequent-loss-spiking-in-continue-training-llm%2F55080%2F3 "2024-04-04T09:33:44Z")

</div>

> [@hunggggg](#):
>
> ```auto
> r: 16 #experience with rank = 8 but similar loss pattern
> target_modules: null
> lora_alpha: 32
> lora_dropout: 0.05
> 
> ```

It seems odd to me that you arent targeting any modules with LoRa