# Hallucination with trainer.evaluate() on LLMs

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhallucination-with-trainer-evaluate-on-llms%2F48158%3E%5C
**Category:** 🤗Transformers\
**Created:** [July 25, 2023, 11:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhallucination-with-trainer-evaluate-on-llms%2F48158 "2023-07-25T23:44:16Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sadra](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@sadra](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsadra)%5C
**Post date:** [July 25, 2023, 11:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhallucination-with-trainer-evaluate-on-llms%2F48158%2F1 "2023-07-25T23:44:16Z")

</div>

I am trying to use the trainer to do an evaluation with decoder-only LLMs. I want to have custom metrics with the model. this is specifically related to reloading a quantized Lora checkpoint with `do_eval=True` and `do_train=False`.

The issue is model has a lot of hallucinations on predictions, even in the prompt portion. While the same model generates almost perfectly when loaded externally and making predictions using the `model.generate()` method.

This is specifically on llama-2, but the same issue also shows up with Falcon.  
The code is mostly inspired by the run\_clm.py example and the Falcon training script.

Here is the code that I came up with:

```python
peft_config = PeftConfig.from_pretrained(script_args.peft_model)
base_model = AutoModelForCausalLM.from_pretrained(#quantization_config=bnb_config
peft_config.base_model_name_or_path,trust_remote_code=True,torch_dtype=torch.bfloat16, device_map={"": 0},load_in_8bit=True)
lora_model = PeftModel.from_pretrained(base_model, script_args.peft_model).to('cuda')

```

here is the helper functions for metrics

```python
    def compute_metrics(eval_preds):
        preds, labels = eval_preds
        acc_metric = evaluate.load("accuracy")
        acc = acc_metric.compute(predictions=preds, references=labels)

        preds = np.where(preds != -100, preds, tokenizer.pad_token_id)
        decoded_preds = tokenizer.batch_decode(preds)
        print(decoded_preds)
        return {"ACC":acc}

    def preprocess_logits_for_metrics(logits, labels):
        if isinstance(logits, tuple):
            logits = logits[0]
        return logits.argmax(dim=-1)

```

here is the trainer and evaluation:

```python
trainer = Trainer(
      model=lora_model,
      train_dataset=train_dataset,
      eval_dataset=dev_dataset,
      args=training_arguments,
      data_collator=DataCollatorSupervisedFineTuning(tokenizer),
      compute_metrics=compute_metrics,
      preprocess_logits_for_metrics = preprocess_logits_for_metrics,
  )
  if training_arguments.do_eval:
      metrics = trainer.evaluate()
      print(metrics)

```

Here is an example of what happens on decoded\_preds vs. output of using model.generate()

```auto
## Exampleru
##ract the from the file text and the above above above.

```

vs.  
model.generate()  
with default, params expect extending max\_new\_tokens.

```auto
# Instruction
Extract data from the following document using the rules specified below.

```

library versions:  
transformers 4.32.0.dev0  
peft 0.5.0.dev0

---

<div class="post-metadata">

**Author:** ![celsowm](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/celsowm/32/50357_2.png) [@celsowm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcelsowm)%5C
**Post date:** [February 19, 2024, 11:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhallucination-with-trainer-evaluate-on-llms%2F48158%2F2 "2024-02-19T23:54:44Z")

</div>

did you find the solution?