# Fine tune LLMs on PDF Documents

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-llms-on-pdf-documents%2F71374%3E%5C
**Category:** Models\
**Created:** [January 31, 2024, 5:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-llms-on-pdf-documents%2F71374 "2024-01-31T17:59:08Z")\
**Posts on this page:** 1\
**Showing post:** 9

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [May 1, 2024, 1:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-llms-on-pdf-documents%2F71374%2F9 "2024-05-01T13:02:00Z")

</div>

There are 2 options here I’d say:

- either you fine-tune a text-only LLM (like Mistral, LLaMa, etc.) on the OCR text of the PDF along with a text prompt and corresponding targets. Refer to my notebook on fine-tuning Mistral-7B (or any other LLM) on a custom dataset: [Transformers-Tutorials/Mistral/Supervised\_fine\_tuning\_(SFT)\_of\_an\_LLM\_using\_Hugging\_Face\_tooling.ipynb at master · NielsRogge/Transformers-Tutorials · GitHub](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Mistral/Supervised_fine_tuning_(SFT)_of_an_LLM_using_Hugging_Face_tooling.ipynb). It leverages Q-LoRa with the PEFT library. It’s based on the official scripts from the Alignment Handbook: [GitHub - huggingface/alignment-handbook: Robust recipes to align language models with human and AI preferences](https://github.com/huggingface/alignment-handbook). Basically one needs to prepare the dataset in a format that the model expects (by calling `tokenizer.apply_chat_template` on the inputs). What I’d recommend is applying an OCR engine of choice on the PDF documents (such as Tesseract, or closed-source APIs like the ones from Google Cloud or Azure).
- either you fine-tune a vision-language model (like Idefics2, Llava, Llava-NeXT, PaliGemma) to take in the image(s) of the PDF, typically one image per page, + an optional text prompt, and produce a certain desired target. One example could be take in a PDF and produce corresponding JSON, containing desired keys and values from the PDF. Refer to the [example notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/PaliGemma/Fine_tune_PaliGemma_for_image_%3EJSON.ipynb) regarding fine-tuning PaliGemma or the [example notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Idefics2/Fine_tune_Idefics2_for_JSON_extraction_use_cases_(PyTorch_Lightning).ipynb) regarding fine-tuning Idefics2. The benefit of Idefics2 is that it encodes images very efficiently which means you could easily fine-tune it on PDFs which consists of multiple pages.

The latter approach is basically this:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/f/1/f156503a23ff8e0bfc7a3ed36ac85327cb414265.jpeg)

Update: more and more VLMs now support fine-tuning on multi-page documents. Some recent example include [Qwen2-VL](https://huggingface.co/docs/transformers/main/en/model_doc/qwen2_vl#single-media-inference) and [LLaVa-OneVision](https://huggingface.co/docs/transformers/main/en/model_doc/llava_next#multi-image-inference).

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-llms-on-pdf-documents%2F71374)._