# LLM fine-tune with domain specific pdf documents
**URL:**
**Author:**  [@NajiAboo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNajiAboo)%5C
**Post date:** [June 7, 2023, 2:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F1 "2023-06-07T02:25:16Z")
We are looking to fine-tune a LLM model. We have domain specific pdf document. We need to fine-tune a LLM model with these documents and based on this document LLM model has to answer the asked questions.We trained gpt2 model with pdf chunks and it’s not giving answers for the question. We also tried with bloom 3B , which is also not giving as expected.
Any suggestions or support please .
---
**Author:**  [@MichaelKoo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMichaelKoo)%5C
**Post date:** [June 14, 2023, 12:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F2 "2023-06-14T00:05:47Z")
It is dependent on multiple factors.
It is depends on the type of LLM model you are using. Different models have different capabilities such as text generation vs. text classification or name entity recognition. So you need to make sure you are choosing a language model that is able to do what you need. For example if you are want a model that is similar to ChatGPT you need a model that has text2text generation capabilities. Or if you want a question and answering model you need to choose one similar.
Also when using the transformer, on a trained model you need to configure the pipeline to do what you ask. For example configuring a causal language model can be used to answer multiple choice questions only if configured correctly.
Also for training purposes, PDFs as the text going in are not usable because models can only take raw text. So you first need to get rid of useless information such as empty lines, extra spaces, bullet points, and etc.
---
**Author:**  [@NajiAboo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNajiAboo)%5C
**Post date:** [June 14, 2023, 1:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F3 "2023-06-14T01:13:16Z")
Thanks for the reply. This really helps. The end goal of the fine-tuned model is to answer the questions based on the trained documents. As per my understanding, if we go with the Question and Answer type model, we need to have a set of questions in hand. As of now, we have only a limited question-and-answer set. and preparing the complete question and answer will be costly and time-consuming. The current approach we are planning is to train the cleaned document on Bloom, which will give us a base model. Then again fine-tune it on the question and answer set that we have. I believe this will give the model to learn about the domain and the model will be able to answer the question which was not there in the question and answer set.
Do you think this will work? please suggest . Once again thanks for taking time to reply .
---
**Author:**  [@tsan1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftsan1)%5C
**Post date:** [June 22, 2023, 12:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F4 "2023-06-22T00:33:05Z")
This is a very interesting question. Eagerly awaiting some reply.
---
**Author:**  [@Vikram9503](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FVikram9503)%5C
**Post date:** [July 26, 2023, 2:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F5 "2023-07-26T14:43:11Z")
Any help on this I am in the similar situation .
---
**Author:**  [@EquinoxElahin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FEquinoxElahin)%5C
**Post date:** [September 13, 2023, 4:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F6 "2023-09-13T16:18:58Z")
I’m very interested @NajiAboo . Do you have any lead to answer your own question ?
---
**Author:**  [@dbur](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdbur)%5C
**Post date:** [October 29, 2023, 12:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F7 "2023-10-29T00:36:13Z")
Any progress? Does fine tuning add pdf knowledge in to the model? Working any better than prompt based methods?
---
**Author:**  [@imvbhuvan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fimvbhuvan)%5C
**Post date:** [January 31, 2024, 5:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F8 "2024-01-31T17:57:24Z")
We are looking to fine tune Mistral model on the pdfs data we have about 48 documents. We need to train the model using peft. We are unable to find proper resources to fine tune it, and more importantly how do we prepare those documents to fine tune the model, where do we store them, how to supply and do it. Can anyone help us in this regard please ?
---
**Author:**  [@imvbhuvan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fimvbhuvan)%5C
**Post date:** [February 22, 2024, 5:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F9 "2024-02-22T05:08:15Z")
Hii @NajiAboo
Could you please provide the code which you used to train the LLM model on the PDFs document, we want to do the same. We need to train Mistral model on some 50 PDFs but we are unable to find any resources since all the blogs and videos make use of RAG and Vector database, which is not feasible in our case.
Please do help us in this regard, thank you.
---
**Author:**  [@gurusanand](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgurusanand)%5C
**Post date:** [March 2, 2024, 1:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F10 "2024-03-02T13:28:42Z")
Hi @imvbhuvan, @NajiAboo
If you happen to have the code, could you please provide the code which you used to train the LLM model on the PDFs document, we want to do the same. We need to train Mistral model on some 50 PDFs but we are unable to find any precise resources. Thank you
---
**Author:**  [@Jims6367](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJims6367)%5C
**Post date:** [March 12, 2024, 11:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F11 "2024-03-12T11:29:42Z")
Is it possible to perform PEFT finetuning on pdf data since it is unstrcutured?
---
**Author:**  [@br4tp1t](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbr4tp1t)%5C
**Post date:** [March 15, 2024, 7:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F12 "2024-03-15T07:20:53Z")
what i did to train my LLM on our documents, ive used GPT-4 API and wrote python code, to send text from document, and asked GPT to give me 20 questions to each document and the resposne was in json format with INPUT (question + doc text) and OUTPUT as answer. than ive finetuned my model with this data, and its working preety impressive, ive also added vector database where i store new documents and even on documents that are not in LLM is working very well.
---
**Author:**  [@saurabhksa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsaurabhksa)%5C
**Post date:** [April 2, 2024, 9:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F13 "2024-04-02T21:06:48Z")
I just used RAG on mistral and it gives very nice results. No need to fine tune (atleast for me) . And its not just domain specific, i tried mixed pdfs at once and results are great.
Let me know if you like to try out the code. Everything works locally so to avoid sending data externally.
---
**Author:**  [@dijktsjv](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdijktsjv)%5C
**Post date:** [April 3, 2024, 2:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F14 "2024-04-03T14:36:40Z")
I Would like to try the code! sound very interesting
---
**Author:**  [@saurabhksa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsaurabhksa)%5C
**Post date:** [April 4, 2024, 8:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F15 "2024-04-04T08:43:15Z")
You can try this one
> **[GitHub - Saurabh11811/ragchat](https://github.com/Saurabh11811/ragchat)**
>
> Contribute to Saurabh11811/ragchat development by creating an account on GitHub.
just a POC code, you may need to define HF\_TOKEN in env variable if you don’t have model locally available and this will download the necessary models.
---
**Author:**  [@swtb](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fswtb)%5C
**Post date:** [April 9, 2024, 1:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F16 "2024-04-09T13:43:10Z")
RAG + Few Shot Prompting Mistral for example.
---
**Author:**  [@bitschips](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbitschips)%5C
**Post date:** [April 15, 2024, 12:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F17 "2024-04-15T12:54:40Z")
It sounds nice, thanks.
---
**Author:**  [@dsbitcoding](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdsbitcoding)%5C
**Post date:** [April 19, 2024, 6:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F18 "2024-04-19T06:07:39Z")
Hi @imvbhuvan
I noticed your work with training the LLM model on PDF documents, and I’m interested in doing something similar. However, most of the resources I’ve found focus on using RAG and Vector database, which isn’t feasible for my project. Did you manage to find a solution to train the Mistral model on PDFs? If so, could you please share your code or any resources you found helpful? I’d greatly appreciate your assistance in this matter. Thank you in advance for your help!
---
**Author:**  [@SaikiranBondi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSaikiranBondi)%5C
**Post date:** [July 3, 2024, 10:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F19 "2024-07-03T10:09:43Z")
Hi Mr Bhuvan, I am looking for this specific use case of finetuning llama3 or mistral downloaded in local, on my manuals pdfs, can you please guide me if you have found any solution for this.
Best Regards
Saikiran B
---
**Author:**  [@Shubham-1991](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FShubham-1991)%5C
**Post date:** [September 13, 2024, 4:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338%2F20 "2024-09-13T04:49:46Z")
Using Few Shot Prompting +RAG i got the best result , and was able to remove the unwanted answer .I had taken data of General Knowledge questions
[Next page](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-fine-tune-with-domain-specific-pdf-documents%2F42338.md%3Fpage%3D2)