# Repost: Wikipedia (or something else) text to input output
**URL:**
**Author:**  [@Bigfoot302](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBigfoot302)%5C
**Post date:** [November 17, 2024, 4:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frepost-wikipedia-or-something-else-text-to-input-output%2F125321%2F1 "2024-11-17T16:35:26Z")
_ **Repost because my original post is being hide by bots** _:
Hey, I’m new to topics related to LLMs. I’m not sure if this is the right category, but I’ve come across many datasets containing texts from sources like Wikipedia, the web, books, and documentation. However, I’m unsure how to train a chatbot model using text that doesn’t include question-and-answer pairs.
Additionally, I’m curious about how companies like OpenAI and Meta train their models using text from the internet.
One more question (I know I’m asking a lot): how can I speed up training a text classifier model on a dataset with 441k rows?
Apologies for any mistakes; English isn’t my native language (I speak French).
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 18, 2024, 1:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frepost-wikipedia-or-something-else-text-to-input-output%2F125321%2F2 "2024-11-18T01:56:11Z")
That bot thing was a disaster.
If you’re not sure about the category, it’s fine to use ‘Beginners’. I’ve never really paid attention to the category…
I don’t know the details of the LLM training, so I’ll get someone else. @not-lain You’re probably the right person for this question.
---
**Author:**  [@not-lain](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnot-lain)%5C
**Post date:** [November 18, 2024, 12:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frepost-wikipedia-or-something-else-text-to-input-output%2F125321%2F3 "2024-11-18T12:59:47Z")
Hi @Bigfoot302
this blogpost might interest you :
> **[Fine-Tuning Gemma Models in Hugging Face](https://huggingface.co/blog/gemma-peft)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
> Additionally, I’m curious about how companies like OpenAI and Meta train their models using text from the internet.
these companies train their AI models on 2 to 3 steps :
1. pretraining : feeding the AI vast amounts of text data (ie wikipedia, math, poems, facts, …) just to teach the AI how to talk and some common general knowledge
2. instruction tuning : in here we examples of conversations to the AI to teach it how to reply to instructions or conversations, this also helps to teach the AI when to stop talking instead of generating a long paragraph (example when I say `Hi` i expect the AI to say `Hello` instead of creating an wikipedia like reply, a bad example of a reply without finetuning is the following `Hi is a word people use to greet each other ...`
3. preferance optimization and further enhancements
> One more question (I know I’m asking a lot): how can I speed up training a text classifier model on a dataset with 441k rows?
I would say get a big GPU and increase the batch size, else get multiple GPUs and setup multiple GPU parallelism, the second method is a bit complex, so try to research more about it in advance or use the trainer API (already used in the blogpost above) which comes with multiple GPU support by default.
Hope this helps
---
**Author:**  [@Bigfoot302](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBigfoot302)%5C
**Post date:** [November 18, 2024, 5:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frepost-wikipedia-or-something-else-text-to-input-output%2F125321%2F5 "2024-11-18T17:47:55Z")
Thank you for the reply.
> [@not-lain](#):
>
> Pretraining: feeding the AI vast amounts of text data (e.g., Wikipedia, math, poems, facts, etc.) to teach the AI how to communicate and gain some general knowledge.
How can I “feed” a model with text? As far as I know, when training a model, I need `input_ids` and` attention_mask` for the inputs and `labels` for the outputs. However, I don’t see how it’s possible to train a model using just text.
Edit: I see that if I leave the input blank and put the text I want in the output, it works.