# Preprocessing step for fine-tuning language model

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreprocessing-step-for-fine-tuning-language-model%2F4355%3E%5C
**Category:** Beginners\
**Created:** [March 12, 2021, 3:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreprocessing-step-for-fine-tuning-language-model%2F4355 "2021-03-12T15:38:59Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![maxpower](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@maxpower](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmaxpower)%5C
**Post date:** [March 12, 2021, 3:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreprocessing-step-for-fine-tuning-language-model%2F4355%2F1 "2021-03-12T15:38:59Z")

</div>

Hi,

I want to fine-tune the language model of BERT/DistiLBERT (and later add a sequence classification head; the task is a kind of sentiment analysis). I have a mix of tweets, other social media posts, and speeches.

Now, I’m wondering what are necessary preprocessing steps?

I was thinking about removing urls, hashtags, and user mentions. Is this necessary? O shall I replace them with a special token?

Thanks,  
Max

---

<div class="post-metadata">

**Author:** ![neuralpat](https://avatars.discourse-cdn.com/v4/letter/n/35a633/32.png) [@neuralpat](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fneuralpat)%5C
**Post date:** [March 12, 2021, 4:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreprocessing-step-for-fine-tuning-language-model%2F4355%2F2 "2021-03-12T16:31:47Z")

</div>

Since BERT works with a WordPiece tokenizer, I wouldn’t do any of that and see what happens, before you put effort into pre-processing.  
Since your texts aren’t really domain-specific, you may see decent results for your “kind of sentiment analysis” without doing any pre-processing 😉