# Highlighting important tokens for input into LLM

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhighlighting-important-tokens-for-input-into-llm%2F62698%3E%5C
**Category:** 🤗Transformers\
**Created:** [November 18, 2023, 7:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhighlighting-important-tokens-for-input-into-llm%2F62698 "2023-11-18T19:58:46Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jeffyjeff2893](https://avatars.discourse-cdn.com/v4/letter/j/d9b06d/32.png) [@jeffyjeff2893](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjeffyjeff2893)%5C
**Post date:** [November 18, 2023, 7:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhighlighting-important-tokens-for-input-into-llm%2F62698%2F1 "2023-11-18T19:58:46Z")

</div>

I’m currently trying to create a classifier that classifies code as vulnerable or non vulnerable (0 or 1) but I want the model to align to the tokens that it should consider important. I have a dataset of source code functions where vulnerable functions are collected from patches where the function before the patch is vulnerable, I want the model to consider the tokens changed in the patch as more important, but I don’t want the model to overfit to the presence of these special tokens. I’m using codebert and I want to prepend the important tokens before the actual function like this: `[CLS] important tokens [SEP] function [SEP]` My problem is that the important tokens would only show up in vulnerable functions so I would think the model would overfit to the presence of the tokens, is there any way I can get around this?