# Finetuning CTRL

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-ctrl%2F3419%3E%5C
**Category:** 🤗Transformers\
**Created:** [January 28, 2021, 5:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-ctrl%2F3419 "2021-01-28T17:13:13Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprajjwal1)%5C
**Post date:** [January 28, 2021, 5:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-ctrl%2F3419%2F1 "2021-01-28T17:13:13Z")

</div>

I want to perform finetuning with CTRL with custom control codes. The control codes are specific tokens (not the broad domains as in the paper, wikipedia, finance etc.). Since the `transformers` officially supports, it would be great if some information can be provided about finetuning it in Pytorch.

- Control codes have to be prepended before the sequence to get high priority. So do we have to just prepend them ? After prepending, do we add a token as a seperating marker ? The `CTRLTokenizer` does not seem to use tokens similar to `[CLS]` `[SEP]` or `</s>` token ? Even after adding them through `add_special_token` method, the tokenizer does not use them. From this [line](https://github.com/salesforce/ctrl/blob/64bcf20d9a25e5dfa345a7deb94e57e589690d3d/training_utils/make_tf_records.py#L68) it seems that `control codes`, `sentence1` and `sentence2` are not separated through any special tokens. Is this correct ?
- Apart from making the `labels` as `[-100]` for context and making `token_type_ids` as 1s for `input_ids[:context_end_position]`, do we have to do anything else ?
- I’m seeing the warning `BPE merge indices are not consecutive. Please ensure that your tokenizer is not corrupted`. Why is this warning being raised ?
- What is the use of `tokenizer.control_codes` exactly ? The values are just `input_ids` of keys.

How can we generate diverse text better ? Greedy decoding produces deterministic. Beam search is repetitive. Top-k top-p can create random generations. Any tips with parameters. Greedy decoding with repetition penalty produces decent results but sometimes they don’t generate anything (empty output).

---

<div class="post-metadata">

**Author:** ![katze](https://avatars.discourse-cdn.com/v4/letter/k/5e9695/32.png) [@katze](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkatze)%5C
**Post date:** [June 10, 2022, 3:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-ctrl%2F3419%2F6 "2022-06-10T15:25:16Z")

</div>

Hi @prajjwal1, did you in the end manage to finetune with new control codes? IF so, did you need to add separators after the first token?