# Translate from one tokenizer to another

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftranslate-from-one-tokenizer-to-another%2F81811%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [April 15, 2024, 9:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftranslate-from-one-tokenizer-to-another%2F81811 "2024-04-15T09:03:38Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![broadwayj](https://avatars.discourse-cdn.com/v4/letter/b/9dc877/32.png) [@broadwayj](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbroadwayj)%5C
**Post date:** [April 15, 2024, 9:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftranslate-from-one-tokenizer-to-another%2F81811%2F1 "2024-04-15T09:03:39Z")

</div>

I got tokens encoded with one tokenizer which I want to yield to the LM with another tokenizer. problem seems natural so may be there exist some convenient and unified solution to do so? Here’s what I use rn

```auto
def translate_to_other_tokenizer(ids, tokenizer_from, tokenizer_to):
    text = tokenizer_from.batch_decode(
        ids,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=True
    )

    output = tokenizer_to(
        text,
        truncation=False,
        padding=True,
        return_attention_mask=True,
        return_special_tokens_mask=True,
        return_tensors='pt',
    )

    return text, output

```