# Caching tokenization

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-tokenization%2F69072%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [January 14, 2024, 12:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-tokenization%2F69072 "2024-01-14T12:16:23Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kason123](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@Kason123](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKason123)%5C
**Post date:** [January 14, 2024, 12:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-tokenization%2F69072%2F1 "2024-01-14T12:16:23Z")

</div>

Hi I tokenize my data as follows but every time I try to run it, the code does the mapping scratch although there is a cached one in the respective folder. Can anyone help to avoid this redundant process?

```auto
tokenizer=AutoTokenizer.from_pretrained(script_args.model_name, cache_dir="hf_cache_dir", local_files_only=True)
def tokenize_function(example):
    return tokenizer(example["text"], truncation=True)

tokenized_datasets = dataset.map(tokenize_function, batched=True, load_from_cache_file=True)

```