# Load tokenizer from vocab file that's been read into python

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-tokenizer-from-vocab-file-thats-been-read-into-python%2F697%3E%5C
**Category:** Beginners\
**Created:** [August 12, 2020, 3:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-tokenizer-from-vocab-file-thats-been-read-into-python%2F697 "2020-08-12T15:51:09Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![MobiusLooper](https://avatars.discourse-cdn.com/v4/letter/m/cc9497/32.png) [@MobiusLooper](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMobiusLooper)%5C
**Post date:** [August 12, 2020, 3:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-tokenizer-from-vocab-file-thats-been-read-into-python%2F697%2F1 "2020-08-12T15:51:09Z")

</div>

Hi there,

I’m trying to instantiate a tokenizer from a vocab file _after_ it’s been read into python. This is because I want to decouple reading objects from disk from model loading, so I want to load files into python in a different way, and then use those python objects to instantiate the hugging face objects. I can do this with actual model itself like this:

```auto
with open('pytorch_model.bin', 'rb') as f:
    buffer = io.BytesIO(f.read())
    
with open('config.json', 'r') as f:
    config = DistilBertConfig.from_dict(json.load(f))
    
torch_model = torch.load(buffer, map_location=torch.device('cpu'))
model_test = DistilBertModel.from_pretrained(pretrained_model_name_or_path=None, state_dict=torch_model, config=config)

```

But I can’t find a way to do it with the tokenizer. Does anyone have an idea how to do this?

Cheers!