# Inference Toolkit - custom inference with multiple models

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-toolkit-custom-inference-with-multiple-models%2F73622%3E%5C
**Category:** Amazon SageMaker\
**Created:** [February 17, 2024, 9:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-toolkit-custom-inference-with-multiple-models%2F73622 "2024-02-17T21:44:19Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![nilpferd](https://avatars.discourse-cdn.com/v4/letter/n/db5fbb/32.png) [@nilpferd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnilpferd)%5C
**Post date:** [February 17, 2024, 9:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-toolkit-custom-inference-with-multiple-models%2F73622%2F1 "2024-02-17T21:44:20Z")

</div>

Hello,

I’m trying to perform custom inference, where I need to a model and a tokenizer hosted at 2 different repositories on HuggingFace. I have looked at [Sample customer inference notebook](https://github.com/philschmid/sample-custom-inference-sagemaker-huggingface/tree/master), however it only uses a single model.

Similarly, the [code summarization notebook](https://github.com/marshmellow77/text-summarisation-project/blob/main/4a_model_testing_deployed.ipynb) also uses a tokenizer and model from the same directory. I want to implement something similar, but with model and tokenizer hosted at 2 different HuggingFace repositories.

If I am to load a model and a tokenizer hosted at 2 different HuggingFace repos, zip them to a tar.gz file and finally push them to s3, what should the directory structure of this tar.gz be? I have tried the following with no success:

```auto
model.tar.gz
    /model1
        model_config.json (along with other model files)
       /code
           inference.py
    /tokenizer
        tokenizer_config.json

```

my custom model loading function looks like:

```auto
def model_fn(model_dir):
    model = AutoModel.from_pretrained(f"{model_dir}/model1")
    tokenizer= T5Tokenizer.from_pretrained(f'{model_dir}/tokenizer')
    return model, tokenizer

```

And my model creation function looks like:

```auto
from sagemaker.huggingface.model import HuggingFaceModel

huggingface_model = HuggingFaceModel(entry_point='inference.py',
                                        source_dir='code',
                                        model_data=s3_location,
                                        role=role,
                                        pytorch_version='1.7.1',
                                        py_version='py36',
                                        transformers_version='4.6.1'
                                    )

```

The error I’m seeing is:

OSError: file /.sagemaker/mms/models/model/config.json not found.

I’d appreciate some help figuring out if my directory structure is incorrect, and how should it be. If there are better ways to achieve this task, please suggest.

---

<div class="post-metadata">

**Author:** ![akshat-kumar-akight](https://avatars.discourse-cdn.com/v4/letter/a/858c86/32.png) [@akshat-kumar-akight](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fakshat-kumar-akight)%5C
**Post date:** [April 4, 2024, 2:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-toolkit-custom-inference-with-multiple-models%2F73622%2F2 "2024-04-04T14:52:02Z")

</div>

as per my understanding your “code” folder should be outside where it is currently. Also, you need to download a model snapshot, place “code” folder in it, place other files that your tokenizer might require and .tar.gz them.