# How to avert 'loading checkpoint shards'?
**URL:**
**Author:**  [@EugeneRene](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FEugeneRene)%5C
**Post date:** [August 6, 2023, 1:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-avert-loading-checkpoint-shards%2F49686%2F1 "2023-08-06T13:08:18Z")
Hello, I have downloaded the model to my local computer in hopes of it would help me avoid the dreadfully slow loading process. Sadly it didn’t work as intend with the demo code. Is hat possible, and if so how can I adapt the code to do it?
```auto
from transformers import T5Tokenizer, T5ForConditionalGeneration
import torch
torch.cuda.set_per_process_memory_fraction(1.0)
tokenizer = T5Tokenizer.from_pretrained("LOCAL_PATH")
model = T5ForConditionalGeneration.from_pretrained("LOCAL_PATH", device_map="auto")
input_text = "INPUT"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")
outputs = model.generate(input_ids)
print(tokenizer. Decode(outputs[0]))
```
---
**Author:**  [@jtlicardo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjtlicardo)%5C
**Post date:** [January 20, 2024, 7:40pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-avert-loading-checkpoint-shards%2F49686%2F3 "2024-01-20T19:40:24Z")
Split your code into two parts. Load the model once in a Jupyter notebook cell, and run the generation in a separate cell. This way, you load the model only once, speeding up the process.
First cell (run once):
```python
from transformers import T5Tokenizer, T5ForConditionalGeneration
import torch
torch.cuda.set_per_process_memory_fraction(1.0)
tokenizer = T5Tokenizer.from_pretrained("LOCAL_PATH")
model = T5ForConditionalGeneration.from_pretrained("LOCAL_PATH", device_map="auto")
```
Second cell (run as needed):
```python
input_text = "Your input text"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")
outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))
```
---
**Author:**  [@tiago-machado](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftiago-machado)%5C
**Post date:** [May 14, 2024, 6:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-avert-loading-checkpoint-shards%2F49686%2F4 "2024-05-14T18:59:38Z")
Is there a way to do it without using notebooks?
---
**Author:**  [@pesonen](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpesonen)%5C
**Post date:** [August 21, 2024, 9:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-avert-loading-checkpoint-shards%2F49686%2F5 "2024-08-21T09:25:58Z")
You can load the model into a local server, Flask for example.
---
**Author:**  [@friedahuang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffriedahuang)%5C
**Post date:** [November 1, 2024, 4:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-avert-loading-checkpoint-shards%2F49686%2F6 "2024-11-01T04:01:55Z")
You can use FastAPI.
This is the code that works for me:
```auto
from contextlib import asynccontextmanager
from fastapi import FastAPI
service = Service()
app = FastAPI()
@asynccontextmanager
async def lifespan(app: FastAPI):
# Note: this will only be called once
service.load_model()
@app.get("/")
async def root():
return {"message": "ColPali Search API", "docs": "/docs", "health": "/health"}
@app.post("/search")
async def search(query: str):
response = service.search(query=query)
return {"response": response}
```
---
**Author:**  [@johnmikelregida](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjohnmikelregida)%5C
**Post date:** [October 17, 2025, 10:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-avert-loading-checkpoint-shards%2F49686%2F7 "2025-10-17T10:40:23Z")
This worked for me