# How do I construct a function to inference?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-construct-a-function-to-inference%2F23052%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [September 13, 2022, 2:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-construct-a-function-to-inference%2F23052 "2022-09-13T02:00:40Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![psyche](https://avatars.discourse-cdn.com/v4/letter/p/848f3c/32.png) [@psyche](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpsyche)%5C
**Post date:** [September 13, 2022, 2:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-construct-a-function-to-inference%2F23052%2F1 "2022-09-13T02:00:40Z")

</div>

I trained Flax model for the sequence classification well.  
After that, I wrote the code for the inference function for the random single text.  
But It is very slower than the PyTorch, Tensorflow model.  
The code for the inference is below,

```auto
import jax
from transformers import AutoTokenizer, FlaxBertForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("<model_name_or_path>")
model = FlaxBertForSequenceClassification.from_pretrained("<model_name_or_path>", from_pt=True)

def inference(text:str):
    tokenized = tokenizer(text, return_tensors="jax", truncation=True, max_length=512)
    return model(**tokenized)

```

(it takes about 2 sec on timit test but Pytorch, TensorFlow take about 400ms)  
(I also try to make it as jitted, but it also takes 600ms(below))

```auto
def inference(text:str):
    return model(tokenized).logits

tokenized = tokenizer(TEXT, return_tensors="jax", truncation=True, max_length=512)['input_ids']
jitted = jax.jit(inference)
jitted(tokenized).block_until_ready()

```

If there is a way to enhance inference performance, please give me some wisdom…!  
(What I want is the fastest way to inference (not training))
