# How to set batchsize of inference

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-set-batchsize-of-inference%2F112443%3E%5C
**Category:** Beginners\
**Created:** [October 17, 2024, 12:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-set-batchsize-of-inference%2F112443 "2024-10-17T00:55:25Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![allenwang37](https://avatars.discourse-cdn.com/v4/letter/a/bbe5ce/32.png) [@allenwang37](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fallenwang37)%5C
**Post date:** [October 17, 2024, 12:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-set-batchsize-of-inference%2F112443%2F1 "2024-10-17T00:55:25Z")

</div>

I am a beginner and it seems that Transformers only supports processing one request at a time. Can we actually increase the parallel capability to process multiple requests in parallel like batchsize=n?

---

<div class="post-metadata">

**Author:** ![mahmutc](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mahmutc/32/64626_2.png) [@mahmutc](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmahmutc)%5C
**Post date:** [October 17, 2024, 10:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-set-batchsize-of-inference%2F112443%2F2 "2024-10-17T10:00:32Z")

</div>

hi @allenwang37  
I don’t know if this answers your question:

> <https://github.com/huggingface/transformers/issues/13199>
>
> I use transformers to train text classification models，for a single text, it can… be inferred normally. The code is as follows
> 
> from transformers import BertTokenizer, TFAlbertForSequenceClassification
> text = 'This is a sentence'
> model\_path ='../albert\_chinese\_tiny'
> tokenizer = BertTokenizer.from\_pretrained(model\_path)
> model = TFAlbertForSequenceClassification.from\_pretrained('../model\_tf/20210818')
> encoding = tokenizer(text, truncation=True, padding=True, max\_length=30, return\_tensors="tf")
> result = model(encoding)
> 
> When I predict more than one text at a time, an error will be reported. The code is as follows
> 
> texts = \['This is a sentence', 'This is another sentence'\]
> encodings = \[\]
> model\_path ='../albert\_chinese\_tiny'
> tokenizer = BertTokenizer.from\_pretrained(model\_path)
> model = TFAlbertForSequenceClassification.from\_pretrained('../model\_tf/20210818')
> for text in texts:
> encoding = tokenizer(text, truncation=True, padding=True, max\_length=30, return\_tensors="tf")
> encodings.append(encoding)
> result = model(np.array(encodings))
> 
> The error information is as follows:
> tensorflow.python.framework.errors\_impl.InvalidArgumentError: Value for attr ‘Tindices’ of string is not in the list of allowed values: int32, int64
> ; NodeDef: {{node ResourceGather}}; Op\<name=ResourceGather; signature=resource:resource, indices:Tindices → output:dtype; attr=batch\_dims:int,default=0; attr=validate\_indices:bool,default=true; attr=dtype:type; attr=Tindices:type,allowed=\[DT\_INT32, DT\_INT64\]; is\_stateful=true\> \[Op:ResourceGather\]

If you have multiple GPUs:

> **[Distributed inference](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.