# Hyperparameter search

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhyperparameter-search%2F4281%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 10, 2021, 10:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhyperparameter-search%2F4281 "2021-03-10T10:10:19Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![pramod](https://avatars.discourse-cdn.com/v4/letter/p/71c47a/32.png) [@pramod](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpramod)%5C
**Post date:** [March 10, 2021, 10:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhyperparameter-search%2F4281%2F1 "2021-03-10T10:10:19Z")

</div>

Hi  
How can I pass any parameter values in search space such as Number of attention heads, vocab size etc. with Raytune.

I have implemented it with optuna but not getting anything woth Ray Tune. Please help me to get it done i Ray tune as backend in Transformer Trainer API for Hyperparameter serch.

def model\_init(trial):

```
if trial is not None:
    num_attention_heads=trial.suggest_int("num_attention_heads", 1, 2 )
    vocab_size=trial.suggest_int("vocab_size", 1000, 1500)
else:
    num_attention_heads = 6
    vocab_size=1000

config = RobertaConfig(
    vocab_size=vocab_size, #output vocabulary size 
    max_position_embeddings=514, #position embedding 
    num_attention_heads=num_attention_heads, #number of possible attentions 
    num_hidden_layers=3,
    type_vocab_size=1,
    )

#basic model skeleton is RobertA with new configuration
model = RobertaForMaskedLM(config=config)  
print("Model Parameters {}".format(model.num_parameters()))
return model

```

but how i can achieve same with Ray Tune as backeend