# Multi Objective Hyperparameter Optimization
**URL:**
**Author:**  [@josephleyva](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjosephleyva)%5C
**Post date:** [March 6, 2025, 6:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmulti-objective-hyperparameter-optimization%2F144500%2F1 "2025-03-06T18:19:12Z")
Hello, I was looking at the documentation of the hyperparameter\_search() method : [Documentation Link](https://huggingface.co/docs/transformers/v4.49.0/en/main_classes/trainer#transformers.Trainer.hyperparameter_search) from the Trainer API and I came up with a question.
This is related to multi objective optimization:
- **direction** (`str` or `List[str]`, _optional_, defaults to `"minimize"`) — If it’s single objective optimization, direction is `str`, can be `"minimize"` or `"maximize"`, you should pick `"minimize"` when optimizing the validation loss, `"maximize"` when optimizing one or several metrics. If it’s multi objectives optimization, direction is `List[str]`, can be List of `"minimize"` and `"maximize"`, you should pick `"minimize"` when optimizing the validation loss, `"maximize"` when optimizing one or several metrics.
Can anyone show me an example of this use case?
Like for example, what if I want to maximize the accuracy and f1 score, but also minimize the eval loss?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [March 6, 2025, 10:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmulti-objective-hyperparameter-optimization%2F144500%2F2 "2025-03-06T22:37:39Z")
Hmmm…
* * *
To answer your question about using the `direction` parameter in `hyperparameter_search()` for multi-objective optimization, let’s break it down using the context you provided and the documentation.
* * *
### **Example Use Case: Multi-Objective Optimization**
Suppose you want to **maximize the accuracy** and **maximize the F1 score** , while simultaneously **minimizing the evaluation loss**. This is a multi-objective optimization problem because you have three conflicting objectives: two to maximize and one to minimize.
#### **Steps to Implement This Use Case**
1. **Define the Metrics** :
Create a `compute_metrics` function that returns the metrics you want to optimize (e.g., accuracy, F1 score, and evaluation loss). The order of the metrics returned in `compute_metrics` must match the order of the `direction` list in `hyperparameter_search()`.
2. **Set Up the Trainer** :
Initialize the `Trainer` with your model, datasets, and the `compute_metrics` function.
3. **Define the Hyperparameter Search Space** :
Specify the hyperparameters you want to optimize (e.g., learning rate, weight decay).
4. **Perform Hyperparameter Search** :
Use the `hyperparameter_search()` method with the `direction` parameter set to `["maximize", "maximize", "minimize"]` to reflect your optimization goals: maximize accuracy, maximize F1 score, and minimize evaluation loss.
* * *
### **Key Points to Note**
- The `direction` parameter must be a list of strings (`["maximize", "maximize", "minimize"]`) to specify the optimization direction for each objective in the order they are returned by `compute_metrics`.
- If you do not provide a custom `compute_objective` function, the default behavior is to sum the metric values, which might not always align with your optimization goals. Therefore, you may need to define a custom `compute_objective` function to handle multi-objective optimization properly.
- The `backend` parameter determines the optimization framework (e.g., “optuna”, “ray”, “sigopt”). Each backend may have its own requirements for the hyperparameter space, so refer to the documentation for the backend you are using [1].
* * *
### **References**
For additional details, refer to the following sources:
- [1] Documentation on `hyperparameter_search()` and its parameters: [Trainer](https://huggingface.co/docs/transformers/v4.49.0/en/main_classes/trainer#transformers.Trainer.hyperparameter_search)
- [2] Examples of hyperparameter search implementations: [Trainer](https://huggingface.co/docs/transformers/v4.49.0/en/main_classes/trainer#transformers.Trainer.hyperparameter_search)
---
**Author:**  [@josephleyva](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjosephleyva)%5C
**Post date:** [March 6, 2025, 11:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmulti-objective-hyperparameter-optimization%2F144500%2F3 "2025-03-06T23:47:32Z")
I noticed that the `eval_loss` is calculated internally by the Trainer and returned on the dictionary of `metrics`.
Let’s say I have the following objective
```python
def objective(metrics):
# could be any computation
return metrics["eval_accuracy"] + metrics["eval_f1"]
```
The Trainer will add that _objective_ as an extra metric. If I want to maximize it, but also minimize the loss, how should the hyperparameter\_search knows the order of metrics when the `objective` appears in order to put the `direction` parameter correctly?
```python
best_run = trainer.hyperparameter_search(
hp_space=hp_space,
compute_objective=objective,
direction= ...
backend="optuna", # or "ray", "sigopt", etc.
n_trials=32,
)
```
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [March 7, 2025, 2:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmulti-objective-hyperparameter-optimization%2F144500%2F4 "2025-03-07T02:11:14Z")
by [HuggingChat](https://huggingface.co/chat/)
* * *
To handle multi-objective optimization in Optuna where you want to maximize one metric and minimize another, follow these steps:
1. **Modify the Objective Function** : Update the objective function to return a tuple containing both metrics. This allows Optuna to treat each metric separately with its own direction.
2. **Set Directions in hyperparameter\_search** : Specify the directions for each objective in the `direction` parameter as a list.
By structuring your objective function to return separate metrics and specifying the respective directions, Optuna will correctly optimize each objective according to its direction.
**Answer** : The solution involves modifying the objective function to return both metrics as separate elements and specifying the directions as [“maximize”, “minimize”]. This allows Optuna to handle each metric with its respective optimization direction.
### Answer
To handle multi-objective optimization in Optuna where you want to maximize one metric and minimize another, you should modify your objective function to return a tuple of the metrics you want to optimize. Then, you specify the direction for each metric as a list in the `direction` parameter of the `hyperparameter_search` method.
For example, if you want to maximize the combined accuracy and F1 score while minimizing the loss, you should structure your objective function to return both metrics, and set the direction as `["maximize", "minimize"]`. Here’s how you can implement it:
```python
def objective(metrics):
# Combine accuracy and F1 score (to maximize)
combined_metric = metrics["eval_accuracy"] + metrics["eval_f1"]
# Loss (to minimize)
loss = metrics["loss"]
return (combined_metric, loss)
best_run = trainer.hyperparameter_search(
hp_space=hp_space,
compute_objective=objective,
direction=["maximize", "minimize"], # [1]
backend="optuna",
n_trials=32,
)
```
This setup ensures that Optuna optimizes the combined metric in the “maximize” direction and the loss in the “minimize” direction[1].
[1]