Hyperband

class olympus.hpo.hyperband.Hyperband(fidelity: olympus.hpo.fidelity.Fidelity, space: sspace.space.Space, seed: int = 0, **kwargs)

Bases: olympus.hpo.optimizer.HyperParameterOptimizer

Hyperband works by removing successively removing half of the worst trials periodically until only a few remains, by doing so it does not waste resources training badly performing configurations and it favors configurations that train quickly.

This can cause issue if the best configurations are a slow learners and quick learners start to plateau.

Parameters:

fidelity: Fidelity

used to generate fidelity budget. Fidelity.min can be used to create a grace period during which no trials are removed from the optimization. This will shift all the fidelity by the grace period up to the max fidelity.

Notes

The performance of hyperband is dependent on when the configurations are killed. If it happens too soon it might remove good configuration that had a slower start. To mitigate this issue you can specify a grace period using Fidelity.min. While increasing the grace period will improve performance it will also increase the total number of epoch to run.

The red paths highlight the configurations that have survived up to the last round. The gray ones are the paths that have been killed early.

Work schedule of Hyperband with 10 workers with fidelity=Fidelity(1, 30, base=2)

Visualization of Hyperband space exploration Promotion have been kept to highlight how hyperband picks configuration.

space = {
    'a': 'uniform(0, 1)',
    'b': 'uniform(0, 1)',
    'c': 'uniform(0, 1)',
    'lr': 'uniform(0, 1)'
}

References

[1]	Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar, “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization”

Attributes:

budget

Methods

compute_budgets(reduction_factor)	Compute the budgets used for each execution of hyperband
info()	Return information about the algo configuration
insert_manual_sample([sample, fidelity_override])	Can be used to force a specific configuration to be considered
is_done()	Return true if the optimization is finished
new_result(identity, result)	Event sent when a new result is received
new_trials(trials)	Event sent when a new configurations are sampled
observe(identity, collections.OrderedDict], …)	Observe the result of a given trial
remaining()	Return the number of remaining trials, this is used to kill superfluous worker if possible
result()	Return the configuration with the smallest objective
sample([count])	Sample new configurations and register them
suggest(**variables)	Return configuration to run

from_dict
load_state_dict
max_trials
promote
state_dict

budget

static compute_budgets(reduction_factor)

Compute the budgets used for each execution of hyperband

static from_dict()

info()

Return information about the algo configuration

Returns:

Dictionary, the content is dependant on the algorithm

unique_samples: int

Number of unique configuration that was sampled

total_epochs: int

Total number of epochs performed

parallelism: int

Average number of trial alive at the same time

is_done()

Return true if the optimization is finished

load_state_dict(state)

max_trials()

new_trials(trials)

Event sent when a new configurations are sampled

promote()

remaining()

Return the number of remaining trials, this is used to kill superfluous worker if possible

state_dict(compressed=True)

suggest(**variables)

Return configuration to run