ribs.schedulers.BanditScheduler

class ribs.schedulers.BanditScheduler(archive: ArchiveBase, emitter_pool: Sequence[EmitterBase], result_archive: ArchiveBase | None = None, *, num_active: int | integer, reselect: 'terminated' | 'all' = 'terminated', zeta: float | floating = 0.05, add_mode: 'batch' | 'single' = 'batch')[source]

Schedules emitters with a bandit algorithm.

This implementation is based on Cully 2021.

Note

This class follows the similar ask-tell framework as Scheduler, and enforces similar constraints in the arguments and methods. Please refer to the documentation of Scheduler for more details.

Note

The main difference between BanditScheduler and Scheduler is that, unlike Scheduler, DQD emitters are not supported by BanditScheduler.

To initialize this class, first create an archive and a list of emitters for the QD algorithm. The BanditScheduler will schedule the emitters using the Upper Confidence Bound - 1 algorithm (UCB1). Every time ask() is called, the emitters are sorted based on the potential reward function from UCB1. Then, the top num_active emitters are used for ask-tell.

Warning

If constructing many emitters at once, do not pass something like [EmitterClass(...)] * 5, as this creates a list with the same instance of EmitterClass in each position. Instead, use [EmitterClass(...) for _ in range 5], which creates 5 unique instances of EmitterClass.

Parameters:
archive: ArchiveBase

An archive object, e.g., GridArchive.

emitter_pool: Sequence[EmitterBase]

A pool of emitters to select from, e.g. ribs.emitters.GaussianEmitter. On the first iteration, the first num_active emitters from the emitter_pool will be activated.

result_archive: ArchiveBase | None = None

An additional archive where all solutions are added. For example, in CMA-MAE, this archive stores all the best-performing solutions, since the main archive does not store all the best-performing solutions.

num_active: int | integer

The number of active emitters at a time. Active emitters are used when calling ask-tell.

reselect: 'terminated' | 'all' = 'terminated'

Indicates how emitters are reselected from the pool. The default is “terminated”, where only terminated/restarted emitters are deactivated and reselected (but they might be selected again). Alternatively, use “all” to reselect all active emitters every iteration.

zeta: float | floating = 0.05

Hyperparamter of UCB1 that balances the trade-off between the accuracy and the uncertainty of the emitters. Increasing this parameter will emphasize the uncertainty of the emitters. Refer to the original paper for more information.

add_mode: 'batch' | 'single' = 'batch'

Indicates how solutions should be added to the archive. The default is “batch”, which adds all solutions with one call to add(). Alternatively, use “single” to add the solutions one at a time with add_single(). “single” mode is included since implementing batch addition on an archive is sometimes non-trivial. We highly recommend “batch” mode since it is significantly faster.

Raises:

  • TypeError – The emitter_pool argument was not a list of emitters.

  • ValueError – Number of active emitters is less than one.

  • ValueError – Fewer emitters in the pool than the number of active emitters.

  • ValueError – The emitters passed in do not have the same solution dimensions.

  • ValueError – The same emitter instance was passed in multiple times. Each emitter should be a unique instance (see the warning above).

  • ValueError – Invalid value for add_mode.

  • ValueError – The result_archive and archive are the same object (result_archive should not be passed in this case).

Methods

ask()

Generates a batch of solutions by calling ask on all active emitters.

ask_dqd()

This method is not supported for this scheduler and throws an error.

tell(objective, measures, **fields)

Returns info for solutions from ask().

tell_dqd(objective, measures, jacobian, **fields)

This method is not supported for this scheduler and throws an error.

Attributes

active

Boolean array indicating which emitters in the emitter_pool are currently active.

archive

Archive for storing solutions found in this scheduler.

emitter_pool

The pool of emitters available in the scheduler.

result_archive

An additional archive for storing solutions found in this scheduler.
ask() → ndarray[source]

Generates a batch of solutions by calling ask on all active emitters.

The emitters used by ask are determined by the UCB1 algorithm. Briefly, emitters that have never been selected before are prioritized, then emitters are sorted in descending order based on the accurary of their past prediction.

Note

The order of the solutions returned from this method is important, so do not rearrange them.

Returns:

A (batch_size, dim) array of solutions to evaluate. Each row contains a single solution.

Raises:

RuntimeError – This method was called immediately after calling an ask method.

ask_dqd() → None[source]

This method is not supported for this scheduler and throws an error.

Raises:

NotImplementedError – This method is not supported by this scheduler.

tell(objective: ArrayLike | None, measures: ArrayLike, **fields: ArrayLike | None) → None[source]

Returns info for solutions from ask().

The emitters are the same with those used in the last call to ask().

Note

The objective and measures arrays must be in the same order as the solutions created by ask(); i.e. objective[i] and measures[i] should be the objective and measures for solution[i].

Parameters:
objective: ArrayLike | None

(batch_size,) array where each entry contains the objective function evaluation of a solution. This can also be None to indicate there is no objective, which would be the case in diversity optimization problems.

measures: ArrayLike

(batch_size, measure_dim) array where each row contains a solution’s coordinates in measure space.

**fields: ArrayLike | None

Additional data for each solution. Each argument should be an array with batch_size as the first dimension.

Raises:
tell_dqd(objective: ArrayLike | None, measures: ArrayLike, jacobian: ArrayLike, **fields: ArrayLike | None) → None[source]

This method is not supported for this scheduler and throws an error.

Raises:

NotImplementedError – This method is not supported by this scheduler.

property active : ndarray

Boolean array indicating which emitters in the emitter_pool are currently active.

property archive : ArchiveBase

Archive for storing solutions found in this scheduler.

property emitter_pool : Sequence[EmitterBase]

The pool of emitters available in the scheduler.

property result_archive : ArchiveBase

An additional archive for storing solutions found in this scheduler.

If result_archive was not passed to the constructor, this property is the same as archive.