Coding a User Defined Island#
While pagmo provides a number of UDIs (see List of islands) to provide access to a number of parallelization technologies, the expert user
can code his own expanding pygmo functionalities. In this tutorial we will show how to code a UDI. Remember that UDIs are classes that can be used
to construct a
island which, in turn, is what a
archipelago can use to distribute / parallelize optimization
tasks and have solutions migrate improving the overall optimization quality via the generalized island model.
We encourage the user to read the documentation of the class
island to have a detailed list of methods that can be, or have to be,
implemented in a UDI. Also the tutorial Use of the class island is a good starting point to understand the overall use.
>>> import pygmo as pg >>> class my_isl: ... def run_evolve(self, algo, pop): ... new_pop = algo.evolve(pop) ... return algo, new_pop ... def get_name(self): ... return "It's my island!"
We have also included above the optional method
get_name(self) that will be used by various
__repr__(self) to provide human readable information
on some pygmo classes. The above UDI can then be used to construct a
island (similarly to how UDP can be used to construct
>>> isl = pg.island(algo = pg.de(100), prob = pg.ackley(5), udi = my_isl(), size = 20) >>> print(isl) Island name: It's my island! C++ class name: ... Status: idle Algorithm: DE: Differential Evolution Problem: Ackley Function Replacement policy: Fair replace Selection policy: Select best Population size: 20 Champion decision vector: [... Champion fitness: [...
That was easy! Lets now understand what we actually did. The object
isl now contains our UDI and, upon construction will open a thread and delegate the execution of
run_evolve(self, algo, pop) to it upon call to
run_evolve() method must return the algorithm object used for
the evolution and the evolved population.
But there is a catch. We are in CPython! CPython, generally speaking, serialises the execution of Python code due to the presence
of the global interpreter lock (GIL). So, while the code above is perfectly fine and will
work with pygmo, a set of
my_isl running evolutions will not run in parallel as each
island, when executing its
method, acquires the GIL and holds it during the
As a consequence, the following code:
>>> archi = pg.archipelago(n = 5, algo = pg.de(100), prob = pg.rosenbrock(10), pop_size = 20, udi = my_isl()) >>> archi.evolve()
will not run evolution in parallel (only using different threads).
To code properly an UDI one need to code the
def run_evolve(self, algo, pop) so that the GIL is released during the offload of the evolution task to a separate process.
An example on how this can be achieved using, for example the multiprocessing module of python. Let us have a look at some code snippets from the
>>> def _evolve_func(algo, pop): # doctest : +SKIP ... new_pop = algo.evolve(pop) ... return algo, new_pop >>> class mp_island(object): # doctest : +SKIP ... def __init__(self): ... # Init the process pool, if necessary. ... mp_island.init_pool() ... ... def run_evolve(self, algo, pop): ... with mp_island._pool_lock: ... res = mp_island._pool.apply_async(_evolve_func, (algo, pop)) ... return res.get()
The full details are here not reported and can be read in the
mp_island code. In a nutshell, what happens is that the
algo.evolve(pop) gets offloaded to
a process (in a shared pool inited upon construction calling the
init_pool() static method). The instruction
res.get(), makes the thread where
remain waiting for the process execution and while doing so it releases the GIL, making parallelization effective.