This talk showcases the use of serverless computing to solve large-scale optimization problems efficiently.

Starting from a real case study, we discuss different cloud-based architectures and we show how cloud functions enable a 96% reduction in elapsed time while keeping the cost low.


Solving large-scale optimization is challenging.

However, some optimization problems can be easily parallelized. In this case, serverless computing can drastically cut the elapsed time, while keeping the cost low.

In this talk, we consider a real case study. While creating power market scenarios, we need to find the optimal way of using power storage facilities.

This results in a convex optimization problem, where the power exchanged between each facility and the grid in each hour is an optimization variable. Over a scenario range of 50 years, this generates hundreds of thousands of variables.

Fortunately, the problem can be split into subproblems which can be solved in parallel.

Nonetheless, a conventional solution, based on a 16-core virtual machine, requires about 2 hours to perform the computations. We show how a serverless approach based on cloud functions is capable of cutting the elapsed time by 96% while keeping the operating costs as low as 3€ per run.

Finally, we show how to connect the parallel optimization module to the rest of the scenario processing pipeline, implemented in AirFlow.

After attending this talk, you will be aware of the capabilities of serverless for high-performance computing, and you will have a better feeling about how to design a cloud architecture.