Smart Resource Management for Data Streaming using an Online Bin-packing Strategy

← Back to publications

Published: 2020-11-15

Formatted citation

Stein O, Blamey B, Karlsson J, Sabirsh A, Spjuth O, Hellander A, and Toor S.. Smart Resource Management for Data Streaming using an Online Bin-packing Strategy.
2020 IEEE International Conference on Big Data. , 2207-2216. (2020). DOI: 10.1109/BigData50022.2020.9378241

Abstract

Data stream processing frameworks provide reliable and efficient mechanisms for executing complex workflows over large datasets. A common challenge for the majority of currently available streaming frameworks is efficient utilization of resources. Most frameworks use static or semi-static settings for resource utilization that work well for established use cases but lead to marginal improvements for unseen scenarios. Another pressing issue is the efficient processing of large individual objects such as images and matrices typical for scientific datasets. HarmonicIO has proven to be a good solution for streams of relatively large individual objects, as demonstrated in a benchmark comparison with the Apache Spark and Kafka streaming frameworks. We here present an extension of the HarmonicIO framework based on the online bin-packing algorithm. The main focus is to compare different strategies adapted in streaming frameworks for efficient resource utilization. Based on a real world use case from large-scale microscopy pipelines, we compare two different strategies of auto-scaling implemented in the HarmonicIO and Spark Streaming frameworks.