Marco Capuccini to present at Spark Summit East 2017
20 Dec, 2016
Marco Capuccini will present at Spark Summit East 2017: “EasyMapReduce: Leverage the power Of Spark And Docker To scale scientific tools In MapReduce fashion”
Abstract: High-throughput methods in various scientific fields produced massive datasets in the past decade, and using Big Data frameworks, such as Apache Spark, is a natural choice to enable large-scale analysis. In scientific applications, many tools are highly optimized to resemble, or detect, some phenomenon that occurs in a certain system, and the effort of reimplementing such tools in Spark cannot be sustained by research groups.
Application containers are gaining a tremendous momentum, as they allow to wrap whole software stacks, that can be then easily fired up and teared down on demand, in a matter of seconds. Docker emerges as the most broadly used containerization tool, and it represents the perfect candidate to wrap scientific application stacks.
In Uppsala University (Sweden) we developed EasyMapReduce, a Spark-based utility to run Docker containers in MapReduce fashion, in order to process a large-scale distributed dataset. In this talk we will present the challenges that scientists have to face, in order to run scientific tools over large datasets, and how EasyMapReduce helped us to rapidly implement many use cases in our research group. In addition, we will discuss challenges and future plans for the EasyMapReduce implementation.
Keywords: Spark, Docker, HDFS, Scientific Workflows, Bioinformatics Takeaways: Learn how serial software can be run in parallel, over a distributed dataset, using Spark and Docker GitHub: https://github.com/mcapuccini/EasyMapReduce