Bliss, ExTASY, DRIHMS and FrAGrenSCe: Get your AIMES Right!

This will be a talk in two parts: In the first part, we will motivate via a series of applications the concept of a scalable and general-purpose Pilot-Job. We will discuss the P* Model of Pilot-Jobs and present BigJob a SAGA-based Pilot-Job. In the second part of the talk, we focus on extension of the Pilot-Job concept to the challenge of "Big Data". Science that involves and depends upon large amounts of data, also requires overcoming various challenges, including managing large-scale data distribution and co-placement/scheduling with computing resources. Although there exist multiple approaches to addressing these challenges, an integrative and interoperable approach is missing. To address the fundamental challenges of co-placement and scheduling of data and compute in heterogeneous and distributed environments, we define the concept of Pilot-Data, in analogy with Pilot-Jobs. We present an overview of Pilot-Data and show how it supports multiple production distributed cyberinfrastructure, e.g., XSEDE, Open Science Grid, as well as Clouds. We validate the concept of Pilot-Data by establishing that it provides a simple abstraction for managing data placement, whilst supporting interoperability and late-binding.

Bio:

André Luckow is researcher in the Radical group at Rutgers University.
His current research focuses on the challenges of "Big Data" and the
design of abstractions and systems for supporting data and compute in
distributed, heterogeneous environments. Before joining Radical, André
studied at the University of Potsdam where he obtained a doctorate
degree in the field of distributed computing. His further research
interests include large-scale distributed systems, data-intensive
computing, machine learning and mobile systems.