Título:
|
Multi-tenant Pub/Sub Processing for Real-Time Data Streams
|
Autor/a:
|
Villalba, Álvaro; Carrera, David
|
Otros autores:
|
Barcelona Supercomputing Center |
Abstract:
|
Devices and sensors generate streams of data across a diversity of locations and protocols. That data usually reaches a central platform that is used to store and process the streams. Processing can be done in real time, with transformations and enrichment happening on-the-fly, but it can also happen after data is stored and organized in repositories. In the former case, stream processing technologies are required to operate on the data; in the latter batch analytics and queries are of common use.
This paper introduces a runtime to dynamically construct data stream processing topologies based on user-supplied code. These dynamic topologies are built on-the-fly using a data subscription model defined by the applications that consume data. Each user-defined processing unit is called a Service Object. Every Service Object consumes input data streams and may produce output streams that others can consume. The subscription-based programing model enables multiple users to deploy their own data-processing services. The runtime does the dynamic forwarding of data and execution of Service Objects from different users. Data streams can originate in real-world devices or they can be the outputs of Service Objects.
The runtime leverages Apache STORM for parallel data processing, that combined with dynamic user-code injection provides multi-tenant stream processing topologies. In this work we describe the runtime, its features and implementation details, as well as we include a performance evaluation of some of its core components. |
Abstract:
|
This work is partially supported by the European Research Council (ERC) un-
der the EU Horizon 2020 programme (GA 639595), the Spanish Ministry of
Economy, Industry and Competitivity (TIN2015-65316-P) and the Generalitat
de Catalunya (2014-SGR-1051). |
Abstract:
|
Peer Reviewed |
Materia(s):
|
-Àrees temàtiques de la UPC::Informàtica -High performance computing -Big Data -Analytics -Stream Processing -Real-time Data Processing -Programming Models -Internet of Things -Supercomputadors |
Derechos:
|
|
Tipo de documento:
|
Artículo - Versión presentada Objeto de conferencia |
Editor:
|
Springer
|
Compartir:
|
|