Biblio

Any Big Data scenario eventually reaches scalability concerns for several factors, often storage or computing power related. Modern solutions have been proven to be effective in multiple domains and have automated many aspects of the Big Data pipeline. In this paper, we aim to present a solution for deploying event-based automated data processing tools for low code environments that aim to minimize the need for user input and can effectively handle common data processing jobs, as an alternative to distributed solutions which require language specific libraries and code. Our architecture uses a combination of a network exposed service with a cluster of “Data Workers” that handle data processing jobs effectively without requiring manual input from the user. This system proves to be effective at handling most data processing scenarios and allows for easy expandability by following simple patterns when declaring any additional jobs.