Design and implementation of an open-source platform with Elasticsearch to manage Networks of IoT devices.

This master thesis takes advantage of the continuous growth of the Internet of Things and is supported by two current technological innovations such as search engines and distributed systems. In this way, an open-source platform for the management of peripheral devices based on Elasticsearch is developed. The platform not only integrates essential functionalities to allow communication between users with their devices, but also integrates modules to centralize the maintenance of the platform using the Elastic Stack tools. Hence, the solution proposed in this master thesis stores and manages large volumes of data in real-time, coming from devices, servers and services on the platform, allowing its visualization and analysis.

Traditionally, these types of platforms were centralized and dedicated to specific use cases. However, due to the popularization of the Internet of Things, this model results inefficient to process and store large volumes of data. Therefore, a platform that provides scalability and performance to process data from different sources is presented. In addition, thanks to its distributed and replicated architecture, it guarantees the availability and consistency of the stored data. However, the management and maintenance of distributed architectures require complex mechanisms to detect and anticipate potential errors in its modules. To carry out this work efficiently and easily, this project proposes a solution that centralizes the storage, management and visualization of the systems and services of the platform, using the Elastic stack tools: Elasticsearch, Beats, and Kibana.

In this master thesis a design and implementation of an Elasticsearch infrastructure with nine nodes to store documents, structured and unstructured, and index them in real time is presented. In this way, making queries and searches on the stored information are carried out quickly and efficiently. Subsequently, a communications module between the gateway and the application server through Apache Kafka is presented. It allows the streaming of messages in a secure, reliable, and independent way. Additionally, modules that collect and send information about the metrics, logs, and network traffic of each server and system on the platform through Beats agents are designed and implemented. Finally, the graphical representation of the data stored in Elasticsearch is provided to the users and administrator of the platform. Finally, dashboards to manage devices and monitor and control platform status are provided using Kibana tools.

Master's Thesis:  Master of Science in Telematic Services and Network Engineering, Universidad Politécnica de Madrid

Author: Gonzalo Calderón Lillo

Director: Guillermo del Campo Jiménez
Co-director: Carlos Ángel Iglesias Fernández