Designing a Big Data Architecture for Production Data Processing

One of the greatest advances of the last decade has been the creation of tools capable of processing large amounts of data. In the early 2000s, Google began researching methods to process and store data in a distributed manner, leading to a distributed file system (GFS) and a processing method called MapReduce.

From there, and thanks to the innovation of multiple companies that donated their methodologies and code repositories to the community, the tools that today allow us to process enormous and multiple data sources were developed: “Big Data“.

When we talk about Big Data, we refer to massive data that can be generated from individual user devices or actions which, when related to each other, provide relevant information on the behavior of complex systems.

With the arrival of IoT devices, the Internet of Things, it is possible to retrieve data from practically any machine. In the logistics and production field, it is possible to analyze data from the machines that carry out industrial tasks to determine their efficiency, when they are failing, or to optimize their operation.

An efficient design of a Big Data architecture in a production environment would be capable of performing the following functions:

• Real-time data processing to determine the efficiency of industrial processes
• Mixing data generated by batch processes (for example, data from an ERP or CRM) with data generated in real time to perform financial analysis
• Generation of dashboards that update as data is generated and processed
• Incident analysis to generate predictive maintenance plans from data generated by IoT devices
• Process optimization, such as route optimization or resource reallocation with near-immediate response times
• Trend analysis and generation of recommendations based on standard operations

The perfect architecture for Big Data

There is no perfect, standard architecture. Each project, each specific case, will determine the characteristics of the ideal architecture. To reach that ideal architecture, it is necessary to evaluate which and how many devices can offer this data or can become smart through sensors capable of collecting the data required for analysis. Based on this data, depending on the frequency of data generation and the necessary correlations with other processes such as databases with business information, a technological architecture is created that, using Big Data processing tools, can be capable of generating relevant information for business decision-making, improving the efficiency of production processes and anticipating possible incidents that could decrease production capacity.

At LIS Data Solutions, we work with cutting-edge data processing technologies such as Kafka and Spark from the Apache Hadoop suite, as well as dashboard visualization technologies like Qlik Sense, Power BI, and Grafana. These technologies have allowed us to create data processing systems capable of offering our clients relevant information about their production processes, enabling them to analyze and optimize the operation of their business.