Big Data: Data Parallelization with Apache Spark

Apache Spark is software responsible for distributing or parallelizing data during analysis.

Nowadays, there is much talk about parallelizing data across different machines to process information faster, but what does that actually mean? Parallelizing data across different machines simply involves dividing the data into smaller files. Each of these smaller files is sent to a different machine. This way, each machine processes a small part of the initial file instead of analyzing the entire file.

Suppose we have the following file: consumos.csv with 2 columns; the first indicates the time the data was collected, and the second indicates the consumption in kWh. This file contains 10 different moments when data was collected. We create a Spark cluster as shown in the following figure:

One of the machines is responsible for controlling operations and is called the master, while the other two machines communicating with the master are called workers. If we request the sum of kWh from our cluster, it proceeds as follows:

1. It divides the consumos.csv file into 2 files, particion1.csv and particion2.csv, each consisting of 5 rows.
2. It sends the particion1.csv file to one machine in the cluster and the particion2.csv file to the other.
3. Each machine in the cluster calculates the sum of the second column and returns the result to the master.
4. Once the master has received the results from each machine, it sums the two quantities received.
5. Spark returns the total sum of the consumption column.

Now, if we want to find the average consumption, each machine returns the sum as it did before and also returns the number of rows in the file it analyzed. In our example, it would act as follows:

1. The machine processing the particion1.csv file would return a consumption sum of 0.981 and the number of rows analyzed: 5.
2. The machine processing the particion2.csv file would return a consumption sum of 0.864 and the number of rows analyzed: 5.
3. The master machine sums the number of rows sent by each machine, sums the totals received from each machine, and performs the following division:

Let’s look at a graphical representation of how the average consumption would be calculated:

We can therefore conclude the simplicity and speed that Spark provides when facing large-scale problems. Divide and conquer!