In this section
Using web scraping and RPA tools, medication prices are extracted from websites. This data is combined with information from open data sources, feeding a data warehouse that contains time series of prices and their evolution over time.
Objectives
- Extract medication prices from different sources
- Generate a data warehouse with time series of medication prices
- Create a price lookup tool
Medication and medical supply prices within the European Union are heavily regulated, with the governments of each country, or even each region, setting prices according to socioeconomic conditions and their classification as generic, sole-source, or non-subsidized. The data published by countries is usually in formats that are difficult to process, mainly PDF.
What we do
We generate a powerful dataset to enable advanced studies on the data in later phases. Therefore, in this first phase, we focus on obtaining and building the data warehouse, connecting to data published by public and private organizations, using RPA (Robotic Process Automation) and web scraping technologies in some cases to extract information from almost any document or website.
How we do it
Catalog and classification.
The first phase involves designing the data model that will contain the catalog of references, countries, and classification.
This highlights the importance of the information that will be available, as well as the volume of data that can be extracted.
Extraction.
We design and develop web scrapers and robotic price extraction processes for each of the identified sources, as well as for each country’s health portals.
We implement a data warehouse and data model to store all the identified time series.
Storage and visualization.
We adapt the data to the model through ETL processes, completing the process with storage in the implemented data repository.
We develop and deploy dashboards, reports, and KPI indicators that make it easy to query and explore the information.