Standardizing data from heterogeneous sources

Discover how our solution for standardizing data from heterogeneous sources simplifies the management and integration of information from different sources, for better quality and a unified view of your data.

The need

Our client, a major passenger and freight transport company, wanted to reduce the time (weeks or even months) required to collect the input data needed for a project.

The client's data teams therefore began designing an intranet portal where internal project managers could in just a few clicks find the data they needed for their projects.

The problem: each department producing potentially reusable data provided this information in a datasheet published in a format of its own, meaning that several hundred different formats existed.

The purpose of the portal was to allow cross-searches of the data sets generated by different departments. Datasheet harmonization was therefore essential to the success of the portal project.

Proposed solution

Establishment of a single format for the MSDS.

Format import: Tale of Data uses the target format to automatically suggest to the user the data transformations required to switch from the current format to the target format.

The client's data team used Tale of Data to produce, for each input datasheet format, lists of the data transformations that would be needed to create an output datasheet.

Automation of the entire process: every day, new MSDSs are deposited by the various departments on the customer's private cloud (Microsoft Azure). Tale of Data retrieves these MSDSs and automatically applies the corresponding transformations (depending on the originating service and the nature of the MSDS).

Once in pivot format, the records are deduplicated, then sent by Tale of Data to the portal (via API), where they are indexed and made available for search.

Earnings

Tens of millions of euros saved thanks to a spectacular reduction in start-up times for new projects.

The portal is now regularly used by project managers to gather the data they need for their projects.

Therate of data reuse has risen sharply, with a significant reduction in the number of datasets purchased from external service providers, as the project manager had no way of knowing that they were already held by the company.

Standardized locations (construction sites, warehouses, depots, etc.) enable precise geospatial searches to be carried out on datasets within the portal.

The risk of failure has fallen sharply, as projects get off the ground faster and with the right input data.