The Business Value of Incorta Direct Data Mapping (DDM)

It can be challenging to understand the business impact of a seemingly esoteric and technical matter like data management. At many companies, the managers have their work, while the “data guys” (or gals) labor in what feels like a totally separate universe. There are some cultural and organizational reasons for this, but the two areas of business life should be a lot closer to each other than the current situation would suggest. This article offers some reasons why looking at the business implications of data management and analytics practices.

How Data Siloes Affect Business Profitability and Cash Flow

IT systems are no longer monolithic. The old days of having a single computer system running a whole company are long gone. Instead, what most businesses have today is a collection of systems, each performing a different function—accounting, operations management, sales management, human resources, and so forth. Each, typically, has its own data repository.

Unconnected, these data sources become isolated silos of information. To get a proper view of the business, managers have to rely on reports from each data silo. Or, they have to engage in a cumbersome data integration process. Industry research suggests that data analysts and scientists invest 70% or more of their projects' time in collecting data and reformatting it into a clean, unified structure before starting the real work of analysis. The need to analyze external data sources like market research and financial rates further compounds the data integration challenge.

To understand why data silos matter in business management, consider the following simplified example. A company purchases raw materials based on sales forecasts. If their silo-constrained Business Intelligence (BI) tool suggests that sales volume will be 100 units, they will buy 100 units worth of raw materials. However, with data silos getting in the way of accurate analysis, the company might under- or over-buy raw materials. They could buy 100 units worth of supplies, but only need to manufacture 95 units. Five units of supplies will remain unused.

Is that a big deal? If your investors care about cash flow, it’s a problem to accidentally parked five units worth of cash in the warehouse, perhaps never to be used. There are opportunity costs of capital and inventory carrying costs. In the case of one of our clients, a major poultry processor, unused raw materials (e.g. chickens), might spoil and be a total write-off. Thus, we see a correlation between high-quality data and optimal cash flow.

Limitations of Data Warehouses & Data Lakes

The data analytics discipline has developed two approaches to dealing with data silos and the obstacles presented by data integration issues: The data warehouse (DW) and the data lake. The data warehouse is basically a large, structured database that ingests copies of business data from multiple sources. Using BI tools, data specialists, and in many cases, regular employees can run reports against the data in the warehouse and provide insights used in business decision making.

The DW is a popular construct, but it has its share of problems. Running a DW means using traditional Extract-Transform-Load (ETL) processes. ETL often breaks down during times of significant change, however. The modeling process takes too long and forces premature design decisions. As inevitable changes in the business and its data occur over time, most data warehouses become out of date. As they lag behind business realities, executives lose out on accurate data on which they can base their decisions. In addition, the overhead of maintaining DW can be a drag on earnings.

The data lake, in contrast to the DW, is unstructured. Keeping up with the metaphors, it does away with the “building” implied by the DW and dumps all the data into a large, fluid pool. A data lake has neither hierarchy nor organization amongst the data it holds. Data exists in raw form, unprocessed, and analyzed.

There are several advantages to the data lake approach

• It removes many of the data integration difficulties from the equation.
• There is no data schema to maintain.

The problem with data lakes comes from that very lack of structure.

• It’s hard to perform fast, interactive data analysis with a data lake.
• And, they usually require a high-level data management skillset that can be hard to find on the labor market.

How Incorta Direct Data Mapping (DDM) Addresses Difficulties with DWs and Data Lakes

Incorta has a solution for the problem of data silos that overcomes the drawbacks inherent in DWs and data lakes. It’s called Direct Data Mapping (DDM). The DDM architecture aggregates complex business data in real-time, eliminating the need for a data warehouse and traditional star schema approach. This way, it is possible to run queries against the data with the same shape in the data source at high speed. There is no need for ETL, reshaping, or aggregating data in the lake before using it for BI.

When users load data into Incorta, they can opt to load all the data at once. This overwrites any data with the same name in Incorta. Or, they can do an incremental load, which only adds new data to Incorta. The choice gives users flexibility and saves time. In many business contexts, even being able to save an hour or two in preparing BI reports can make a difference in critical decisions.

Users can schedule loads, upload data manually, or write a Python script to import and load data from a source they specify.

The impact of Incorta data connectors

Incorta adds to its fast, flexible BI capabilities with over 40 data connectors. For virtually any common data format and source, Incorta has an existing connector. This includes the major (and most of the minor) relational database management systems (RDBMSs) like SAP, Oracle, Microsoft SQL Server, and IBM DB2. The solution also has connectors for data lakes on AWS, query services like Athena and Apache Drill, and file systems such as Box, Google Drive, SharePoint, and so forth.

The Incorta SAP connecter is receiving a particularly enthusiastic reception in the market. Because so many companies use SAP to run their businesses, and because SAP is often a complex ERP solution, the data connector is quite valuable for BI. The SAP data connector removes most of the hurdles to connecting data inside the SAP system to BI.

DDM and data connectors give companies two distinct advantages in the use of BI for business. They enable faster, more accurate data to be used in reporting and data visualization. This leads to decisions being made based on up-to-date, high-quality data. Coming back to the example discussed earlier, there is less of a chance of over-ordering raw materials when executives can base their forecasts on better data. Then, there’s the cost of running the BI operation itself. DDM and connectors help cut down on the time and resources required to perform BI. That’s a cost-saving that goes right to the bottom line.

If you would like to get more details, you can get a free cloud trial, free training courses with certificates, and you can attend a weekly live demo with Incorta experts.