Data Demystified: Supply Chain

The last two articles in this series have focused on foundational principles for promoting data as a product, as opposed to simply declaring a product mindset without changing or challenging the status quo.

Now it's time to get practical.  Every product starts out as an idea.  Successful products are a result of applying business principles and fiscal discipline to the challenges of getting the product to market and satisfying ongoing demand.

This involves marketing, sales, and the order to cash processes.  This article will focus on the latter because no matter how effective your marketing strategy and pipeline processes are, if you fail to deliver a quality product in a timely manner, you will fail.

Unfortunately, this is all too often how data teams operating under the IT/PMO paradigm approach developing and delivering data products.

What's always missing, in my experience, is the build out of the data supply chain.

I think of raw business data as a commodity that can be rapidly processed in bulk to create ingredients for a variety of consumable products.  In businesses like packaged foods, these basic products are multi-billion dollar industries because they've perfected mass production techniques.

These manufacturers are the foundation of a much more complex supply chain, but they succeed in simplicity.

Data that's extracted from a variety of sources and minimally enhanced to provide reliable quality, freshness, historicity, and semantic meaning - and delivered to the closest point of utility is a valuable product in and of itself.

When talent and technology are dedicated to achieving those objectives, great things are possible downstream.  However, most data teams are organized to try and cover the entire process within the IT-mindset of cost avoidance and magical thinking about how many things a single human being can do at once.

Not so long ago, I spent 2 months defining and prototyping something I called an ingestion factory to land, profile, and create a raw data vault data from a variety of business systems.  Then I spent 3 weeks ingesting every single data source the client had spent a considerable sum to have another consultancy load and curate into a platform they were incapable of supporting.

The engagement was focused on migrating data platforms to support dashboard reporting, but when the CIO started asking about leveraging the platform's AI/ML capabilities, guess what was already in place?

You guessed it.  Raw data that had structure, was domain-adjacent, and had a foundation of metadata and lineage baked in.  Plus, the architecture and processes to expand its footprint and scope.

What's the value proposition of supply chain thinking when it comes to delivering data products?

I cost the client less than $100K in billable hours to build a foundation of a readily available and trustworthy supply of data ingredients suitable for an extensive range of business use cases in less than two months.

Conversely, my employer made well over $1M in billable hours from this client chasing dashboards that were riddled with defects and a governance program that was little more than talk and fighting resistance at all levels of the organization over two years.

This is an extreme case.  Not all clients are this short-sighted, nor all consultancies this inept.  However, the mindsets that drove these egregious failures are pervasive.  No business leader in their right mind would build a line of business around a product without first understanding the costs of goods sold and optimizing the supply chain to maximize a return on investment.

Yet, for some reason, few exercise the same due diligence when it comes to data.

It's time for a change.  It's time to stop accepting bloated IT budgets for data teams and consultancies with little to no accountability for business outcomes.

Who's got the courage to risk $50K for the chance to create a transformational foundation for your data program?

Or would you rather keep throwing good money after bad to preserve the status quo?