Introduction

In the ever-evolving landscape of data analytics, data products have emerged as a pivotal concept, particularly with the advent of methodologies like Data Mesh. Implementing a robust analytics strategy using data products can drive substantial business value and accelerate delivery times. This article outlines our approach using the Data Product Pyramid to translate business problems into iterative analytics, focusing on practical implementation and key learnings from real-world engagements.

Data Product Pyramid

The Data Product Pyramid serves as a comprehensive framework for building data products that directly address business needs. This strategy involves breaking down high-level business goals into specific, actionable data products, each designed to solve a piece of the puzzle.

Understanding Data Products

A data product needs to:

1. Drive P&L directly or at least inform business decisions.
2. Solve a specific business problem.
3. Provide direct value to business customers.
4. Follow a product life-cycle with customer interaction and feedback.

By starting with key business drivers and iteratively developing data products, we create a dynamic and adaptive business strategy that evolves with changing needs.

The Data Product Funnel

The Data Product Funnel is a process that translates business changes into actionable data products. This involves:

1. Business Change Drivers: Initiating changes based on business, environmental, or operational needs.
2. Use-Case Selection and Prioritization: Valuing and prioritizing initiatives.
3. Business and Solution Framing: Ensuring the right questions are asked.
4. Iterative Development: Building and refining data products in a pyramid structure.
5. Deployment: Publishing products on a Data Product Foundation for use across the business.

By following this iterative process, we achieve rapid delivery of valuable analytics use cases that align with business strategies.

Translating Business Problems into Iterative Analytics

Implementing the Data Product Pyramid begins with understanding the business’s needs and translating them into data products through a structured approach. We're using the Data Product Pyramid to establish a next-gen analytics capability for a complex, multi-business firm, working directly with their business and data teams. Our goal? To translate the business’s data and analytics needs into actionable insights that solve current and future problems.

A Collaborative Journey

Working closely with customer-facing and internal business teams (including Sales, Account Management, Operations, Finance, and more), we've been navigating the Data Product Pyramid process together. This collaborative approach has allowed us to distill their myriad requests into small, encapsulated data products and interactive visualizations, delivered quickly and iteratively.

From Insights to Action

By creating a method that bridges the gap between high-level business issues and their underlying causes, we enable teams to perform what-if predictive analysis. This capability transforms their approach from relying on historical metrics to leveraging multi-level, business-focused data products. The result is a comprehensive storytelling and decision-making tool that serves everyone from the CEO to frontline operators.

Key Learnings

In this blog post we will be sharing some fascinating insights and lessons learned from this journey. Here’s a some Key Learnings:

• Prioritize Business Needs First: When developing data and analytics product capabilities (whether it’s a data mesh, data fabric, lakehouse, or data warehouse), always start by understanding what the business wants to achieve. Focus on solving their current and future problems—even if they can't articulate them fully.
• Iterative Development: Engage business teams throughout the process, delivering solutions in small, manageable iterations. This ensures the end product is aligned with their needs and can be adapted as those needs evolve.
• Integrated Storytelling and Decision-Making: Develop tools that not only provide historical data but also support predictive analysis and decision-making. This holistic approach empowers businesses to act on insights rather than just observe them.

This project has truly been a game-changer for delivering end-to-end analytics capability—from customer requests to development to operation. I’m excited to share more about our approach and the incredible outcomes we’ve achieved.

Data Therapy & Capturing What the Business Wants

In this phase, we use a storytelling approach to map high-level business questions to transactional data, creating a hierarchical framework. For these engagements, we use a User Journey-driven (storytelling) approach that begins with high-level business questions, such as “Is my service line profitable?” These questions are then mapped into a journey from high-level business problems down to transactional data, creating a hierarchical framework as we go.

Grouping Analytics Requests

We group the requests into analytics buckets to avoid creating bespoke data products for each individual. Here's how we do it:

• Split into External & Internal Groups: We divide the requests based on whether they are customer-facing or internal.
• Define Personas: We outline personas ranging from operators at the lowest organizational level to the C-Suite.
• Define Service Levels: We establish different levels of capability sold at various prices for customers.

Capturing Requests

We capture requests through workshops with users from different organizational groups. Here's the process:

• Frame the Objective Clearly: It’s crucial to stay focused on the objective, despite the tendency of users to treat these sessions as “therapy” where they vent about all their issues (e.g., “no trust in the data,” “bad data quality,” “old data”). These points are captured for later but we maintain focus on the primary goal.
• Ask Targeted Questions: Using the DPP process, we ask each business user what they want to measure, know, and decide based on their persona, service level, and internal/external grouping.
• Classify Questions & Decisions: We use Data Product types (Decision, Knowledge, Information) to classify questions and decisions into groups, including business analytics model types (e.g., metric, what-if, simulation, predictive, correlation).

This approach helps create data product candidates from a business perspective without delving into technical solutions.

Capturing User Journeys

We also capture tasks such as drill-downs, drill-across, root-cause analysis, and conclusions on new directions as analytics-based user journeys. The end result of this stage is a set of requirements grouped by business, organizational, and analytics types in hierarchical order with user journeys. These will be implemented as pyramids of data products with linkage and dependency graphs between them, which are equally important.

Key Learnings

• Capture All Feedback but Stay Focused: It's essential to gather all feedback but keep the objective in mind to avoid getting sidetracked.
• Use a Clear Classification Framework: An easy-to-understand framework helps guide the business through the process.
• Present Visually & Early: Visual aids and early presentations help clarify the business's mental model and ensure alignment.

Creating Stories through D&A User Journeys

In previous phase, we focused on understanding what the business wants to achieve. In this phase, we map out the stories via data and analytics-driven user journeys.

User Experience Process

We follow a User Experience process with the Data Product Pyramid, but with key differences, such as using actual data and building Data Products as we go.

How Do We Do It?

We start with a set of business scenarios and decisions, walking through the Data Product Pyramid with the users of the personas captured in above. We show how we answer the questions at the appropriate levels of the pyramid.

The diagram below (simplified for illustration) outlines a user journey:

High-Level Management

At the top, we map out key business areas a particular persona wants to manage or measure. This persona has a prioritized list of key aspects needing attention. For example, customer service operations need attention as they’ve hit a threshold, but the issue remains unclear.

First Drill Down

We delve deeper to discover the customer service team is over capacity, and customer resolutions are trending upwards—an unfavorable trend. The reason is still unknown.

Further Analysis

We analyze various aspects consuming the team's capacity, such as types of customer requests and team activity. We find that Team A is spending excessive time on cancellations.

Actionable Insights

Armed with this information, we engage the product and logistics teams to investigate the cancellation issue.

Predictive Modeling

We model capacity forecasts by creating a team forecast capacity simulation as a knowledge product. Each visualization in this journey is a Data Product, typically corresponding to the level of the pyramid (though not always).

Another important element is the Data Product Taxonomy, which aids in traversing the data products. This will be covered in a later post focusing on Data & Tech Architecture.

Key Learnings

• Process Mapping: Often, businesses may not instinctively know their processes, so we map and re-engineer them if necessary.
• Using Actual Data: Utilizing real data helps the business trust the process.
• Art-of-the-Possible: Showcasing potential improvements through models and simulations helps the business think beyond their day-to-day mental models.

This simplified version of the process with basic visualizations gives an idea of how we translate business scenarios into actionable data insights.

Business Navigation

In the previous phase, we explored how to create a single user journey. In this instalment, we bring it all together, creating a map for the entire business.

Mapping the Business Layers

The attached diagram illustrates the layers of this process, providing a way to navigate an entire business using Data Products as "waypoints."

1. Data Fabric Layer

   • Unified Data Exposure: We start from the bottom layer, where data from various sources is unified through a Data Fabric. This infrastructure presents data assets (both internal and external) in a cohesive manner and allows for semantic overlays like graphs, data dictionaries, and other metadata.

2. Domain Mapping Layer

   • Business Context Overlays: The next layer overlays organizational concepts onto the data using metadata in a glossary, which we call the "Domain Map." This includes domains, organizational structures, business terms, etc., making the data comprehensible within a business context.

3. Data Product Creation Layer

   • Data Product Development: Using the Data Product Pyramid process, Data Products are created. Each product is associated with a specific part of the business, ensuring clarity in understanding and ownership.
   • Marketplace for Easy Access: All products and underlying data artifacts (e.g., datasets) are published on a Data Product Marketplace, facilitating easy search, browsing, and in-place execution.

Linking and Navigating Data Products

Once the Data Products are in place, we need a method to link and traverse them, enabling journey creation and storytelling.

1. Describing Business Journeys

   • Business Contextualization: We start by describing the journeys in a business context. This involves beginning with a decision product and drilling down to supporting products. This is achieved using a business taxonomy that describes each step as an effective (but simple) business process.

2. Navigating the Journeys

   • Seamless User Experience: To navigate the journeys, we need to traverse from one Data Product to another seamlessly. For instance, if I am viewing one visualization and want to drill down, I should be able to navigate to the underlying one with a single click.
   • Executable Ontology-Based Graph Processing: For this, we use an executable ontology-based graph processing engine. This engine maintains a graph of Data Product API calls executed at the vertices (joins) with traversable links (edges) to others that correspond to parts of one (or more) journeys.
   • Building Business Narratives: Each link has a semantic description, e.g., "Sales Projection Product" -> "Gets Historical Sales from" -> "Total Monthly Sales Product." This builds a business narrative for each journey.
   • Agile Business Mapping: This approach allows us to map the entire business, not as a one-time, offline exercise, but as an agile, evolving, and incremental process.

Key Takeaways

• Unified Data Infrastructure: Start with a Data Fabric to unify and expose data.
• Business Context Overlays: Use a Domain Map to overlay business concepts onto the data.
• Data Product Marketplace: Create and publish Data Products for easy access and use.
• Business Taxonomy: Use business taxonomy to describe journeys and processes.
• Graph Processing Engine: Implement an ontology-based graph processing engine for seamless navigation.

Conclusion

The Data Product Pyramid framework allows businesses to transform their data into actionable insights, fostering better decision-making and operational efficiency. By starting with business needs, developing data products iteratively, and integrating them into a comprehensive business navigation system, we enable organizations to respond swiftly to changing market conditions and internal demands.

Call to Action

Are you ready to revolutionize your data strategy and turn business challenges into opportunities with the Data Product Pyramid? Reach out to learn how we can help you build a next-gen analytics capability tailored to your organization’s unique needs. Let’s embark on this journey together and unlock the full potential of your data. Contact us today to get started!