Understanding Data Products

Traditional data management practices often isolate data handling from core business strategies, which can lead to misaligned objectives and underutilized data assets. To combat this, we propose a business-centric approach to Data Products that ensures alignment with the strategic needs of the company and transforms raw data into actionable insights.

Business Alignment

Data Products must be developed with clear business objectives in mind, ensuring they address specific strategic questions and contribute directly to business goals. For instance, a retail company may create a Data Product to analyze customer purchasing patterns to optimize inventory management. This alignment ensures that the Data Product directly supports the company’s goal of reducing stockouts and overstock situations, thereby improving customer satisfaction and reducing costs.

Actionable Insights

Moving beyond simple data reporting, Data Products should provide actionable recommendations that can directly influence decision-making processes. For example, a predictive maintenance Data Product for manufacturing equipment might analyze sensor data to forecast potential failures, allowing maintenance teams to address issues before they cause downtime. This proactive approach not only enhances operational efficiency but also extends the lifespan of equipment.

Targeted Solutions

Each Data Product should be designed to address a distinct business challenge, equipped with specific metrics to measure its impact and effectiveness. A financial services firm might develop a Data Product to identify fraudulent transactions in real-time, employing machine learning algorithms to detect anomalies and flag suspicious activities. By focusing on this specific challenge, the Data Product provides a targeted solution that enhances security and reduces financial losses.

The Data Product Pyramid

The Data Product Pyramid offers a structured approach to develop and deploy Data Products that address complex business challenges incrementally.

Pyramid Structure

This model organizes Data Products across different levels, each layer building upon the insights and data provided by the previous one, thereby enabling a comprehensive analysis of business problems. The pyramid starts with foundational data services, progresses through information products, and culminates in decision products that directly inform strategic actions.

For instance, a company might start with foundational data services that aggregate sales data across various regions (the base layer). The next level could involve information products that analyze this data to identify trends and patterns. The top level would consist of decision products that recommend specific actions, such as increasing marketing efforts in underperforming regions.

Types of Data Products

The pyramid levels are composed of different types of products, each solving a specific piece of the puzzle with a distinct approach and purpose:

• Decision Products: These are models, algorithms, or decision-making tools that either provide actionable recommendations or execute actions based on decisions. For example, a recommendation engine in an e-commerce platform might automatically purchase a product when a user clicks "yes" to a suggestion.
• Knowledge Products: These products provide insights about the business derived from underlying data and information products. They often involve models or algorithms that predict outcomes or explain why something has happened, such as a Single Customer View or classification models.
• Information Products: As the name suggests, these products provide information. They are positioned above transactional data and answer specific, fine-grained questions like "What is my total historical cost?"
• Data Services: Although not Data Products per se, Data Services are critical components that deliver specific data in the required format and operational profile (e.g., SLA, access patterns) for all Data Products. They are tailored for specific use cases and are not designed for general reuse. These services assemble data from transactional systems and other products to create structures usable by specific Data Products. They also include annotations and mark-ups to facilitate reuse, provided they do not compromise the original use case.

Incremental Development

By adopting an incremental approach, organizations can rapidly prototype, test, and refine Data Products, thereby reducing time-to-market and enhancing the ability to adapt to new information or changing business conditions. This method allows for early wins and continuous improvement, making the development process more agile and responsive.

For example, an e-commerce company might initially develop a Data Product to analyze website traffic patterns (initial prototype). Based on user feedback and performance, the product can be refined to include more granular insights, such as the effectiveness of different marketing campaigns, before finally evolving into a comprehensive recommendation engine that personalizes the shopping experience for each visitor.

Feedback Integration

The pyramid allows for continuous feedback from business users at each level, ensuring that Data Products remain relevant and are iteratively improved based on actual business needs and user experiences. Regular feedback loops facilitate a deeper understanding of user requirements and help in fine-tuning the Data Products to better serve their intended purpose.

For instance, a healthcare provider developing a patient management system can use feedback from doctors and nurses to enhance the product’s functionality, ensuring it meets the practical needs of healthcare professionals and ultimately improves patient care.

Implementing the Data Product Strategy

Successful implementation of a Data Product strategy requires a clear process that integrates business needs with data capabilities. This involves a structured approach that ensures each Data Product is aligned with strategic business goals and delivers actionable insights.

Business Change Drivers

The starting point for the process is reacting to business change drivers. These are summarized into three areas:

• Business Driven: This occurs when the business strategy changes due to an aspiration by the executive team or a shift forced by market forces. For example, a new competitive threat or an executive decision to enter a new market.
• Environment Driven: These changes happen when the external environment shifts, necessitating a business response. A recent example is the COVID-19 pandemic, which forced many businesses to adapt to new ways of operating.
• Operationally Driven: Business-as-usual (BAU) changes such as operational breaks, the introduction of new systems, or the emergence of new data. These require adjustments in processes and systems to maintain operational efficiency.

Value Framework

The Value Framework helps organizations prioritize Data Products based on their potential business impact, ensuring resources are allocated to initiatives that offer the highest return on investment. A structured value framework aligns data initiatives with business objectives, facilitating strategic decision-making and resource allocation.

For example, a telecommunications company might use a value framework to prioritize the development of a customer churn prediction model over other potential projects, given the high impact of retaining customers on overall profitability.

Adaptive Implementation

A flexible, iterative development process allows teams to experiment and iterate on Data Products quickly. This agility is crucial for adapting to findings and evolving market conditions. The adaptive implementation process involves rapid prototyping, user testing, and continuous refinement, ensuring that the final product meets the dynamic needs of the business.

For instance, a logistics firm might develop a prototype for optimizing delivery routes based on real-time traffic data. By iterating on the prototype based on driver feedback and real-world performance, the firm can enhance the product’s accuracy and reliability, leading to more efficient operations.

Robust Infrastructure

A solid Data Product Foundation provides the necessary technological and organizational infrastructure to develop, deploy, and manage Data Products efficiently. This foundation should support a diverse range of data sources and analytical tools, facilitating seamless integration and scalability.

For example, a multinational corporation might establish a robust data infrastructure that includes cloud storage solutions, data lakes, and advanced analytics platforms. This infrastructure enables different business units to develop and deploy Data Products that leverage shared resources and insights, fostering collaboration and innovation across the organization.

The Data Product Funnel

The Data Product Funnel is a process used to implement Data Products to address business changes. This structured approach allows for the incremental development of Data Products, ensuring flexibility and continuous improvement.

Process Overview

Implementation Process

So we know what we need to deliver (the pyramid) and we know how it fits into a strategy, but how do we implement this?

We break implementation capability into three parts, all executed as an iterative cycle:

1. Value Framework: Guides the business to select the most valuable business decisions to be made with the right questions to ask.
2. Fast Adaptive Implementation Process: Allows the data and analytics teams to quickly experiment, develop, and refine the decisions/use-cases into Data Products.
3. Flexible and Robust Data Product Foundation: Provides tools and software to develop, deploy, and operate Data Products, supporting both the other two pillars. This typically includes technical concepts from approaches such as Data Mesh and Data Fabric.

Value Framework

The Value Framework enables the business strategy to be incrementally defined based on actual value and links the business vision and aspirations to the deliverables. It breaks down into:

• Value Framework: A way for the business to agree on how value is measured and applied to analytics business outcomes.
• Business Value Map: As initiatives are defined incrementally, this process maps the actual agreed value of each initiative for prioritization and focus.
• Business and Solution Framing: Each initiative is decomposed into business use-case candidates that get triaged to ensure the right questions are being asked in the right way. For example, an Operations head might want to reduce costs by reducing headcount, but lower-hanging fruit might be reducing infrastructure costs.
• Use-Case Selection and Prioritization: As each use-case is created and framed, feasibility is assessed along with other prioritization tasks to allow the winners to fill an implementation funnel to be delivered as Data Products.

Fast Adaptive Implementation Process

This iterative way of quickly creating and refining Data Products for each decision/use-case breaks down into:

• Use Case Implementation Process: An iterative development methodology to quickly decompose the use-case into hierarchies of data products and data services understood by the business.
• Data Product Design Framework: A library of Data Product archetypes, templates, and patterns (composition, definitions, code artifacts, etc.) that solve different pieces of the use-case and can be created quickly with little to no boilerplate code.
• Data Product Portfolio/Marketplace: A business-friendly searchable marketplace where Data Products get deployed and run. Each Data Product has a full product life-cycle – created, used, and retired.
• Low Friction Federated Data Governance & Management: Enables each Data Product to implement its policies and rules defined globally (for the organization) and locally for the decision/use-case.

Data Product Foundation

This is the data infrastructure and development capability for building and deploying Data Product software and data artifacts. It breaks down into:

• Cross-Business Data Product Infrastructure: Allows Data Products to be deployed in a controlled and low-friction way by all teams in each business. It supports multiple technologies, data, and analytics patterns – there is no single stack.
• Rapid Data Product Innovation Capability: Enables fast prototyping and experimentation of new data, technology, and products in secure environmental sandboxes.
• Incremental Creation of Business & Data Function Artifacts: Shared data artifacts such as business dictionary, policies, domain maps, ontologies, etc., are built and refined incrementally and iteratively as each Data Product pyramid is built.

It is important to note the highly experimental and iterative nature of the early parts of the process. Data Products can be created in very rough form to garner feedback from a business initiative and get more mature as they are tested and refined.

Finally, once they have shown to be fit for purpose, they can be tagged as "production" ready and published as such on the infrastructure.

Data Products can have long life-cycles (e.g., quarterly reports) or short-lived ones (e.g., ad-hoc research for an imminent business event). They can all be created and retired easily as the business changes.

Also, it is important that any type of Data Product is supported. Data Products that need high operational profiles (e.g., real-time recommendation engines) can be published on the Data Foundation alongside those with low profiles (e.g., ad-hoc market research reports). Products can be real-time/batch or have any other type of operational profile, as the business does not distinguish between them.

Domain Ownership

For this approach to work in large, complex organizations, the process and supporting foundation needs to support Data Products being owned, built, and run by teams in different areas of the business (referred to as Domains).

In this example, the data team member charged with the top-level use-case only builds the top-level decision product. They liaise with four other domain teams to build each of the different products and services to form the pyramid.

It is important that the work for each product is carried out by the team that owns that part of the requirement. The Data Product should be delivered directly by the team that owns the business process that delivers that insight or data.

Key Outcomes

If we follow this process, we achieve the following outcomes:

• An Adaptive Business Strategy: The business strategy can start small and evolve and adapt as the business changes.
• Rapid Delivery of Highly Valuable Use-Cases: Use-cases are based directly on the business strategy, ensuring high relevance and impact.
• A Portfolio of Real-World Products: Develop a portfolio of actual, specific Data Products that can be used and reused across the business, fostering efficiency and innovation

Translating Business Requirements into Data Products

The following diagram illustrates a structured approach to translating business requirements into actionable Data Products. It outlines the key questions that need to be addressed at each stage of the decision-making process and links them to the corresponding types of Data Products.

This process involves several steps:

1. Start with What the Business is Trying to Achieve: Begin by clearly defining the business goals and the key decisions that need to be made.
   
   
2. Identify Key Information Needs: Determine the critical information required to make informed decisions.
   
   
3. Integrate Information: Develop methods to compile and analyze the necessary data.
   
   
4. Predictive Analysis: Use historical data and predictive models to forecast future outcomes.
   
   
5. Historical Metrics: Collect and analyze historical data to understand past performance.
   
   
6. Data Sourcing: Identify and gather the data needed to support the analysis and decision-making process.

Example: Reducing Operational Costs

Consider a scenario in which a company aims to reduce its operational costs. The strategy might involve:

Defining Key Metrics

Determining which aspects of operations contribute most to costs and identifying potential areas for savings. Key metrics might include energy consumption, labor costs, and equipment maintenance expenses. By establishing clear metrics, the company can focus its efforts on the most impactful areas.

Developing Targeted Data Products

Creating specific tools that predict future costs under various scenarios and recommend actions to reduce costs. For instance, a Data Product might use machine learning algorithms to forecast energy usage based on historical data and recommend optimal scheduling of high-energy activities to minimize costs.

Iterative Improvement

Continuously refining these tools based on real-world performance and changing operational conditions to ensure they remain effective and relevant. This iterative process involves regular reviews and updates to the Data Products, ensuring they adapt to new data and evolving business needs.

For example, a manufacturing company might initially develop a Data Product to monitor and analyze energy usage patterns. Based on the insights gained, the product can be refined to include predictive capabilities and automated recommendations for energy-saving actions, ultimately leading to significant cost reductions.

Detailed Implementation with a Data Product Pyramid

To solidify this concept, let's work through a detailed example. Imagine the head of Operations has set a goal to reduce operational costs across all business areas.

1. Setting the Decision: The primary decision is to lower operational costs. This decision would be agreed upon as a priority using the Value Framework.

2. Key Questions: We need to identify what information is necessary to make this decision, which leads to questions like:

   • What is my target cost reduction? This question helps define the overall measure of success for the initiative.
   • Where will my highest costs be? Identifying target areas for cost reduction.

3. Drilling Down: These questions further break down into more specific inquiries, such as:

   • Future costs correlated to business needs.
   • Historical and future costs by business line.
   • Historical and future transaction volume by business line (a simplified model for this example).

4. Data Sources: The answers to these questions derive from various data sources:

   • Lowest level costs from HR, operations, and other systems.
   • Staff rosters from scheduling and timesheet systems.
   • Transaction data from OLTP systems.

Each of these questions is addressed by a pyramid of Data Products and Data Services, building on each layer to provide comprehensive answers.

• Top Level (Decision Product): 
  • A staff optimization engine that creates new rosters based on optimal events and costs calculated at lower levels.
• Second Level (Knowledge Products):
  • First product forecasts future costs.
  • Second product correlates future business costs to future business events.
• Third Level (Information Product):
  • A metric providing historical costs by business line.
  • Two Data Services:
    • One for historical transactions.
    • One for staff roster details.
• Base Level (Data Services):
  • Various Data Services that compile data from all transactional systems across the business lines, feeding into the upper layers of the pyramid.

This example demonstrates how each layer of the Data Product Pyramid builds upon the previous one to answer specific business questions, ultimately supporting the top-level decision to reduce operational costs.

Conclusion

This advanced approach to Data Products positions data at the heart of business strategy, enabling organizations to leverage their data assets not just for insight, but for actionable intelligence that drives competitive advantage. By aligning Data Products with business outcomes, companies can transform data from a passive asset into a dynamic tool for strategic decision-making.

Implementing a Data Product strategy involves a structured approach that integrates business needs with data capabilities, facilitates rapid development and iteration, and ensures continuous feedback from business users. Through this iterative and flexible methodology, organizations can develop a robust portfolio of Data Products that drive tangible business value and foster a culture of innovation.