How Palantir Foundry Fuels Your Data Platform

Escalating complexity demands more than an “assembly line” approach to data and analytics.

By Akshay Krishnaswamy, Chief Architect, Palantir

Editor’s note: This piece is a follow-on post to How Palantir Foundry Extends Your Data Platform.

Today’s data platform architectures excel at providing an “assembly line” for data and analytics. In this paradigm, information flows from data producers (such as data engineers and scientists) to data consumers (such as operators and business analysts). These data assembly lines can generate helpful read-only insights and are designed to summarize — rather than operationalize — an intricate reality.

For organizations experiencing increasing complexity, however, this linear assembly line can feel slow and restrictive. What is needed instead is a dynamic flow of data, models, and decisions — across systems and teams — that is capable of self-optimizing, without complete reliance on resource-constrained centralized teams.

In our experience, even with customers who have mature data platforms, organizations looking to achieve this type of agility begin to ask two related questions: (1) How do we unlock existing data from our core operational systems and integrate it into our enterprise data platform?, and (2) How do we continuously capture new data on operational decisions back into these systems so the organization can learn and improve?

Having seen these challenges in some of the world’s most critical institutions, we designed Foundry to supercharge your existing data platform architecture in two ways: software-defined data integration and decision orchestration.

Foundry Unlocks the Operational Core with Software-Defined Data Integration

ERPs, CRMs, and other similar systems reside at the heart of today’s large enterprises. Bringing this data into your data platform, however, can be extremely challenging. Questions that seem straightforward (e.g., “how much less does it cost to manufacture Product X vs. Product Y?” or “how long does it usually take for Product Z to arrive from our suppliers?”) can prompt weeks, months, or even years of technical roadblocks and effort decoding rigid, complex data models.

Encountering this dilemma at many of our customers led us to develop a new technology we call software-defined data integration (SDDI). This approach, encoded in Palantir HyperAuto, connects to your existing data stores such as CRM and ERP and automates integration of that data into your data platform.

Key advantages to software-defined data integration include:

• Speed-to-Value. Unlike with manual integrations, SDDI automates integration of ERP/CRM, populating your data platform in hours. Within days, end users can begin to interact with the data and configure visualizations, analytics, and even business-centric workflows.
• Pipeline Generation. SDDI generates complete, efficient pipelines out-of-the-box. It intelligently reads the shape of data and relationships to adapt pipeline generation to the source systems, all while administering frequent data health and data accuracy checks. The finished pipelines ship with source-specific cleaning logic and honor the of settings of the source system.
• Metadata Inference. SDDI learns from the shape of the source data and adapts the dynamic pipeline generation to that shape. Rather than being prescriptive, it seamlessly adapts to business-specific customizations, without expensive additional maintenance or intervention.
• Data Harmonization. Using the same inference methods, SDDI can determine or recommend matches between datasets across different source systems, such as Oracle NetSuite, SAP S/4HANA, or Salesforce Sales Cloud. Users simply confirm or reject SDDI’s recommendations for resolving entities, enabling a simple, scalable way to combine hundreds of disparate sources.
• Integrated Security. SDDI includes an integrated security model (auth-n, auth-z, etc.) across services, purpose-based access controls, row- and column-level access controls, and out-of-the-box integration with security tools from your identity provider.
• Complete Data Engineering Support. SDDI features end-to-end support for all data engineering work, including automating traditionally labor-intensive capabilities such as scheduled ingests, health checks, entity resolutions, pipeline auditing, and data cleansing.
• Customizable Integration. While SDDI ships with encoded knowledge, it also enables customization by providing a framework for users to capture other business logic (via both code and no/low-code interface). SDDI intelligently leverages this logic to continuously improve and deliver pipelines tailored to your organization’s specific needs.

In our view, SDDI achieves a core integration objective: It automates complex, time-consuming tasks and allows your teams to focus on the highest-value things. To learn more about software-defined data integration, check out our series of posts on the origins, capabilities, and impact of Palantir HyperAuto.

Foundry Continuously Fuels Your Data Platform with Decision Orchestration

While integrating data from core systems into your data platform is a necessary step, it does not by itself produce a lasting advantage. Enduring success depends on your organization’s ability to power decisions on the operational front lines — and then capture those decisions back into your data platform to allow for continuous improvement.

As a result, once an enterprise has unlocked data with a technology like SDDI, a new question emerges: How do we dynamically connect our end-user workflows, data platform, and core operational systems to optimize decisions and accelerate learning?

Foundry offers this “closed loop” in the form of decision orchestration. When an end user makes a decision based on an alert, scenario analysis, recommendation, or other analytical insight, Foundry captures all relevant data — including the decision itself, the business impact, possible alternatives, state-of-the-world context, and any motivating trigger such as the alert. It then coordinates the capture of that information back to your operational systems and data platform where it can inform analytics and future decisions.

Foundry’s decision orchestration functionality is grounded in:

• Native data connectors (including 200+ out of the box), enabling safe propagation of results and decisions to operational systems (e.g., ERP, Asset Config) and data platforms.
• The Foundry Actions framework, providing a structured, secure, and auditable mechanism for publishing decisions (and other data) to external systems.
• Open access to your raw and derived data sets, made possible by REST APIs, standard protocols (ODBC/JDBC), direct filesystem access (FoundryFS), and out-of-the-box export services.
• Custom webhooks and write-back procedures, which can be authored directly in Foundry’s applications to support process automation and write-back to external systems.
• Complete lineage from data to decision, enabling the organization to model the state of the world when a particular decision occurred. Platform tools and REST APIs make this readily accessible.

Conclusion

Palantir Foundry fuels your enterprise’s data platform along two dimensions. First, it uses software-defined data integration to populate it from “below” — unlocking complex operational data from ERPs/CRMs and other source systems. Second, it helps fuel it from “above” — continuously orchestrating decisions and context from end users back to your core operational systems and data platform.

These modalities can add value independently or together, leading to remarkable acceleration in outcomes:

• At an oil and gas multinational, the organization used SDDI to integrate 5 legacy ERPs with 300+ datasets into a unified data asset. Within weeks, inventory savings of more than $50M were identified. Decisions related to inventory are orchestrated back the organization’s data lake as well as operational systems.
• At a Fortune 100 consumer goods company, Foundry harmonized more than 7 legacy ERPs to create a functional supply chain workflow within 5 days. With access to this data alongside key operational insights and simulations, the organization identified 1–2% production improvement that could lead to an estimated $100M in Year 1 savings. Decisions on raw material purchases flow back to the organization’s Google data platform as well as its ERP systems.

See Palantir Foundry and your data platform in action

Curious how these technologies play out in practice? Interested to see how Foundry can integrate with your existing architecture? Get in touch with a member of our Foundry product team.