Palantir Edge AI in Space

Designing software for in-orbit data processing

Editor’s Note: In December 2022, we published an update about our on-orbit imagery processing efforts here.

On April 1, 2022, Palantir and Satellogic launched our first Edge AI-enabled satellite into space as part of the SpaceX Transporter 4 mission. Incorporating Palantir’s Edge AI technology onboard Satellogic’s NewSat will allow us to begin processing imagery data on orbit, separating signal from noise in high-scale data to make the best use of limited bandwidth and enabling more opportunities for customers to gain near real-time data at scale.

The April launch, part of Satellogic’s previously announced plans for 2022, expands its fleet to 22 satellites delivering high resolution data from space. Satellogic aims to launch a total of 34 commercial satellites to orbit by the first quarter of 2023, offering up to seven daily revisits of any point of interest, with the goal of orbiting more than 200 satellites by 2025. This will provide Satellogic with the capacity to remap the entire Earth daily.

The new Edge AI-enabled NewSat is the culmination of a six-month project during which Satellogic adapted its hosted payload program and edge computing hardware to run Palantir’s Edge AI platform. Our joint project delivers new capabilities which will have broad applicability across mission sets in both government and the commercial sector.

Overview

Use Cases

This launch introduces four major technical capabilities that lay a foundation for our larger strategy of Edge AI in space.

• Reduced Latency and Fast Insight Generation: The primary objective of moving our AI capabilities to the edge (onboard the satellite) is to help increase decision-making speed. Running processing on the ground — especially for high-resolution imagery — introduces latencies that Edge AI can eliminate. Edge AI starts processing data upon capture, delivering actionable insights faster than traditional ground processing. For example, we can inform future data collection onboard (closed loop) or selectively downlink pre-processed images based on their analytical value.
• Bandwidth Optimization: Traditionally, satellite sensors can capture higher volumes of data than they can currently send down. Edge AI allows us to make AI-driven decisions about which data will be the most valuable to our customers and to adapt to shifting priorities in near real-time. This means satellites can better utilize their sensor capacities to capture larger amounts of data that are most relevant to our customers for making time-sensitive decisions.
• Lightweight Model Update: A static algorithm on board a satellite can quickly become obsolete as mission requirements change and technologies improve. Enabling continuous algorithm update in-flight makes the asset adaptable to constantly changing priorities and operating environments. This is why our Edge AI-enabled NewSat has built-in mechanisms to update onboard models and software throughout the lifespan of the satellite. (See below for more on our upgrade infrastructure.)
• Third-Party Vendor Hot Swaps: We believe that the most capable Edge AI deployments need to manage and deploy algorithms from multiple vendors. On board the NewSat there are Palantir object and cloud detection models, a Xailient ship detector model, a MindFoundry image segmentation model, and a Pilot.ai multi-class detection model. We are able to hot-swap between models and chain capabilities, depending on mission requirements and terrain. For example, when the satellite passes over a port we can run the cloud detector, image segmentation, and ship detector models, and when passing over a town we can switch to vehicle and building detectors.

Satellite Architecture

Satellogic designs and builds every component of its satellites (i.e., cameras, onboard computers, power subsystem, sensors and actuators, optics, radios, and propulsion systems). Each spacecraft carries both a sub-meter multispectral camera and a 29-band hyperspectral camera, and includes an additional bay for proprietary or third-party payloads. This approach provides a massive cost advantage and enables shorter R&D cycles to scale and iterate more efficiently.

For compute purposes, we optimized our software for NVIDIA’s Jetson Platform which runs within Satellogic’s payload system. Satellogic’s network of ground stations will be used for communication with the Satellite and the onboard Edge AI system.

Fig. 1 Overview of the Satellogic hosted payload bay and edge computing hardware (from Satellogic)

A Unique Deployment Environment

Deploying Edge AI on Satellogic’s NewSat platform introduced a host of new challenges, including:

• Limited Compute and Random Restarts: It’s true: working in space is hard. In satellite operating environments, power is a scarce and finite resource. We carefully ration power between the Edge AI system and the satellite’s imaging and communications hardware. Practically speaking, the Edge AI system has only minutes of runtime on board the Jetson TX-2i. Changes to the satellite’s orbital trajectory and assigned capture tasks can further reduce this runtime, requiring our software to have increased fault tolerance.
• Novel Raw Imagery Input: The raw imagery presents a unique challenge for models. An image consists of four different types of data in a single channel: Near Infrared (NIR), Red, Green, and Blue light (Red, Green, and Blue light are collectively referred to as “RGB”). Most computer vision models for image processing are typically designed for three-channel RGB imagery. Model performance degrades severely when dealing with raw L0 data. Retraining models within most AI architectures remains a challenge due to the scarcity of existing data and the wide variance in identified features across the image.
• Limited uplink bandwidth: Communication with satellites is typically highly asymmetric. Bandwidth constraint means network throughput ranges from GBs for downlink to a few MBs for uplink per pass of the satellite, posing a significant challenge to updating satellite software.

To tackle these unique deployment constraints, we introduced the following innovations to Edge AI:

Image Pre-processing

Drawing on lessons learned from prototyping the unique 4-band, single channel raw image data on board a satellite the previous summer, we invested in a robust pre-processing solution to improve model accuracy without retraining.

We developed a custom image registration service that can take in images from the multispectral camera and efficiently process them into 3-channel RGB imagery. We can intentionally ignore the NIR data since our models don’t require it. In practice, this means a capture of dozens of frames yield a handful of RGB images that we can run our models on.

When the registration process is finished, we have the full 3-channel RGB image to pass to downstream models. This allows us to plug in third-party models trained for standard RGB images, bypassing the intricacies of the satellite sensor.

Fault Tolerance

With this launch, we are introducing a new, highly fault-tolerant design for Palantir’s Edge AI system.

We carefully considered each aspect of the system across the stack, from the underlying hardware to the architecture of the data pipeline, to achieve better reliability guarantees. On the software side we leveraged Edge AI’s modularity — each of our individual services guarantees data validity as data flows through the pipeline, dynamically purging invalid data in the process.

These services are backed by a checkpointing system based in an external storage module attached to the Jetson. Combined with our data integrity logic, this checkpointing system allows Edge AI to recover state and seamlessly resume processing from where it left off in the event of unexpected termination without any user intervention.

On the throughput side, we take full advantage of multicore processing. Palantir’s modular design enables the saturation of Edge AI’s pipelines in mere seconds, dramatically increasing throughput.

AI Integration

Palantir’s Edge AI platform provides a robust framework for orchestrating the multiple AI models on board the satellite. Once pre-processing is complete, we can start running a variety of models against our formed image.

Along with our own in-house models, we are excited to have numerous third-party models on board:

• Palantir Omni: A multi-class computer vision (CV) model capable of identifying land, sea, and air vehicles as well as buildings.
• Palantir Overcast: A highly performant cloud detector.
• Xailient: A fast, lightweight ship detector.
• Pilot.ai: A highly performant, lightweight model for detecting aircraft and ships.
• Mind Foundry: An ultra low-power image segmentation model that identifies terrain types such as forest and ocean.

Our Edge AI platform adapts the input image according to each model’s requirements and stores relevant inferences. Our downstream post-processing services then use this information to generate additional output. Edge AI is capable of sequencing multiple models in parallel. For example, we can run Overcast, MindFoundry, and Omni in sequence. The segmentation model can ignore the clouds identified by Overcast, leading to improvements in segmentation accuracy. Next, these segmentation results can be used by Omni to improve its recall by removing false positives, a major advantage of Edge AI. By allowing models to complement each other with our best-in-class capabilities, we can realize significant improvements in model performance.

Post Processing

Once the models have finished generating their onboard insights, we post-process the outputs to prepare them for downlink.

Projections. All of the models accompanying Edge AI return inferences in pixel space. To locate these inferences in the real world, we operate a projection service downstream of the models that can take in satellite telemetry information and convert inferences from pixel coordinates to geodetic coordinates. This service integrates with global elevation datasets to ensure accurate projections when capturing images over areas with a high variance in elevation. We also account for geometric simplifications of the Earth’s model within the telemetry data that are then rectified when we perform projections. Finally, the service can also correct for satellite telemetry sensor biases over time. These refinements allow us to locate inferences on the Earth more accurately than what raw telemetry would permit.

Thumbnails. For each particular inference of interest, we generate lightweight PNG thumbnails which are significantly smaller in size and faster to downlink than the full image. For example, in an image over water we can downlink only the ships identified by the models, greatly reducing our network throughput requirements.

Once these processes have finished, we are ready to downlink our outputs.

In-Orbit Upgrade Infrastructure

Due to uplink bandwidth constraints, we developed a custom upgrade framework to handle in-orbit updates. Instead of shipping a new version of our software to the satellite, we can compute and ship an optimized package that encodes the differences between the new version and the current version installed on the satellite. Once on board, our upgrade infrastructure can reconstruct the new version of our software before instantiating the Edge AI system. This upgrade capability extends to our models; we can retrain models on the ground and update model weights in orbit as needed, continually improving the onboard AI capabilities.

Security

We have implemented various cryptographic mechanisms to validate the data sent down by the satellite and so that only Palantir-authorized updates can happen on board the satellite. This ensures the integrity of our communications which is especially important since we do not control all the hops in the network. Satellogic mission operations utilize different cryptographic tools to ensure the privacy, integrity, and security of the communications, both for TT&C and data transfers to and from the ground.

What’s next?

As we look to future launches, we are considering new capabilities we could add to our current offering. Here’s a quick overview of what’s next:

• Georegistration: We are committed to achieving sub-10m accuracy for our inferences on board the satellite. Satellite telemetry data can only get us within hundreds of meters of the actual inference’s location due to onboard sensor inaccuracies. We’re excited to announce that Palantir’s new in-house georegistration capabilities will be part of our next Edge AI system launch. Our georegistration software is already processing downlinked satellite images, but we are working on integrating this capability into our in-orbit system through Edge AI’s modular interface to dramatically improve our in-orbit inference accuracy.
• Hardware security: At Palantir, we view security as an ever-moving target. To best protect our edge deployments against constantly evolving threats, we are planning to introduce new hardware-backed security mechanisms on board the next generation of our Edge AI system for satellites. These will complement our existing mechanisms and allow us to offer stronger guarantees around confidentiality and integrity for our Edge AI system.
• Tasking support: We are working closely with Satellogic to create capabilities for more granular control of our Edge AI system through the Palantir MetaConstellation software. Beyond system upgrades, we are looking to dynamically adapt Edge AI’s processing to upcoming mission requirements, taking latency, power budgets, and bandwidth constraints into account to configure models for improved accuracy depending on the expected imagery data. Stay tuned for more on our new tasking functionality.
• ISLs and Upgraded Comms: For future launches, we are investigating more communication options including Inter-Satellite Links (ISL). These will enable low latency autonomous tipping and cueing between our Edge AI-enabled satellites, including cross-constellation tasking, all done at the edge. This is a revolutionary capability that we are laying the foundation for with our current Edge AI work.
• Foundry Deployability to the Edge: We are continuing to invest in a framework that enable seamless integration with our Foundry platform. This will enable users to author logic and models in Foundry, then utilize our Apollo delivery platform to push that model up to a satellite. We currently provide this functionality in ground-based environments but are excited to expand to space. This integration is crucial to us to enable CI/CD of models all the way to the edge.

Conclusion

For years, our customers have used Palantir software to process satellite imagery on the ground. We have continually worked to bring this processing closer to the edge to better support our customers’ mission areas and operations. The Palantir-Satellogic collaboration represents a significant step in expediting delivery and dramatically improving the quality of data sent back to customers. This pioneering application of Edge AI in space aims to serve as the foundation for advanced analytics that can inform timely solutions to address critical global priorities, like combatting climate change, preventing disasters, and improving emergency response.

Authors

Rob Imig, Head of USG Research & Development, Palantir

Mueed Ur Rehman, Senior Software Engineer for Edge, Palantir

This post contains forward-looking statements within the meaning of Section 27A of the Securities Act of 1933, as amended, and Section 21E of the Securities Exchange Act of 1934, as amended. These statements may relate to, but are not limited to, expectations regarding the terms of the partnership and the expected benefits of our software platforms. Forward-looking statements are inherently subject to risks and uncertainties, some of which cannot be predicted or quantified. Forward-looking statements are based on information available at the time those statements are made and were based on current expectations as well as the beliefs and assumptions of management as of that time with respect to future events. These statements are subject to risks and uncertainties, many of which involve factors or circumstances that are beyond Palantir’s control. These risks and uncertainties include Palantir’s ability to meet the unique needs of its customers; the failure of its platforms and solutions to satisfy its customers or perform as desired; the frequency or severity of any software and implementation errors; its platforms’ reliability; and the ability to modify or terminate the partnership. Additional information regarding these and other risks and uncertainties is included in the filings Palantir makes with the Securities and Exchange Commission from time to time. Except as required by law, Palantir does not undertake any obligation to publicly update or revise any forward-looking statement, whether as a result of new information, future developments, or otherwise.