Comic courtesy of XKCD, via Creative Commons License

Our frontend engineering candidates go through many of the same types of interviews as our backend candidates, i.e., algo and coding. But because they’ll be building the user-facing parts of our system, we want to make sure candidates have some design chops as well. Hence the UI design interview.

Why have a design interview for an engineer?

Perhaps my favorite part of being on the frontend team at Palantir is that we get to work on both the design and the implementation of our own UIs, from inception to finished product. Engineers collaborate on the design of the product with other engineers as well as with designers, and we openly debate the merits of all our decisions. The UI interview simulates this kind of cooperative design and debate. Typically, this means you’ll be asked to design and/or evaluate one or two pieces of UI during the interview, while your interviewer plays both collaborator and critic.

Read on for some helpful points to keep in mind as you work on a design problem with your interviewer.
Read the rest of this entry »

“You’re asking us to test our platform’s programming language? How am I supposed to do that?”

My head itches from trying to recall the bits and pieces of what I learned in high school about programming, specifically the semantics of a programming language. Sure, I did a bit of programming for homework assignments in college, but I was no CS major. This was a much different challenge for a QA engineer to test. Compared to an application, a programming language is completely open ended; there are no specifications to test, guidelines to follow, or limits to break.

The Hedgehog language had the basic set of tools laid out for me already: I could declare variables, create data structures, and use loops for iteration. As I was trying out individual usage examples, such as how to structure if statements or how to cast an object to a different type, I realized that this was no way to test something as powerful and flexible as an entire language. It would be like a doctor who claims that since each individual organ works fine, there are no problems with the entire system. This is insufficient: one needs to look at the system as a whole, including examining the interactions between each component. I decided I needed to create much larger and elaborate code samples in order to test the Hedgehog language in a larger scope.

Using the Hedgehog language, I had programmed several algorithms, solving puzzles that would output a number. This was getting bit boring, since once the output value was matched the expected number, there was nothing more to be done. I wanted to create something more dynamic, a toy I could play around and experiment with. And opportunity presented itself in the form of one of our newest tools: the spreadsheet application. With the capability of setting the value of each cell programmatically and then coloring them depending on their value… hmm what could I do with this?

Hedgehog is a powerful tool in coding functions and workflows that directly interact with our applications. Most of the time, the language is used to write expressions for an input value, create custom metrics that return values after a set of calculations, or even to set inputs, calculate, and save documents. Given the language’s ability to integrate with Spreadsheet, the capabilities of the Hedgehog language can literally be visually shown to the user, resulting in some stunning displays. Below are three examples I’ve coded in Palantir Finance’s own language: calculating and drawing the Mandelbrot fractal, simulating Conway’s Game of Life, and solving a Nonograms puzzle.
Read the rest of this entry »

We talk a lot about Human-Computer Symbiosis on this blog – it’s a systems design approach that guides us in our construction of our technology stacks. Given that, we’re always on the lookout for example of HCS systems built by other people.

Here’s an unlikely example: the layout of names in the memorial was made according to ‘meaningful adjacencies’ (as described by Jer Thorp in his blog post, All The Names: Algorithmic Design and the 9/11 Memorial):

The project was to design an algorithm for placement of names on the 9/11 memorial in New York City. In architect Michael Arad‘s vision for the memorial, the names were to be laid according to where people were and who they were with when they died – not alphabetical, nor placed in a grid. Inscribed in bronze parapets, almost three thousand names would stream seamlessly around the memorial pools. Underneath this river of names, though, an arrangement would provide a meaningful framework; one which allows the names of family and friends to exist together. Victims would be linked through what Arad terms ‘meaningful adjacencies’ – connections that would reflect friendships, family bonds, and acts of heroism. through these connections, the memorial becomes a permanent embodiment of not only the many individual victims, but also of the relationships that were part of their lives before those tragic events.

Read on for details on the approach they used and how it embodies HCS architecture (not to mention, a video of their tool in action).
Read the rest of this entry »

A few months back, Kevin introduced us to the Hedgehog Programming language – (here’s the post if you missed it).

The Palantir Finance programming language — Hedgehog as we know it — is an interpreted, statically typed, object-oriented language. With a syntax that’s based loosely on Java, it mixes roughly Java-style semantics and a few idiosyncrasies that make it a really interesting case study in language design. It’s built to be extremely efficient for batch operations on time series, which is the heavy lifting in financial analysis.

In this video, Eugene and Dave, two of the engineers that work on the language and platform features needed to support it, give a talk that goes into a number of areas around the Hedgehog language, including why we needed to build a language, how it makes the platform more powerful, how we built dev tools into the UI to make debugging easier, and a bunch of the nitty-gritty features that go into the strange (but fitting) beast that is the Hedgehog Language.

As a final note: this is one of things that I love about working at Palantir Technologies. We study a problem pretty hard before we decide that we need to re-invent the wheel – and then when we do, we go all out. It’s one of the benefits of working with the incredibly talented and motivated folks here. When someone says, “well, we need to build a programming language. No, we’re sure,” we just roll up our sleeves and do it. We can add it to the list of: JMX monitoring system, refined Lucene search engine, speeding up Map-Reduce-like systems to interactive time, and implementing our own GIS platform.

One thing about being a developer on the Palantir Finance product that doesn’t get nearly enough publicity is the fact that we have our own programming language. I’m pretty excited about it so let me repeat, with emphasis: we have our own programming language. Yeah, it’s awesome. All those late hours you spent in the lab working on your final project in compilers: turns out they’re actually good for something other than getting into grad school.

Building this language ourselves — as opposed to, say, using an existing language that already just works — wasn’t an easy decision. In fact, it wasn’t even a single decision. We wracked our collective brain dozens of times trying to think of a better approach. But every which way we sliced it, the problems we needed to solve always pointed to building our own language. I still question this decision sometimes, but on the whole I’m very happy with how things have turned out.

Read the rest of this entry »

This is a response to Ari’s awesome post on human-computer symbiosis. Ari and I were chatting about the equation he developed and I was wondering if there were some further refinements that are possible… let’s take a look:

We are attempting to understand the total analytic capability for a given task a of a human-computer team. Analytic capability in this case probably means:

(1)

Where A is the answer to the analytic problem in question and t_A is the time needed to arrive at the answer based on the inputs available. In the case of chess, A could be the optimum next move given all previous information and t_A would be how long it takes to decide on this move.

Read on for a look at how this generalizes in human-computer symbiotic systems.
Read the rest of this entry »

Software engineering is a craft that blends science and art. This fact is easy to overlook as the artistic aspects are often eclipsed by discussions of the science and technology behind what we do.

This is not one of those times: the art in software engineering is most evident when building compelling visual interfaces, something Palantir knows a thing or two about.

A demo reel is an industry term in the movie business — a short reel that acts as a portfolio when applying for jobs, a highlight reel of the author’s visual career. We’re not in the movie business, we’re in the software business. We do, however, use moving pictures to tell stories, stories backed by data — this is our demo reel: two-and-a-half minutes of data visualization and user interface eye-candy (It has pounding music — you may want to put on headphones or turn down your speakers.):

The movie will take a few seconds to load. It’s 800×600, so expanding to full-screen is suggested. We’ve done our best to create a streamable-yet-good-looking video. The compression artifacts are there, but shouldn’t be too distracting. In a real Palantir client, there are no compression artifacts and everything looks even better than it does here.

The Palantir family of products is much more that just pretty pictures; we have the underlying intelligence infrastructure to make those realtime animations possible and (more importantly) meaningful. That said, we sure do think they’re pretty.

By the way, if you’re interested in the progression of our interfaces, this not the first time we’ve posted eye candy: we posted a set of updated screenshots a little over a year ago; think of this as the next installment in the series.

And yes, it’s really all Java Swing.

In a previous post, Eric W. covered how we analyze polled system health information. Now we’ll look at pushed information, in the form of logging events.

Use Cases & Constraints

We decided on three kinds of questions we wanted to answer:

• What is the health of the deployment?
  • Example: What errors have occurred in the last 24 hours?
• Which parts of the platform are our users engaged with?
  • Example: How much time do users spend in each application?
• How is our server performing over time?
  • Example: What is the average wait on a search query?

The chief constraint was that we build our platform on Log4J. We already use Log4J all over the project, so converting the logging was out of the question. Besides, Log4J provides a guideline for the kind of metadata our events should support, and Log4J makes it easy to record events to a database.

That left us with two problems to solve: how to store structured data with a Log4j message, and how to analyze the collected data.

Analysis is the easy part: just use Palantir! After all, a sequence of logging events has a lot in common with a time series. The rest is explained below.

Read the rest of this entry »

We put up a post last year on the 2008 VAST Grand Challenge. Well, the IEEE VAST Challenge 2009 is over and the awards are in. We had another strong year, scoring two awards:

• Grand Challenge: Analyst’s Tool Choice (Of 48 submissions, only 3 Grand Challenge awards were given)
• Intuitive Traffic Visualization and Video Description of the Analysis Process

Some background on the event: three years ago, the IEEE began an annual conference called VAST (Visual Analytics in Science and Technology). The VAST symposium focuses on the fundamental research contributions and real-world application of visual analytics. As a part of the conference, the VAST Challenge allows teams to compete on delivering analytic solutions against a synthetic real-world dataset.

A selection of choice quotes from the judges:

• An award for “highly usable integrated exploration environment”, “efficient analytic exploration platform” or something along these lines would be appropriate.
• Survey Question: How much novelty do you see in this submission (data processing, visualization, interaction, hypothesis generation or evaluation, overall process, etc.)? Answer: More so than novelty was the extremely efficient solution approach to this challenge, much more so than other solutions.
• The submission shows two things very clearly: One, it shows the analytical process as being a multi-faceted, simultaneous processing of different information that is quite common among analysts. Two, it shows how multiple perspectives can be displayed on a single monitor, enabling the analyst to visualize what his mind is analyzing. Outstanding!

Our submission

And finally, our submission to the Grand Challenge. Here we have our overview video, with a link to the full video below:

For an in-depth look at the data and techniques used to make this a reality, check out our full submission in Finding a Mole: Cyber Counter Intelligence on the Palantir Analysis Blog.

One of the early architectural challenges that we faced in building the Palantir Finance product was coming up with a good design for firing events from data models to their listeners. There are many different concepts in our product such as charts, portfolios, and indices which are all maintained by different developers. Initially, each developer had their own system for firing events when a data model changed. This quickly became a drag on development as tools became more integrated because we had to learn each others’ event methodologies and translate between the different systems.

The solution was to select a single event firing system. We wanted something that was easy-to-use yet powerful enough to express all the changes that might be made to a data model. Java’s Property Change Support (PCS) was a good fit because it can support arbitrary events in a very lightweight fashion.

Read on for details of our implementation…
Read the rest of this entry »