This is a response to Ari’s awesome post on human-computer symbiosis. Ari and I were chatting about the equation he developed and I was wondering if there were some further refinements that are possible… let’s take a look:

We are attempting to understand the total analytic capability for a given task a of a human-computer team. Analytic capability in this case probably means:

(1)

Where A is the answer to the analytic problem in question and t_A is the time needed to arrive at the answer based on the inputs available. In the case of chess, A could be the optimum next move given all previous information and t_A would be how long it takes to decide on this move.

Read on for a look at how this generalizes in human-computer symbiotic systems.
Read the rest of this entry »

Here at Palantir, a lot of our automatic tests are full-chain tests. A backend server is fired up, client code runs against it, and everything runs much like a production environment. This makes intuitive sense because it’s a faithful approximation of how the system will run in the field.

However, there are some disadvantages to this:

• Full-pass tests don’t always localize the problem. Tests on a client class might fail even if it was the service that behaved incorrectly.
• These full-pass tests are relatively slow. Client code is running against an actual remote service. If a client is being tested, the server code still has to do work — sometimes a lot of work — even if that isn’t the focus of the test.
• The constraints of the test are loose. Full-chain tests can mostly only see whether the operation finished correctly. It’s much harder to figure out whether the operation was done efficiently and without making unnecessary service calls.
• They’re very little setup flexibility. If you want an RPC to return a specific value, you have little choice but to have your test get the service into a state where it can return that value. This is easy in some cases, but prohibitively difficult in others.
• Client tests are forced to share any non-determinism leaked from the service. For example, under real conditions, a request to call A might respond before call B, and sometimes the other way around. This can result in flaky tests or tests that don’t always simulate the conditions you want to exercise.

What’s to be done? Fortunately, there’s an option that handles these cases elegantly. We also test with jMock, a library that dynamically generates mock objects from arbitrary interfaces. These mock objects can be configured to check that particular methods are called with particular inputs a particular number of times, and then give prescribed responses.

Hit the link to see a concrete example of jMock in action.
Read the rest of this entry »

I used to think I understood MVC. In undergraduate CS programs, MVC is taught as an off-the-shelf pattern, explained once and then ready for use in the real world. Wikipedia also makes it seem pretty simple:

Model–View–Controller (MVC) is an architectural pattern used in software engineering. Successful use of the pattern isolates business logic from user interface considerations, resulting in an application where it is easier to modify either the visual appearance of the application or the underlying business rules without affecting the other. In MVC, the model represents the information (the data) of the application; the view corresponds to elements of the user interface such as text, checkbox items, and so forth; and the controller manages the communication of data and the business rules used to manipulate the data to and from the model.

They go on to show the classic triangle diagram and how it’s baked into various GUI and web frameworks. There’s only one clause in the entire article that hints at something deeper: “Though MVC comes in different flavors…”

Different flavors indeed. In fact MVC is not just a pattern but a whole family of patterns: MVC, MVA, MVP, PAC, Model-Delegate…. It very quickly gets very hairy.

In this article I want to describe one of MVC’s lesser-known variants, the Model-View-Adapter (MVA) pattern, and talk about its advantages over traditional MVC in the context of a Java Swing application.

Read the rest of this entry »

Carl juggles with his creation

One of the things our customers love to do is print our beautiful object graphs and tape them to the wall for discussion. What they hate to do is print 30 pages, line them up, and tape them to a poster one at a time. So we bought a plotter, and I started plotting.

I needed to print directly to a Java Graphics object. Unfortunately, the available information on large output printing from Java is thin at best. While there are lots of ways to successfully place ink on paper, I was only able to find one that reliably lets the application pick odd paper sizes that plotters use, like 24×19.7 inches. (The term “plotter” used to mean something with pens for printing blueprints and such. Now it just means a large format printer, commonly printers that can use roll paper as a source.)

One of the first things you’ll learn when you start working with printing in Java is that a language intended to be all things to all people (i.e., cross-platform) is utterly lousy at tasks highly specific to a given environment, such as printing. It will not surprise you to hear that native print services on Windows are pretty different from those available on a Mac, which themselves are pretty different from the CUPS system common to Unix systems.

So, by and large, you are reduced to the least common denominator of printing. Part and parcel of this least common denominator is agreeing on what constitutes a piece of paper and sticking to it. This is fine for people thinking, “My paper is 8.5 inches wide by 11 inches tall.” It poses a bit of a problem for people with plotters who are thinking, “My paper is 24 inches wide by as many damned inches tall as I need.” Even relatively powerful programs like PhotoShop or GIMP don’t seem to support plotters well. I believe Photoshop works by specifying the exact paper size you want to use, but any technique in which the easiest solution for the user is to pull out a calculator does not meet with my approval.
Read the rest of this entry »

Following the discovery that our offices were the birthplace of Java (or least the place where it had its childhood), I invited James Gosling to come visit. For those that don’t know who James Gosling is, he’s more-or-less the father of Java. Java started as a project of James Gosling’s in 1991; today, 17 years later, he’s still at Sun, in charge of guiding the Java platform into the future.

How does one invite such a luminary to come visit one’s offices? One guesses what his email address is and sends him an email out of the blue:

James,

My name is Ari Gordon-Schlosberg, an engineer at Palantir Technologies. I recently became interested in the storied history of our current facilities at 100 Hamilton Ave. in Palo Alto. As Java programmers, our engineering team is really excited to be working in the same place that gave the world Java.

You may not have heard of Palantir, but we’re working on some pretty interesting problems, using Java to build large-scale analysis applications that really push forward the state-of-the-art. We’ve won some accolades for our use of Swing by Romain Guy. If you felt like dropping by the next time you’re in the valley, we’d love to have you come by, see your old digs, and take a peek at what we’re working on.

Sincerely,

Ari Gordon-Schlosberg

To quote the Microsoft Program Manager’s book of proverbs: 90% of making things happen is sending email.

So James dropped by one Thursday for demos, lunch, and schmoozing with our engineers.

The first order of business was to demo our software to James. We got a bunch of the senior engineers together and showed him an abbreviated demo of both Palantir Government and Palantir Finance. We focused less on the problem-space aspects of the software and more on how we’re using Java to build the application. We went over how both of our apps are completely written in Java and that our GUIs are built with custom Swing components.

The most memorable part of the conversation went something like this:

LEAD DEV: So… what do you think of our applications?

GOSLING: It makes me want to weep.

LEAD DEV: Uh… ?

GOSLING: Yeah, we’ve been working on this infrastructure for years to be able to build applications like this and finally someone is doing it.

The rest of the visit was spent talking about Java, its history and its future. Topics ranged from why it’s hard to get dinosaurs like cable companies and mobile carriers to use modern technology to some of the complication in building an optimizing JIT compiler.

After lunch, I walked him to the elevator to see him off. We said our goodbyes and he stepped into the elevator, which was already occupied by the mailman making his rounds. As the doors closed, I hear the mailman say to James:

“Well, I haven’t seen you around here in a while.”

What is wrong with this class and why? I’ll tell you beforehand there are two things I am looking for and they are both in the compareTo function. Yes, this came from the Palantir code base and caused me some issues. It has been modified slightly for illustrative purposes.

class GenericDataSource extends DataSource implements Comparable<datasource> {
    public int compareTo(DataSource o) {
        if(o != null){
            return getName().compareTo(o.getName());
        }
        return 1;
    }
    @Override
    public boolean equals(Object o) {
        if(o == null) return false;
        if(o instanceof DataSource)
            return this.getLocator().equals(((DataSource)d).getLocator());
        return false;
    }
}

Read the rest of this entry »

It seems like a relatively common mistake is to assume that the java.text.SimpleDateFormat class is thread-safe (at least for methods such as format(), which you might not expect to mutate state!). This is not true; SimpleDateFormat is not thread-safe, and format() does mutate state. From the javadoc:

Date formats are not synchronized. It is recommended to create separate format instances for each thread. If multiple threads access a format concurrently, it must be synchronized externally.

Suggestions on handling synchronization:

• Use the Joda Time libraries if possible. They are thread safe. Joda Time is fairly easy to port to from existing JDK compatible code.
• Synchronize calls to format() manually. You probably want to write a wrapper class/function to do this, and make sure nobody calls the original format() method by accident. This seems like it should be safe, but see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=422833.
• Use thread-local storage, e.g.

private final static ThreadLocal<SimpleDateFormat> shortTimeFormat =
     new ThreadLocal<SimpleDateFormat>() {
            protected SimpleDateFormat initialValue() {
                 return new SimpleDateFormat("HH:mm");
             }
      };

N.B. In general, you should be very careful when using ThreadLocal storage, especially when using thread pools (your data won’t be automatically garbage collected and may be visible to other Threads—a security risk) or when storing references to Thread objects in ThreadLocal storage (which might confuse the GC). However, in this case, we should be okay.

.

There comes a time in every developer’s life when they need to write code that processes some XML. Lately, we’ve seen the proliferation of APIs that make XML processing easier, like JAXB (Java API for XML Binding). However, when speed and scale are required, chances are you’re going to need to roll your own processor. Before I continue, let me clear up some terminology, when I say “processor”, I mean the code of yours that’s wrapped around a SAX (tutorial), DOM (tutorial), or an XPP (tutorial) parser, not the guts of the parser itself.

At the end of the day, that’s the interesting part of what you’re doing – the grammar of your data model rather than the minutiae of start and end tags. Building a processor is the interface between the data interchange format and the internal data model of your application.

Click through for a tour of XML parsers and a look at a novel technique for encoding processors that use pull parsers (as usual, we’ve included a WebStart demo, as well as a jar file containing the compiled example along with all of its source code).
Read the rest of this entry »

Every so often (can’t be more than once every two or three days), Swing doesn’t quite do what we need, and we end up writing customized code. In this case, all the available AlphaComposite instances provided with Java were variations on the theme of combining the colors and alpha channel of both source images into a target image. (Wikipedia’s Alpha Compositing article is good background on the topic).

What if what you really wanted was the color from one image and the alpha channel from another? You’d be out of luck, but for the talents of Brien. Here’s what you normally get with a standard AlphaComposite.SRC_OVER sort of technique. In the following two examples, the icon is opaque and the rectangle is partially opaque black fading to transparency.

What we needed looks more like this:

Read on to find out how we did it, and why. Read the rest of this entry »

Everyone agrees that it’s crucial to do validation on user input so that, among other things, your application never tries to write a value that’s too long into a database field with a specific limit. Users of your application shouldn’t, however, be left guessing whether the megabyte they pasted (and you know they will) into the eensy-teensy text field really got saved to the database or not. So you should limit the text field itself so they get immediate feedback, rather than via some Johnnie-come-lately error message, or worse, a bunch of text gets dropped in the bit bucket.

One fairly well established technique is to write a DocumentFilter, and when insertString() or replace() is called, validate the added text and truncate as necessary to ensure the database field length is not exceeded.

Now the fun part. What happens when you try to store your comments on N’Ko, Mongolian, Bopomofo (phonetic markers, now commonly used as an input character set for Mandarin), or even ancient Viking runes? You get two choices, store as ASCII or ISO-8859-1 (aka Latin-1), or whatever, and you lose data. Oops. Or convert to UTF-16 or UTF-8. Hm. Wait a minute, now the value (in bytes) is somewhere between 1-3 times as many bytes as the original String length. So, how do you limit the text field to the number of bytes the database will permit? If you picked UTF-16, it’s pretty simple, divide the database limit by two. But it’s pretty wasteful of space, usually. On the other hand, you can’t predict exactly how many bytes the UTF-8 representation needs until you try it out.

The following algorithm will produce a String which, if converted to supplied Charset, will be no more than maxBytes in length. It could be less, depending on the charset chosen and the text being trimmed. This happens because it removes whole characters at once, which may trim several bytes, jumping you from 1 byte over the limit to two under.

public static String limitStringByBytes(String string, int maxBytes, String encoding) {
if(string == null)
return string;
int i = string.length() – 1;
int shaveBytes = computeByteLength(string,encoding) – maxBytes;
while ( shaveBytes > 0 && i >= 0 ) {
shaveBytes -= computeByteLength( string.charAt( i ), encoding );
i–;
}
if( (i+1) <= 0 )
return “”;
else if( (i+1) >= string.length() )
return string;
else
return string.substring(0, i + 1 );
}

As a final note (thanks to the comments by one of our faithful and numerous readers), we would like to acknowledge that we have indeed ignored the existence of the supplementary planes of Unicode mappings, sticking to the Basic Multilingual Plane in this example. This avoids the even more intricate hassle of dealing with surrogate pairs. If one of these rather obscure character encodings (Byzantine Music Symbols, Phoenician, or my personal favorite, Deseret [editors note: yeah, I didn't know what it was either. Wikipedia to the rescue], for example) should appear, it’s possible that they might be truncated mid-character. According to the Unicode standard, this is an error, but also a very unlikely situation to encounter. Free Palantir t-shirt to the first person who posts a working example that properly deals with surrogates.