Sparse Systems

Sparsity is one of my favorite filters to apply to the world. One of the reasons that sparsity works so well is that various forces conspire to hide that it exists. So if you have an idea of what sparsity is, you have an edge in a world based on information asymmetry.

At a basic level, sparsity just means that, whatever you're looking for, there ain't much of it, and there's a whole lot of area to cover to find it. So if an undiscovered shipwreck is somewhere in the Caribbean, and you go out hunting for it, that's a setup with a sparse system.

Sparsity isn't a useful filter until you get to the game-theoretic implications. In real life, there are teams of people competing to find the shipwreck, and only one of them gets to claim it. Ultimately, sparsity falls under the heading of economics, and it's directly related to notions like competition and risk and scarcity.

Good people who enter sparse systems and don't know it come back bruised and broken. They try to do a bang-up job on whatever it is, and they're persistent in their attempt, and in the end they're defeated. Why? Because that mentality does not at all apply in this arena. Winning in a sparse system means covering as much ground as possible. So until you hit the thing you've been looking for, you need to 1. turn out quick, slapdash jobs and 2. quit sub-areas early; and both of these ideas run counter to the way we usually think.

The one thing you need to be able to do is to identify when you're dealing with a sparse system. That way you'll check the urge to spend too much time searching any one area. Identifying sparsity also lets you make several other assumptions. Compared to the size of the space, you should think of yourself and your effort as small and finite. Don't delude yourself into thinking you can cover the entire space, and disown the idea of getting everything perfect. Succeeding in a sparse environment is a question of resource conservation. You need to distribute your effort in a way that maximizes the territory you cover. Subtasks should be put into time boxes. Tasks that go on for too long require that you time out. Your supplies are limited and you have to be stingy with your fuel.

If you think of a classically sparse area, like outer space, what scares people about it is getting lost. The open oceans or the deserts are just as scary. You'll run out of supplies if you get lost, and it's easy to get lost in a formless waste. The difficult part about navigating a formless environment is the lack of markers. However, if you're crafty about it, you can do what navigators have always done, which is to exploit some hidden structure that isn't immediately apparent in your environment. This is how we get navigation instruments that use magnetic fields, stars, gyros...

Conceptually, navigating in a sparse system appears to be a search problem in an area where there is no readily apparent structure. In practice, though, whatever system you're dealing with will yield some kind of structure, if you look for it. The trick here is that you have to apply your intelligence in unexpected ways. Even if the space appears completely undistinguished, eventually one of the opinions you form about it will fit. This will imbue it with exploitable structure, and then you can begin to solve it.

Once you're comfortable solving sparse problems, there's a neat trick you can do. Sometimes a non-sparse problem can be turned into sparse problem by raising the threshold of what's acceptable. Then the techniques that apply to sparse problems come into play. This makes the problem more difficult, but it also makes the potential payoff greater.