The problem with Numerai’s Erasure protocol

Numerai is a blockchain startup. with a cryptocurrency boldly named Numeraire. They have just proposed a marketplace for predictions called Erasure. The premise is that there are lots of people out there with some ability — it may be divine or it may be that they have data and a good statistical model — to predict things. The problem is that there is so much noise out there having a good prediction of something (say, iPhone demand) does not necessarily mean you can make money buying or shorting Apple stock. But a hedge fund might find the prediction valuable when combined with other capabilities (including predictions) they have.

There are several problems in trying to sell a prediction — which is just information.

1. Quality: how can you know whether a prediction is of high quality before the fact? So you get a familiar ‘lemons’ problem that tends to kill markets like this.
2. Flooding: you can predict anything accurately if you have enough predictions. For instance, if I submit 6 predictions of the roll of a dice, I’ll be 100% guaranteed of being correct. So if by some means, I can sell a prediction and be rewarded for accuracy, by flooding the market and spanning the state space, I can be rewarded.

Here is how Erasure intends to resolve these problems.

Quality will be resolved at the predictor level. You can’t assess the quality of an individual prediction but you can assess the quality of a stream (or feed) of predictions. In other words, there will be a market for predictions of iPhone demand for each quarter but not for iPhone demand for a specific quarter. Thus, prediction sellers need to earn a reputation for accurate predicting.

• Here is where the blockchain comes in. One of the earliest posited applications of the Bitcoin blockchain was to use it to timestamp the contents of a particular digital asset. In this case, you come up with a prediction ahead of time and a date on which the prediction will be revealed to everyone. The timestamp verifies you had indeed predicted ahead of time by comparing your announced prediction to the hash (think encoded) prediction on the blockchain. This ensures that the prediction is a meaningful thing to sell. (By the way, you don’t need a blockchain for this — in the past, newspapers were fine — but a blockchain is easier to use here).
• What this means is that you can place a prediction on Erasure and then, at the appointed time, Erasure can assess the accuracy of the prediction and announce it to the world. Over time, you will develop a score of your accuracy (and maybe other stuff) which will be a signal to people as to whether your prediction feed is worth buying.

Flooding is resolved by staking. You can put forward an amount of money, S, as a stake in a particular prediction feed you supply and buyers can see that stake.

• Now I found it a little difficult to know what happened with a stake if all goes well (that is, your buyers are satisfied). But I think you have it returned at some point. It looks like you stake a feed but I wonder when you get it back. Perhaps when you stop selling that feed.
• If you have dissatisfied customers, then your stake is at risk. One option would have been to have dissatisfied buyers being able to destroy your stake at will — this would be akin to leaving a bad Yelp review. Your stake would be reduced by an amount, d, and future buyers would see you have a smaller stake now and take it as some kind of signal.
• Erasure does not do that. It puts a cost on communicating dissatisfaction to other buyers. In particular, if you want to reduce a stake by d, you have to pay an amount ad (where a < 1) to do so. The factor (called a ‘griefing factor’), a, is set by the prediction feed seller and they can change it over time. The idea is that a low a signals confidence while a high a signals the opposite. Suffice it to say, I would love it if this was part of student evaluations.

Now I looked to see if Numerai had any economists as advisors and they do not appear to have one. Which is why they may not have realised that some very simple economics tells us why this scheme (as thought out as it is), won’t work.

It all hinges on the staking being something real — that is, if you have bad predictions, you lose your stake, S. However, the notion of griefing does not appear to be a solution here. Here is what Erasure are asking buyers to do:

If you are unsatisfied with a prediction feed, pay an amount to tell other buyers to stay away.

See the problem? Why would you want to do that? What this relies upon is buyers being ‘nice to the community’ and paying to ward people away. The problem is that we know it is hard enough for people to leave relatively cheap reviews let alone spend money to tell people they aren’t happy with a prediction. This is even more so given that anyone can see the prediction accuracy score anyhow.

So my prediction is that prediction sellers will not have their stake at risk and, given this, will have an incentive still to flood the market with predictions. After all, if you have one good prediction and 1,000 bad ones, to lose your stake on the bad ones, someone would have to pay 1000aS! That doesn’t sound like much of a risk to a state spanner. Also, you may have time to adjust a to minimise even that.

Moreover, there is a risk to good prediction sellers. If someone sees that your feed is doing well, they might start griefing you so as to keep your good feed to themselves and signal to others to stay away!

In summary, the problem with Erasure is not the idea of using the blockchain to help establish the quality of predictions. The problem is the mechanism they are using to stop state spanning and flooding the market with bad predictions anyhow. My hunch is that they should ditch that part of their protocol and instead provide the prediction market with some way of limiting the number of predictions sold on a given random variable. Maybe they could auction that off.