Whilst Doug would not wish it to be seen as such there are two parts to this challenge:
- Find a way to have a better than 90% detection of a 1C/century trend in a naturally varying climate signal.
- Break the encryption code
Starting with the signal, the first (mental) step is to differentiate the signals in order that the trend now becomes an offset. Now we have a simple question of distinguishing between two frequencies of 0Hz – and a quick mental Fourier transform tells me that we can ignore all other frequencies except the 0Hz because they are all orthogonal (and so provide no useful information).
So, basically this test (except in the fertile imagination of GW believers), boils down to 1000 gradients and deciding which are biggest/smallest so they are most likely to have had the ±1C/century added. We can work out what confidence Doug places on this test from the figures: $100,000 prize and $10 entry fee. This suggests that there is less than 1 in 10,000 chance of allocating 90% of gradients correctly using the only test available: “is it big/small enough to be likely to have the ±1C/century added”.
The only question is what threshold to use to decide it has/has not the ±1C/century added. For this we can go back to the “Fourier transform”, work out the scale of variation and the profile from all the frequencies free from the ±1C/century added. However, although this is a pretty trivial task, it is pointless, because if Doug is capable of understanding the statistics involved, and capable of generating suitable random sequences, he is clearly capable of checking whether he could lose the prize subject to the only test: just a simple threshold test.
The Encryption Key
If Doug is going to lose money, this is where he is vulnerable. The key encoded answer:
3OT0JEN95MslFL7iZvt99rG8s9uaAUP/q9e87KDdjBb9ja5WgJm3cF0GNedfd9XhAXUz4SZEzqJSIXrQrMu6JL2+GXMzkYccPLAF3ghLEtqDMQSXnz1eCU1VhIk6g8dx891u87HBFNobq02IKBdijfL79FC92f0qtFwO5FlJMq7Eidrz7W/zPY68j/sqIv3U4LsMllnUuQooIlZmZH2Wt4s4V2Js3AAl449xrrUkFfvZD5tGUCvmf99blvHQBW/204jDV/UUiMLOcEtkQjhx1AOiQb12zeDLtanxYC78hAkqItDpnidEa+4P3kTzPrPzQi+ihTuuNqSE35eRZ65VamN+husVQ7eRwhHoi+Qi+SZlJBELzj8tvCstPJo46IZZXcAggAFhvgL+VntG2Xl/TOFpPmjoLUG6xCjSq81RlxmLb77Vjb87FG9QJZOZpvxKXt6srDToH9iCCKQ0zhzWcwsCHJC4dUJ9gj+o5uDweG8Crv/U8Fwf
This is 500 characters long and consists of characters a-zA-Z0-9/+ Thus in total 64, so, this is a 64bit number. The first thing I noticed was the unusually high number of of pairs:
99…cc…ll…ooIl…AAl449xrr…99…UU…uu…ZZXcAgg…77…CC…
There are 15 in 500 characters, but there should be only 500/64 = 7.8. Therefore there are twice as many repeating characters as should occur by chance. This means that the 64 bit number we see is not random! This reminds me of the way the German enigma machine was broken (where characters were repeated). Thus it appears that whatever way Doug Keenan has chosen to encode the text, it is not random and thus (unlike the first challenge), it may be broken. Also, unlike the first challenge, the “code” must make sense to a human, so we will be able to check if the decoding is valid because the answer must make sense. This is in sharp contrast to the first part of the test, where there is no way to validate whether the method we chose is valid except by paying this $10 and asking.
The response
Surprisingly, the person who alerted me to this contest was Gavin Schmidt with his comment, which when I traced it back was this twitter exchange:
Bishop Hill @aDissentient
Climate scientists can win $100,000 by demonstrating their statistical prowess! bishop-hill.net/blog/2015/11/1…
There’s Physics @theresphysics
@aDissentient The phrase “more money than sense” springs to mind.
Robert Grumbine @rgrumbine
.@theresphysics @aDissentient Note the entry fee. It’s a handy way to pick up some dollars with no danger of paying out.
Gavin Schmidt @ClimateOfGavin
@rgrumbine @theresphysics might be worth $10 to just answer ‘none’ – the whole thing might just be his idea of a joke.
2:09am · 19 Nov 2015 · Twitter for iPhone
From which I can deduce two simple facts:
- That they are aware of the challenge and cannot now say they did not enter “because we didn’t know about it”.
- That they think it is trivially simple and so they have no excuse that it was “too complex” for them to enter (as I would have guessed from their whole casual attitude to natural variation).
They will not enter
Instead, given their ignorance of the statistics as portrayed in their tweets and yet again by anders on Bishop Hill, they have no idea what they are dealing with – so in order to understand how futile the task is, they will first have to talk to someone who understands the statistics involved. And whilst they will only tell them how futile the task is, it might introduce them to the statistics of natural variation so that they too are on the first step to becoming a sceptics.
Will GCHQ/hackers enter
The idiots on climate haven’t any chance, but in effect what we have here is a $100,000 challenge to break a code that doesn’t appear random, and so appears to be breakable.
There has been an update to the Contest. Briefly, the 1000 series have been regenerated. The reason is discussed at
http://www.informath.org/Contest1000.htm#Notes
It’s a good challenge – but I’m glad you redid it after taking advice.
You were one of the people who pointed out the danger to me. I am very very greatly appreciative.
Glad to have helped.
Pingback: A $100,000 Scam | Izuru
The 64 unique patterns almost certainly means the answer key is base64 encoded. It can be converted back to its binary representation in Linux quite easily: echo “xxx” | base64 -d >keenan-binary
where the x’s are the answer pasted. What you get is binary, 375 bytes. Without the algorithm it isn’t very useful.
You can then use the little Linux program “hexdump” to inspect it.
It seems likely trying to break the encryption runs afoul of the USA’s DMCA
http://www.howtogeek.com/138969/why-watching-dvds-on-linux-is-illegal-in-the-usa/
Base64 isn’t an encryption; it is an encoding that permits posting binary on websites and email. ASCII is itself just an encoding. The difference is whether you need a key.
The Contest has now closed, and the program, answer, remarks, etc. are now posted on my website:
http://www.informath.org/Contest1000.htm
The PRNG that was originally used was a Mersenne Twister. I was astounded that it gave such weak pseudorandomness. Indeed, at first I suspected that the apparent weakness was a fluke, due to an unfortunate choice of random seed; testing showed, however, that the same problem occurred with some other seeds.
Thanks. But can you answer a simple question:-
If I had responded I would have taken a simple average gradient (LR) and picked those with the highest absolute gradient and depending if +ve or -ve have said they were +1 or -1C respectively.
If that method were used, what number would be correct?
ADDED: Just realised you didn’t say how many there were. So, therefore it would be interesting to see a graph based on the scale of gradient chosen as the cut-off (above which it is assumed +1 or -1C/century added) and the percentage right.
Pingback: Who Admits to Committing Fraud? | Izuru