Can ChatGPT solve the GCHQ Christmas Challenge?

You know what GPT is and you are probably aware that GCHQ is the UK’s intelligence, security and cyber agency.

GCHQ states its mission is “to help keep the country safe”. I imagine that among the many known unknowns it wants to keep us safe from are the potential risks posed by developments in artificial intelligence (AI). So by way of an intro to a series of posts about how worried or otherwise we should be about the rise of AI, I thought it might be fun to see if ChatGPT can solve the GCHQ Christmas quiz.

The GCHQ Christmas challenge is comprised of a series of seven increasingly complex puzzles that need to be solved to reveal a Christmassy message. The solution to each individual puzzle is a one-word answer which can follow ‘Christmas’ (e.g. Christmas Quiz).

I found the first couple of puzzles pretty easy. The middle ones were more difficult. I won’t spoil your enjoyment of attempting the quiz yourself by providing solutions to them here. But the rest of this post does provide spoilers to puzzle number seven. So, before reading on to find out how ChatGPT and my answers differed, you might like to download the quiz here and give it a try.

Here’s the fiendish last puzzle.

It looks tricky but it wasn’t as challenging as I thought. Cracking anything requires the testing of assumptions and intuitions via trial and error (well that’s the way I do it). 

I assumed that the punctuation was deliberate and not encrypted. Meaning the message has four lines, each ending in a deliberate full stop. I also assumed that spaces between these encrypted words corresponded to the spaces between the words in the unencrypted solution.

Putting these assumptions together suggests that gq’l = it’s (i.e. the apostrophe is an apostrophe in both versions of the message).

Similarly, the c on its own is probably a one letter word, so A or I. It’s most likely a as c wasn’t capitalised. The letter a also seems to fit the phrase “… a 4-something something.”. Given the context of the quiz you might be able to immediately guess what those somethings are. I have to admit that I didn’t. Well, not immediately. Instead, my immediate thought was “hang on, even if it is Christmas, this seems far too easy for a GCHQ quiz, maybe it’s a red-herring and they want me to get lost down a dead-end”.

Given this was the last and most difficult puzzle, it’s possible that spaces might have been randomly inserted into the encrypted message to obscure the word boundaries in the original message. And the apostrophe, the 4, and the dash (-) might thus be encrypted characters of the original message.

Enter Holmes. That’s Sherlock Holmes as opposed to HOLMES the UK police’s crime cracking computer system.

I vaguely remembered reading The Adventure of the Dancing Men as a child. In this adventure Sir Arthur Conan Doyle describes how Holmes was able to decipher a series of secret messages. The messages took the form of a “series of mysterious stick-like outlines of men in various attitudes“.

Holmes then used frequency analysis to match each stick figure to its corresponding letter in the English alphabet. Following Holmes’s lead I ranked the letters of the encrypted message by order of occurrence.

Rank	Letter	Occurrences
1	m	12
2	d	7
3	k	6
4	q	6
5	g	5

According to Holmes, “E is the most common letter in the English alphabet, and it predominates to so marked an extent that even in a short sentence one would expect to find it most often“.

Predominates to so marked an extent, lets decrypt the archaic parlance of that marvelous manner of speaking. In the vernacular of GCHQ it means the E occurs an order of magnitude more frequently than any other letter. For our small sample of encrypted letters it means that m is highly likely to be E.

Previously, I guessed gq’l = it’s. Now, according to most statistical sources T is the next most common letter, and I ranks in the top 5 most common letters in English. So, so far so good assumptions wise. Let’s take a look at our partially decrypted message so that I can show you what I did next.

1. Agklq ldhum qom ndem.
   *I*ST S***E T*E ***e
2. Gembqgax c 4-hmqqmk vdke. 
   I*E*TI** A 4-*ETTE* ****
3. Hddp mumkxvomkm.
   **** E*E****E*E
4. Ycxim gq’l umkx diugdsl.
   ****E IT’S *E** ***I**S

While doing the frequency analysis and substitution I noticed a number of things.. First the three-letter word qom. The two most common three-letter words are AND and THE. So along with the frequency analysis that gave me q=T and m=E that means o=H. Next, the solution to the puzzle will reveal a 4-letter word, thus it’s very likely the second half of the second line reads A 4-letter word. Thus giving us the following pairings:
h=L, k=R, v=W, d=O, and e=D. We can test the validity of this by adding this new information into our partially decrypted message:

1. Agklq ldhum qom ndem.
   *IRST SOL*E THE *ODe
2. Gembqgax c 4-hmqqmk vdke. 
   IDE*TI** A 4-LETTER WORD
3. Hddp mumkxvomkm.
   LOO* E*ER*WHERE
4. Ycxim gq’l umkx diugdsl.
   *A**E IT’S *ER* O**IO*S

The second word of the 3rd line must be EVERYWHERE. Therefore, x=Y, and u=V. This in turn helped me solve the 2nd word of line one.
*olve must be SOLVE. Which means the first line must be
FIRST SOLVE THE CODE
and the last line *A**E IT’S *ER* O**IO*S becomes
*AY*E IT’S VERY O*VIO*S, which quickly leads to the final solution:

FIRST SOLVE THE CODE
IDENTIFY A 4-LETTER WORD
LOOK EVERYWHERE
MAYBE IT’S VERY OBVIOUS

Thus the final solution is an acrostic where the first letter of each line yields FILM, as in Christmas film.

Now, in the words of Elon Musk, that took me a non-trivial amount of time.

When I asked ChatGPT if “he” could crack it, he was much quicker than me and came back with a confident response in milliseconds. Quick but very, very wrong.

So, it looks like GCHQ will need human code crackers for sometime yet, and us human writers of creative non-fiction might have a job for a little while longer.

In part two I’ll share my conversation with ChatGPT where I discuss the failings of his answer with him and he arrogantly maintains that he is right.