Reverse Engineering Safari's Cookies.binaryCookies file format

An Indie Developer’s short tale in overcoming one of MANY problems developing for the macOS platform

Building products for the Apple ecosystem, while a very rewarding experience can be stressful… especially around the dates for the Worldwide Developers Conference. This is when Apple give us Devs just a few months to prepare for the next macOS release, with all the deprecations and outright course changes that come with it.

The situation

I was working solo on the simple cookie manager app for Safari at the time. The initial project was inherited from previously defunct Safari SIMBL plugin which had been abandoned by the original developer.

First, a little background

I had become interested in Privacy back in the early days of Mac OSX, while backpacking around the US in 2000. Needing ways to clear traces of what I had been looking at on public computers, I discovered HTTP cookies, and promptly Yahoo’d (most likely) how to clear them from the public Browsers I was leeching internet time from. I felt like a hacker… hah… clearing any trace of my browsing sessions before leaving the store/Internet cafe I was in.

After working as a furniture removalist in San Francisco for around 6 months, I’d saved enough cash to buy my first laptop. roughly $4000 - ouch. The 500MHz PowerBook G4, what a beast! One of the first things I did was to look for and install something which would clear my cookies. I discovered a number of SIMBL plugins for Safari and installed them all. The laptop was actually stolen by a so called friend in Calais, France a few months later. :(

Now, I forget the exact name of plugin I grew fond of, but I think somewhere around Mac OS X Tiger, the developer stopped updating it. I was working gigs as a clown/juggler at the time, just married, and was spending a lot of time at home with my young son. I needed a challenge, and managed to somehow find an email for the dev. I asked if he would let me get the plugin updated to work with the latest version of Mac OS X - and Surprisingly he said yes, and sent me his codebase!

Over the next few years, I taught myself to code, and released the plugin as shareware. When the Mac App Store was announced, I knew the only way forward was to rewrite the plugin as a standalone app as SIMBL plugins were slowly being deprecated.

The Problem

So, I had this app in the Mac App Store from the beginning, and with the announcement of Mac OS X Lion (I think?), It was clear that the global NSHTTPCookieStorage object was being siloed into per app sandboxes and the easily read Cookies.plist file was being deprecated in favour of the undocumented Cookie.binaryCookies binary file format. The only way to keep my app working in the next OS X release was to reverse engineer the file format, and there was not a lot of info around its’s content in the early days… in fact I still believe it is a sparsely undocumented file format. Successfully decrypting the file was the only option, as failure would lead to the eventual obsolescence of the app, and a loss of a substantial (at this point) portion of my income.

Research

Apple developer docs were mostly useless, and StackOverflow also didn’t really provide much in the way of what was needed to parse the file. I finally managed to stumble across a security researcher’s work on documenting the format. (I would post a link, but the page is no longer online) It was however unfortunately incomplete, only documenting the headers in the file, but not giving access to the individual cookies present. His implementation was also written in an unknown language to me… Python. Translating the python implementation into Objective-C was not much of an issue, but where to start with decrypting the individual cookie data was much more of a challenge.

Decrypting the file format

I spent many weeks staring at my hex editor trying to make sense of the seemingly garbage bytes. I even at one point resorted to asking a friend studying a PHD in Physics to throw some data points I had collected into MatLab to see if there was some kind of linear formula I could use to read the individual cookies. Unfortunately there was not…. That way forward was a total dead end. With the release of Mac OS X Lion looming, the certain demise of my business model, and many sleepless nights - I changed course, went back to studying the file header and looking for potential similarities to rest of the file. I finally discovered the “key” to unlocking the Cookies.binaryCookies file, which was quite simple in retrospect. Each cookie was stored in a relevant bucket, with headers for location and data points stored within these buckets. Yay. 🎉

Live to fight another day/(macOS release)

With the exception of my family relying on me to pay the bills - I fortunately had no stakeholders to answer too. But in a way, so much more was at stake. Through the course of solving the issue I discovered the importance of not being tied too much to a single way of doing things, and to be prepared to potentially start from scratch at any time. Being in bed with Apple, and macOS…. means that things are constantly being deprecated and updated - which is never boring…