How to find file encoding and decode?

user_279362 0
Viewed 228

I'm trying to understand how to decode a file that Spotify leaves on my system.

The file in question is called context_player_state_restore and lives in /Users/<myuser>/Library/Application Support/Spotify/PersistentCache/, so obviously the expectation is that its structure (and now encoding likely) can change without warning.

The Spotify app updates that file at regular time intervals with the currently playing song, the current context (e.g. currently playing playlist), and some other useful stuff like the history of played tracks.

I want to parse the information in that file for an app I'm building. (I can use the Spotify API for some of this data, but the local file contains more useful information, is immediate (no delay in API calls), and I can react to any changes on it).

Until recently that file was an UTF-8 JSON file and could be parsed with JSON.parse(data).

Recently however, Spotify has changed that JSON / UTF-8 encoding.

Currently, opening that file with vscode (with UTF-8 encoding) gives back some legible text, indicating that some of the JSON structure has been kept, but also shows a lot of illegible text – the odd question mark symbols �� and various UTF characters that don't belong e.g. Ω.

In short, I'd like to understand:

  • whether I can find out what encoding Spotify is now using for this file
  • and whether it is possible to decode the file back to JSON (or a similar format) so it can be easily parsed

Gist link to file dump as seen by vscode

Google Drive download of file

Gist link to hexdump of file (obtained with hexdump -C context_player_state_restore > context_player_state_restore.hexdump)

node.js utf-8 encoding json javascript
3 Answers

Regarding your question

what encoding Spotify is now using for this file

I believe it is the encoding used by Google Protocol Buffers. That page gives the example of the string "testing" encoded as 12 07 [74 65 73 74 69 6e 67], and this pattern "0x12 followed by the string length followed by the UTF-8 encoding of the string" appears throughout your context_player_state_restore file.

If you know the corresponding .proto file, you can decode context_player_state_restore with the protobufjs package.

user_164629500

Spotify may have simply switched encodings, or they may be intentionally obfuscating the data. I'd try reading is as an ISO-8859-1 file, as I've seen these confused with UTF-8 in the wild with some frequency. If that doesn't work, try reading the header bytes and BOM bytes (if present) with a hex editor and see if you can deduce the encoding that way. This is a good hex editor for VSCode.

The first few bytes of a file generally indicate it's encoding. There is are useful tables here and here. However, even reading the header you can never be sure what encoding a file is really using.

For example, a file with the first three bytes 0xEF,0xBB,0xBF is probably a UTF-8 encoded file. However, it might be an ISO-8859-1 file which happens to start with the characters . Or it might be a different file type entirely.

The first few bytes may also include Byte Order Marks (BOM).

user_86056850

"I want to parse the information in that file for an app I'm building...
Is it possible to decode the file back to JSON (or a similar format) so it can be easily parsed."

The format seems easy enough to reverse-engineer. Just use your Hex editor to notice patterns.

For example, most entries seem to follow this format:
[Length in 1 byte]-->[Content (by Length)]-->[END/Delimiter is sometimes byte 0x12]

Looking at the bytes provided...

(1) Starting bytes...
The first 17 bytes are 31 36 35 39 33 36 33 38 33 36 31 35 33 23 08 0F 12
The SIZE here is actually at the ending byte: 0x0F which is 15.
I suspect the (towards) ending byte: 0x23 (which is #) is itself a delimiter to mark some ending point.
By selecting all bytes from file start up to this # delimiter, we get 31 36 35 39 33 36 33 38 33 36 31 35 33

Which makes 1659363836153, which is itself a UNIX timestamp:

GMT: Mon Aug 01 2022 14:23:56 GMT+0000

Maybe the file was updated 5 minutes before your Question was posted here? ie: asked Aug 1 at 14:29.

After the UNIX timestamp there follows the delimiter byte 0x23, followed by an unknown 0x08, followed by a SIZE byte of 0x0F. (2) Parsing begins... The next row (bytes position 16 onwards) has the already described pattern of:

[Length in 1 byte]-->[Content (by Length)]-->[END/Delimiter is byte 0x12]

Entries begin with an 0x0A or an 0x12 (where 0x12 usually means text entry like "false" or similar).

(3) File structure

The file seems to have data in this order:

  • Player setup/config
  • Playlist Names
  • Playlist related images

Then under UTF-8 location: spotify.player.proto.GlobalNode

  • Playlist track details (ie: the album it comes from, related image, etc).

However GlobalNode comes after the prevoiusly mentioned !player config" which has this example setup (not all items included):

Where spotify is a key and player would be a sub-entry (child) in a JSON.

spotify -> player   -> proto    -> PlayQueueNode

audio   -> episode  -> speed
        -> automix
        
video   ->
        -> subtitles_cc
        -> subtitles

player  -> banned   -> tracks
                    -> context_tracks
                    -> albums
                    -> artists
user_20577090
Related