I was bored, so I did a litle investigating.
data.ttarch is an uncompressed archive of thousands of smaller files, ranging in size from 48 bytes to just over 4 megabytes. I don't know how to make sense of any of the files inside, but I know how to extract them.

The format inside is:

A DWORD giving the number of "paths"
(they seem to be subfolders of some sort, eg ".\textures" or ".\animations", but don't seem to correspond
to anything in the Boneville folder. I think they might correspond to different types of file in the archive)

The paths follow straight after. They consist of a DWORD for the path length, followed by a STRING
of that length containing the path.

Then there is a DWORD giving the number of files in the archive.
The file info follows straight after. I think they're listed in alphabetical order.

It consists of:
A DWORD for the filename length, followed by a STRING of that length containing the filename
A DWORD which is always zero
A DWORD containing the file's offset in the archive data
A DWORD containing the file's length

Then there is a DWORD for the offset of the beginning of the archive data in the file
Then there is a DWORD for the length of the archive data

The archive data follows straight after.Sorry if that's hard to follow :)

This FreeBASIC (http://www.freebasic.net) program will extract the files into c:\bonedata.
I've preset the archive data offset to 0x3111D in the program, but you might want to run it through once without extracting the files, to make sure that your copy of data.ttarch has the same value.

option explicit

dim numpaths as long

dim pathlength as long, path as string

dim numfiles as long

dim filenamelength as long, filename as string
dim zero as long, filelength as long, fileoffset as long

dim filedata() as byte

dim dataoffset as long
dim datalength as long

dim seek1 as long
dim i as long
dim extractfiles as string * 1

'presetting dataoffset:
dataoffset = &h0003111d

input "Extract files? ", extractfiles
extractfiles = lcase$(extractfiles)

if extractfiles = "y" then shell "md c:\bonedata"

open "C:\Program Files\Out from Boneville\Pack\data.ttarch" for binary access read as #1

get #1, 1, numpaths
print "Number of paths:"; numpaths

for i = 1 to numpaths
get #1, , pathlength
path = space$(pathlength)
get #1, , path
print path
next i

get #1, , numfiles
print
print "Number of files:"; numfiles

for i = 1 to numfiles

get #1, , filenamelength
filename = space$(filenamelength)
get #1, , filename: print filename

get #1, , zero: print right$("00000000" + hex$(zero), 8),
get #1, , fileoffset: print right$("00000000" + hex$(fileoffset), 8),
get #1, , filelength: print right$("00000000" + hex$(filelength), 8)

seek1 = seek(1)

if extractfiles = "y" then
open "c:\bonedata\" + filename for binary access write as #2
redim filedata(0 to filelength - 1)
get #1, dataoffset + fileoffset + 1, filedata()
put #2, 1, filedata()
close #2
end if

seek #1, seek1

next i

get #1, , dataoffset
get #1, , datalength
print
print "data offset: "; right$("00000000" + hex$(dataoffset), 8)
print "data length: "; right$("00000000" + hex$(datalength), 8)

close
sleep

Cool.
I had a look at the extracted files, of course :)
Most of the files seem to start with "SEBM", the following data is xor'ed with 0xFF. The first bytes of each file seem to use another value but it works for the rest of each file so you can read text.

Could somebody write a program that cuts the first 126 bytes from the *.aud files so that one can listen to the OGGs in the "StreamedData" folder?

I will, if you buy the game

Nice.

Could somebody write a program that cuts the first 126 bytes from the *.aud files so that one can listen to the OGGs in the "StreamedData" folder?

Here you go: Bone Music Extractor (http://quick.mixnmojo.com/files/BoneMusicExtractor.exe)

Nice one CountingPine, that was quick work :)

For those who are lazy like me, here's a gui version of counting_pine's dumper - Bone ttarch Dumper (http://quick.mixnmojo.com/files/BoneTtarchDumper.exe) (hope you dont mind counting_pine :)).

One odd thing I noticed about the archive is that theres 709 bytes of (junk?) data at the end of the file that never gets used.

No, I don't mind. And thanks for crediting me :)

Looking at the data at the end of the file, my theory about the 709 bytes would be that at one point the file used to be longer, and when they wrote the final archive, they didn't clear the data off the end.

Here you go: Bone Music Extractor (http://quick.mixnmojo.com/files/BoneMusicExtractor.exe)
Thanks, Benny, that program is neat! Mind if I put this on PC gameScore (http://www.pcgamescore.de/tools.php)?)

Not at all, go ahead.

Also did anyone else notice that some of those files were .chore files? I dont for a second think there'll be any similarities, but both scumm and grime used 'chores', I think they were used for actor animation or similar, I cant really remember. Also one of the developer mode command lines from Grim Fandango allows you to invoke the 'chore' tool, but only if you have the (lucasarts only) choreface.dll. Anyway, there you go. A tenuous Lucasarts-Telltale link.


[Edit]
It looks like the .d3dtx files are just standard .dds files with a header and xor'ed by 0xFF as John_Doe said. I'll put together a quick viewer or dumper for them later :)

Weird, the images are .dds images but each image has an area of corruption on it:
Example 1 (http://quick.mixnmojo.com/temp/ui_options.d3dtx.png)
Example 2 (http://quick.mixnmojo.com/temp/ui_menu_logo.d3dtx.png)

I dont know why this happens, its very odd.

Different viewers seem to interpret the corupted areas in slightly different ways - I tried WTV (http://developer.nvidia.com/object/windows_texture_viewer.html) and the Compressonator (http://www.ati.com/developer/compressonator.html).

http://img91.imageshack.us/img91/6981/uimenulogowtv8fg.png)
http://img91.imageshack.us/img91/8434/uimenulogocompressinator2rh.png)

I guess that means there's a right way to interpret it...

Don't suppose there are any dds experts about?

I tried with WTV and the dds viewer in InfranView and got the same results. It does it with different levels of DXT compression too so its not just 1 thats wonky. I checked the size and that was correct too.

From memory the header before the dds image was:
4 bytes - DXT compression type
4 bytes - width
4 bytes - height
4 bytes - 0
1 byte - ?
4 bytes - size of dds image (without header)

It could also be that not the whole file is xor'ed but only several bytes, then several bytes not (or with a different value) and so on. This could explain the pattern of the corruptions in the dds images.

The corruption is easiest to see in ui_options.d3dtx because the corruption is at the bottom on a black background.

The corrupt bit is this (before any xor'ing):

840D 180D 9050 A69B 840D 180D 9050 A69B 840D 180D 9050 A69B 840D 180D 9050 A69B 840D 180D 9050 A69B 840D 180D 9050 A69B 840D 180D 9050 A69B 840D 180D 9050 A69B

The thing to do I suppose, is to try and get those values to 00 like the surrounding bytes (or FF as they are when xor'ed).

Another weird thing that I've noticed is this: In most files, when xor'ed with FF there's a path eg. 'F\Telltale\ArtData\UI\Bone\ui_menu_logo.tga' with a dword before that, which specifies the length of the path string. However in ui_menu_options the string is incomplete at the beginning 'elltale/ArtData/UI/Bone/ui_options.tga' and there doesnt seem to be a dword before it to indicate path length. This might mean that the missing bits are xor'ed with a different value.

Hmm... A repeating QWORD, 64 bytes total. I notice it starts at 0x4100.

I've checked a couple of files, and the path ("C:/Telltale/ArtData/...") always seems to start being legible 68 bytes in. So maybe there's another 64 bytes of corruption straight after the SEBM.

Funny how sometimes it's back slashes, sometimes forward slashes.

Which file is that in? Also whats a QWORD? 8 bytes?

Yeah: Byte, Word, DWord, QWord. 8 Bytes.

I don't know if this is any help, but a couple of the files have more extractable data after byte 68, like this (without the spaces):

d3dtx [63][00][00][00] [5B][00][00][00] C:/Telltale/ArtData/Characters/Sk08/TedsBigBrother/Textures/sk08_tedsBigBrother_bandAid.tga

Here you go: Bone Music Extractor (http://quick.mixnmojo.com/files/BoneMusicExtractor.exe)

Nice one CountingPine, that was quick work :)


Hey thanks for the music extractor!!
I tried to download the right plugin for winamp to play standard .aud files, but this didn't work of course.
However, thank to you guys I can listen to the music, which is very cool!

Ted Theme and Possum kids are my favorites.

Jaap

The font files also use dds images and they also have the same corruption.

It doesn't seem to be xor encryption at the beginning of the SEBM files. I tried xoring the first 64 bytes after the header with a QWord that would make some of the bytes legible, but the other ones still wouldn't make sense.

The strange thing is that with some files, there's unencrypted data at the end (i.e. readable ASCII text) while the bytes before are 0xFF-xor'ed and the beginning of the file somehow else.

Perhaps it might help to see what the game makes of any changes.

Bone Ttarch Creator (http://quick.mixnmojo.com/files/BoneTtarchCreator1.0.zip) - select a folder and it'll compile the files inside to a new .ttarch archive. I've tried it with the game briefly and it seems to work fine.

Hmm...the *.lenc files seem to be encrypted Lua scripts but they're not xor'd with 0xFF.
I just got an idea: Iirc there's a tool in the DirectX SDK for debugging DX programs. I think you can also dump all textures uploaded to the system. If one can get some "good" textures we can compare them to their encrypted versions. This would help to find out how the encryption works.

That sounds like a good idea, I'm downloading the sdk now.

Hmm, I installed the August sdk but it doesnt seem to have installed any of the tools. When googling I couldnt find anything about dumping dx textures.

It's called PIX: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/directx9_c/directx/directxsdk/Tools/Performance/PIX.asp)

I've got it and tried using it with Bone. Unfortunately when I use it Bone runs, but nothing is displayed on screen and I seem to get no results. I'll keep trying.

Also, t3.mll is actually a .dll, if you open it with a resource viewer you can see dialogs such as the script debugger and chore tracker.

[Edit] Whoops, I forgot that I'd renamed data.ttarch, I'll try again :~

[Edit2] Nope, I still cant get it to capture any textures or screenshots.

I've been playing with ttg_splash1.d3dtx - since its mostly white its easy to see the corrupted areas and also to know what colour those areas should be.
Some interesting stuff:
This (http://quick.mixnmojo.com/temp/ORIGttg_splash1.d3dtx.png) is what you get when you rip the original dds image
Its DXT1, with no mipmaps and 1024x1024.

I went through the file and replaced the obviously corrupt bytes with what they should be (0000FFFF), I then put the file back into the game - this (http://quick.mixnmojo.com/temp/changed_ttg_splash1.d3dtx.png) is the result.
As you can see its the corruption on the right that gets altered not the black bands - if you also look at that image, its 800x600, while the dds image is 1024x1024 - the aspect ratio is wrong. The black banding must be caused by the aspect ratio. Its possible that when the image is resized to the correct ratio those black bands line up with the corrupt areas.

(I see you're a Paint.NET fan. ;)
Assuming a "square" is 4x4 pixels, it looks like one corruption pattern repeats every 512 squares, while the other repeats every 800 squares.
I'm not sure the corruption is caused by the aspect ratio though. Wouldn't that mean that the game loads, decompresses and scales the image before reversing the corruption?
BTW, what did the image look like in IrfanView after replacing the obviously corrupt bytes?

Like this (http://quick.mixnmojo.com/temp/blah.png) (looks like I missed a bit).

I still think its something to do with the aspect ratio, it would explain why textures like ui_options.d3dtx (http://quick.mixnmojo.com/temp/ui_options.d3dtx.png) dont get the black banding - because they are never resized.

Right now I'm trying to see if I can work out some algorithm for getting the right value to xor the corrupted bytes by.

I would have thought that ui_options.d3dtx.png (http://quick.mixnmojo.com/temp/ui_options.d3dtx.png) doesn't get much corruption simply because of how small it is.


I don't think the corruption is a simple xor.
I tried xoring all the corrupted areas (every 4096 bytes there's another 64 bytes of corruption) with the same value, and I got this (http://img211.imageshack.us/img211/1396/ttg9po.png).
I think this corruption is actually a mapping. Each possible DWord gets corrupted to a different DWord. For a given file, it's reasonably easy to find what "00 00 00 00" and "FF FF FF FF" map to, and maybe a couple could be worked out from the header, but we don't know what the algorithm is for the mapping.

Whatever it is, I suspect it might be "seeded" from somewhere in the first 64 bytes after the header. (If it's seeded by the filename, you could probably show it up by swapping two files, e.g. ttg_splash#.d3dtx, and see if they display correctly.)

EDIT: They do, so it's not seeded by the filename. My money would be on the first 4 corrupted bytes. Not much help though, without knowing the algorithm.

EDIT2: The first 24 bytes (not including SEBM) seem to be common to every file. I'm not sure what this means, but I guess it means it's not encrypted in the same way.

Something funny I just found: There's a string in the t3.mll file "CSIEvidence". Seems they've been working on the CSI game for some time :)

Edit: And a lot more of CSI references :)

I was looking through some files this weekend and I found my notes for the encryption. I'll repost them here in the hope that they can help someone else.

I had switched from looking at the textures, to looking at the .prop files and was trying to get somewhere by comparing the files from different games.

Prop files:
Texas Holdem - All prop files unencrypted
CSI - All prop files in the ttarch are unencrypted, but those outside this (such as user_csi3_global_game.prop and prefs.prop) are encrypted
Out from Boneville - All encrypted
Great Cow Race - Prefs.prop file is encrypted - looks largely the same as in other games - but no 4 byte identifier at beginning



An unencrypted .prop file - from CSI
anim_group_colors.prop

4 - Identifier 'NIBM'
4 - No of classes (3)
4 - length of next string (17)
17 - string (class PropertySet)
4 - PropertySet params (3811171764)
4 - length of next string (11)
11 - string (class Flags)
4 - Flags params (2073868025)
4 - length of next string (11)
11 - string (class Color)
4 - Color params (2319156233)
4 - ? (0)
4 - ? (8)
4 - ? (0)
4 - Size of rest of file - including these 4 bytes (533)
4 - ? (1)
4 - length of next string (11)
11 - string (class Color)

In encrypted files, the first 68 bytes are encrypted, then those after this are xor'ed with FF and then there's usually a few at the end of the file that arent encrypted.

An encrypted .prop file - from Out from Boneville
anim_group_colors.prop

4 - Identifier 'NIBM'
64 bytes encrypted data:
3818 F83F B3C9 03F8 9011 7A96 5546 6F82 A914 1E8F D567 34E4 9CB4 274F BC8A 9256 2F51 EC46
C6A1 CA92 4761 814F CF26 70F6 82B0 8F83 A95A 45B3 F716 70DF E972 E1BC

Layout explained:
4 - Identifier 'NIBM'
4 - No of classes - varies between files. 3818 F83F - probably equates to 0300 0000
4 - length of next string - should be 17
17 - should be string (class PropertySet)
4 - should be PropertySet params (3811171764)
4 - should be length of next string (11)
11 - should be string (class Flags)
4 - should be Flags params (2073868025)

**Differences between prop files appear here. Probably because they reference a different class at this point.**

4 - should be length of next string (11)
11 - should be string (class Color)
**Encryption on encrypted files ends 1 byte after this at offset 68**
**xor'ing with FF begins here**

That all looks a bit of a mess now, but what I found was that in every .prop file, the bytes at offset 12-52 - were always the same and from looking at unencrypted files, I'm pretty sure this equates to:
'class PropertySet..).....class Flags...{'
'636C 6173 7320 5072 6F70 6572 7479 5365 74B4 DD29 E30B 0000 0063 6C61 7373 2046 6C61 6773 F9B6 9C7B'.

In most files the 16 encrypted bytes after this equate to:
4 bytes - string length (11)
11 bytes - (class Color)
In some files though this is different - probably because a different class is referenced there.

As usual I've explained this badly, but essentially what it means is that we now have some encrypted values and know what they should decrypt to. The 'class propertyset', 'class flags' and their preceeding string length values are found in every .prop file. I was hopeful that this would enable me to figure out the decryption, perhaps this info might help someone else.

Interestingly, when comparing files I found that when bytes 4-8 differ (the number of classes) - then the next 4 bytes (string length) also differ. This may indicate that the encryption works in 8 byte blocks.

The updated (http://www.telltalegames.com/bone/boneville) version of Out from Boneville now uses the new style .ttarch that The Great Cow Race uses. I think its a pretty safe bet that Sam and Max will use it too.

After you posted the post before the last :confused: I checked the disassembly again but I couldn't even find a piece of code that even remotely looked like a decryption function, it's so messy because of all the C++ stuff. Usually it helps to find the "file read" function and backtrace from there and then check what's done with the buffer that has been read. But with all this C++ stuff it's 100 times more complicated :(

I remember looking at a function called 'encrypt metastreams' in Bone 1, I guessed that this was what was called when an encrypted file like prefs.prop was saved. I hoped that I could work out how the encryption was done and then reverse it. Unfortunately it was hopelessly ambitious. Since I know next to nothing about assembly, I was just fumbling around in the dark :~

As expected Sam and Max uses the new ttarch format, so far it seems like a good game and everyone should buy it :) There are also 2 .d3dtx files that also feature new encryption (I think they just have a larger encrypted block at the beginning).

Regarding 'encrypt metastreams' in my post above - I did a little more digging. The engine creates a 'MetaStreamTemp_0' temporary file in the same folder as the .exe (in all tt games). If you look at this file, it contains the same data as prefs.prop - only unencrypted and not xor'ed. Perhaps that function does indeed produce the encryption in prefs.prop.

I think I finally found the code that does the decrypting.
As expected, the algorithm seems to operate on 8 byte blocks, it could be Blowfish. There's a function where a rather big table is generated, possibly the password string.
Then it decrypts several bytes, the length depends on the first 4 bytes of each file, then it does the "XOR 255" stuff (actually it simply "NOTs" the bytes).
There's still a lot of stuff besides these two parts I haven't figured out yet and don't know if it's used in the decryption or not.
Well, at least I now know where to look :)

Well done! It sounds like you're on the way to figuring it out :)
It'd be brilliant if you could, you've certainly come much closer than anyone else so far.

Ok, so it's definitely Blowfish. The tables I talked about in my last post are the Blowfish tables. I tried to decrypt the first bytes of the props file but nothing useful came out.
They probably process the data before or after the decryption.

Nice one! So those tables are the key that the data is encrypted with? You're nearly there now!

No, the tables are specific to the algorithm. The key is 56 random characters.
Also, this is just for the archived files, I don't know yet if it's used to encrypt the archive directory, too.

looks like you're right about blowfish. voices use the speex codec. :)

Any news on this? I'd just love to get access to the sam&max files.

Read this news entry from bgbennyboy's site: http://quick.mixnmojo.com/telltale-tools-an-update)

Oh, I see. What a shame, understandable but still...

oh my oh my....
so no luck... in editing the 'game.langdb' from Bone: Out from Boneville

I'm not in the mood for Sam&Max now... but.. the .langdb file from Bone is out there... and nobody was able to edit it ? :(

Ok, John, Blowfish right ? let me check what I can do about it...

btw, I know nothing about it...