[ Beneath the Waves ]

This Dust Remembers What It Once Was

article and software by Ben Lincoln

 

Table of contents

  1. Introduction
  2. Components
  3. Walkthroughs
  4. Results
  5. Thank You
  6. Frequently-Asked Questions
  7. Known Issues
  8. Plans for Future Versions
  9. Update History
  10. Downloads

Introduction

This Dust Remembers What It Once Was ("TDR") is a reverse-engineering toolkit I wrote for use with the NSA'a amazing tool Ghidra. Ghidra is a completely free, open-source binary reverse-engineering toolkit that includes not only a disassembler, but a decompiler that must have been written using black magic. I can't thank its authors and the NSA enough for releasing it last year.

I wanted to use Ghidra to help reverse engineer Soul Reaver, my favourite game of all time, but at least when I started, there were a couple of obstacles in my way: Ghidra doesn't support the proprietary PSX-EXE format used for PlayStation binaries, and it also doesn't support the PsyQ .SYM debug symbol format.

I originally started writing TDR specifically for that one project, but I've tried to generalize it enough to work with any PlayStation title that has PsyQ debug symbols available. The PSX-EXE-to-ELF converter means that any PlayStation binary should be importable into Ghidra, even if it wasn't written using PsyQ at all. It also means that any PlayStation binary can be imported into other tools that support ELF binaries, such as RetDec.

I have some additional componnents in mind for later that will extend it to other gaming platforms, but I'm not sure when I'll have time to get around to that.

Be warned, the current version of TDR should be considered an alpha release, in the traditional sense: it's feature-complete, but it's probably full of bugs. I don't know how frequently I'll be able to work on it, so I wanted to get it out there in case it was useful to someone even in its current state.

TDR is a highly-specialized reverse-engineering tool. The documentation below is pretty barebones at the moment, and assumes extensive pre-existing knowledge. I'd like to expand it in the future.

TDR itself is open-source, licensed under the GPLv3. Warning: you may regret looking at some of the code. This is a project that grew organically over about eight months. It involved lots of on-the-fly design changes because I was learning about some of the low-level details as I went.

If you just want to be able to load PlayStation games into Ghidra and don't have .SYM files, you can also take a look at DrMefistO's outstanding PSX Loader for Ghidra. A future version of this content will include an alternative walkthrough for using that loader in place of the EXE-to-ELF step, for those who wish to compare.

Components

The current version of TDR is made up of four tools (in addition to Ghidra itself, which you'll need to install separately):

CreateSkeleton.exe does the bulk of the work in the current version of TDR. From the input data, it generates the following:

Walkthroughs

I've written five walkthroughs to help people jump into the basics of using TDR. Following these walkthroughs will get you a long way, much faster than manually reverse-engineering these games from scratch, but you'll still be doing a lot of manual work in Ghidra if you want to generate code that will compile back to a working game binary. If all you're looking for is mostly-accurate decompiled code to use as a reference for e.g. reverse-engineering file formats, you might not have to do any additional work, though.

The reason there are five walkthroughs instead of three is that with earlier (pre-0.8) versions of TDR, numerous game-specific manual workarounds were necessary just to get basic decompiled code. Game-specific manual work will always be required to get highly-accurate results, but as of version 0.8, three of my five main test cases (Soul Reaver, EDGECASE, and an unnamed prototype) can all have their first pass done without any of the manual workarounds that were previously documented here. Need for Speed 4 requires one minor fix, but I'm hoping to eliminate that in a future version. Biohazard 2, to my surprise, uses PsyQ memory overlays. It's possible to do a basic decompilation of it easily, but that won't include any of the information from the overlays. I'll add a second section later that covers the overlays in that game.

Important:

When a new version is released, be sure to delete (or archive) all of the generated files, start over from scratch, and re-read the walkthrough. This is still alpha-quality software, and the "right" way to use it is changing essentially with every release. Mixing files from different versions will result in poor output or crashes.

  1. TDR: Practice Using EDGECASE - a basic walkthrough of decompiling a very simple PlayStation PSX-EXE binary which has source code included so you can compare the decompiled code with the corresponding original source.
  2. TDR: Practice Using OVERLAYS - a basic walkthrough of decompiling a very simple PlayStation PSX-EXE binary (also with source code included) which uses PsyQ's memory overlay features (like Diablo). This type of binary requires more manual work, and the use of some additional TDR features.
  3. TDR: Soul Reaver - steps that should work with most debug builds of Soul Reaver, but specifically tested with the 1999-06-01 prototype.
  4. TDR: Need For Speed 4 - tested with the 1999-02-22 (v9.0) version of the game.
  5. TDR: Biohazard 2 - tested against the 1997-10-30 build of Biohazard 2.

Results

This section will be greatly expanded in the future, but for the most in-depth current look at results, see TDR: Practice Using OVERLAYS.

TDR works really well with all of the debug builds of Soul Reaver I've tested it against.

It does a solid job against the 1997-10-30 beta build of Biohazard 2.

As of version 0.6, it does a pretty phenomenal job with the 1999-02-22 prototype of Need for Speed 4, especially considering the extremely complex codebase for that game.

It does not do so well with the 1996-08-05 prototype version of Wipeout XL, because that build was created without the -g flag for CCPSX.EXE, which means the .SYM file essentially only includes labels, not other types of symbols. It's better than nothing, but a significant additional amount of manual work would be required decompile it to working code.

Thank You

It wouldn't have been possible for me to build and refine TDR without the following people and organizations. Sincere thanks to:

Known Issues

Plans for Future Versions

Some changes/additional features that I have planned for future versions. Some of these will probably be in the next release. Others may take longer.

Frequently-Asked Questions

Why write wrappers around Ghidra and Ghidra scripts instead of making TDR a Ghidra extension?

Ghidra extensions need to be recompiled for every Ghidra release. The current model permits users to (in most cases) continue using TDR with whatever the current version of Ghidra is without waiting for someone to recompile TDR.

Additionally, a separate set of tools permits analysis of PlayStation code in other tools, such as RetDec.

Some planned future elements of TDR may be created as Ghidra extensions if that is the best approach for them.

For a game with memory overlays, like Diablo, is there a way to process all of the overlays at once?

You can use SymDumpTE.exe without the --json flag to get low-level information about all of the overlays in text form, and you can include more than one overlay when calling CreateSkeleton.exe and PopulateSkeleton.exe as long as none of the overlays overlap in memory, but you can't process overlapping overlays in the rest of the toolchain. If that doesn't make sense, try going through the TDR: Practice Using OVERLAYS walkthrough.

Update History

In reverse chronological order:

Version 0.9 - released 2019-09-25

Version 0.8 - released 2019-09-11

Version 0.7 - released 2019-08-22

Version 0.6 - released 2019-08-19

Version 0.5 - released 2019-08-14

Version 0.4 - released 2019-08-14

Version 0.3 - released 2019-08-14

Downloads

 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was 654 KiB 0.9 2019-09-25 Ben Lincoln
This is the .NET executable version of the TDR suite. If you want to use the tool, this is probably the file you want to download.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was (Source Code) 5 MiB 0.9 2019-09-25 Ben Lincoln
This is the .NET source code for the TDR suite.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was 644 KiB 0.8 2019-09-11 Ben Lincoln
This is the .NET executable version of the TDR suite. If you want to use the tool, this is probably the file you want to download.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was (Source Code) 1 MiB 0.8 2019-09-11 Ben Lincoln
This is the .NET source code for the TDR suite.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was 578 KiB 0.7 2019-08-22 Ben Lincoln
This is the .NET executable version of the TDR suite. If you want to use the tool, this is probably the file you want to download.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was (Source Code) 1 MiB 0.7 2019-08-22 Ben Lincoln
This is the .NET source code for the TDR suite.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was (Documentation) 586 KiB 0.7 2019-09-09 Ben Lincoln
The documentation for version 0.7 of the TDR suite. Available for anyone who wants to use the old version for some reason. Most of the content applies to versions 0.3 - 0.6 as well.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was 575 KiB 0.6 2019-08-19 Ben Lincoln
This is the .NET executable version of the TDR suite. If you want to use the tool, this is probably the file you want to download.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was (Source Code) 1 MiB 0.6 2019-08-19 Ben Lincoln
This is the .NET source code for the TDR suite.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was 571 KiB 0.5 2019-08-14 Ben Lincoln
This is the .NET executable version of the TDR suite. If you want to use the tool, this is probably the file you want to download.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was (Source Code) 1 MiB 0.5 2019-08-14 Ben Lincoln
This is the .NET source code for the TDR suite.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was 571 KiB 0.4 2019-08-14 Ben Lincoln
This is the .NET executable version of the TDR suite. If you want to use the tool, this is probably the file you want to download.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was (Source Code) 1 MiB 0.4 2019-08-14 Ben Lincoln
This is the .NET source code for the TDR suite.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was 568 KiB 0.3 2019-08-13 Ben Lincoln
This is the .NET executable version of the TDR suite. If you want to use the tool, this is probably the file you want to download.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was (Source Code) 1 MiB 0.3 2019-08-13 Ben Lincoln
This is the .NET source code for the TDR suite.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was 558 KiB 0.2 2019-08-06 Ben Lincoln
This is the .NET executable version of the TDR suite. If you want to use the tool, this is probably the file you want to download.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was (Source Code) 1 MiB 0.2 2019-08-06 Ben Lincoln
This is the .NET source code for the TDR suite.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was (Documentation) 118 KiB 0.2 2019-08-13 Ben Lincoln
The documentation for version 0.2 of the TDR suite. Available for anyone who wants to use the old version for some reason.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was 557 KiB 0.1 2019-08-06 Ben Lincoln
This is the .NET executable version of the TDR suite. If you want to use the tool, this is probably the file you want to download.
 
Download
File Size Version Release Date Author
This Dust Remembers What It Once Was (Source Code) 1 MiB 0.1 2019-08-06 Ben Lincoln
This is the .NET source code for the TDR suite.
 
[ Page Icon ]