Yeah, I know, I kind of suggested (if not exactly promised) in the last post that I would let go of fiddling with the tooling and get back into reconstructing the source code for the game. But let’s stop kidding ourselves; there’s an elephant in the room that we need to talk about.
The pre-release source code for Microprose’s 1994 Fleet Defender got leaked apparently as far back as the 90s, being passed around on BBSes. I felt kind of silly when I heard about it a few years into my F15SE2 project because I was completely unaware of it. Could it be that I had (most) of the source code available from the very beginning and just didn’t know? That would be pretty embarrassing. At least I did not go into the main flight game engine in EGAME.EXE yet, and if I could match the leaked source code to the game binary routines, it could shave years off my reversing efforts.
As far as I understand it, based on superficial similarities and information from ex-Microprose employees that were kind enough to talk to me, the late-80s-to-early-90s Microprose MS-DOS flight sim codebase was evolving and being shared between projects as new games were being developed. The origins of this codebase are somewhat murky, but Sid Meier gives some hints in his book:
(…) just a few months after Red Storm Rising hit shelves, I took the opportunity to return to the flight simulator genre with a game called F-19 Stealth Fighter. It was a half upgrade, half sequel to an existing game called Project Stealth Fighter, with the major distinction being that this version would be developed on the IBM personal computer. A few older games had been directly ported up to the new system, but they didn’t take advantage of the new technology; they just looked like C64 games running on a bigger machine. (…) I was intrigued by the chance to explore this topic with an entirely new code base (…)
That makes it sound like F-19 was largely developed from scratch even though it’s possible they picked some bits from their earlier DOS titles. The timeline of this codebase would therefore go something like this:
- F-19 Stealth Fighter (1988)
- F-15 Strike Eagle 2 (original 1989, expansion 1991)
- F-117A Nighthawk Stealth Fighter 2.0 (1991)
- F-15 Strike Eagle III (1992)
- Fleet Defender (1994)
There have been a bunch of MPS games released in between these dates, including some flight simulators. So, does that mean that parts of that common code are in Knights of the Sky (1990), Gunship 2000 (1991), B-17 Flying Fortress (1992) or even M1 Tank Platoon (1989)? I don’t know, probably yes. I’m pretty sure parts of the video display code, sprite and overlay handling are shared even with non-sims like Civilization (1991). I especially don’t know what happened to the code between F-117 and F-15 III because they look radically different and I’m not sure if there was an intermediate step, or did F-15 III just take the F-117 code and improved on it heavily. It’s also a big unknown how the ports were handled; F-19 was on both the Amiga and Atari ST, F-15 II was additionally ported to the Sega Genesis, F-117 was likewise on the Amiga, and it was only with F-15 III that MPS started targeting DOS exclusively with this codebase. I wonder if any of the code written for the PC ended up in any of the ports. There were C compilers available for the Amiga like Aztec C and Lattice C, but I think the emitted code would be too inefficient for the needs of game development, and I expect that the whole thing was rewritten in assembly for any ports MPS might have done, but I could be wrong.
Bottom line, there’s at least 3 product generations between F-15 II and Fleet Defender. That makes it unlikely for them to be largely identical, but surely some routines for flight dynamics and/or 3D projection could have remained (mostly) unchanged? The possibility that the answers to F15SE2’s most difficult riddles might lay in the Fleet Defender code repository is something that I could not resist, so I downloaded it and started looking around.
There are of course ethical/legal questions involved. The source code was not officially released and is therefore for all intents and purposes, stolen. I thought long and hard about whether or not I should look at it at all, because I don’t want to compromise my project, but I finally decided that the opportunity was too great to miss. However, I decided I would under no circumstances lift any part of the code from the leaked repository into my own project. My usage of the leaked code will be limited to studying and potentially applying the conclusions to my project – if there are any to apply. The leak was of a pre-release version and the source code is incomplete (more on that below), so any harm to the sales of Fleet Defender would be negligible, and it’s more that 30 years in the past. These days, the leaked source code is an artifact of historical research. I think of it something like one of the works of ancient art in Berlin’s Neues Museum. Next to many of these items you will find notes euphemistically saying “Acquired in 1935” or “Found in Greece in 1918”, which means they were picked up for peanuts or just straight up stolen from other countries and brought over to Germany back when hardly anybody cared about those treasures, or realized they could become valuable. Their return is a contentious matter that’s still under discussion. There’s also an argument to be made that if the Germans hadn’t taken and preserved the items, they would probably have been lost to neglect or chopped up to make road gravel. But does that mean that we can’t or shouldn’t admire them in the meantime? No, that would be an even greater waste, so I’m not going to pretend like this gem of early 3D gaming technology doesn’t exist, either.
With that out of the way, let’s see what we can learn from that source code. There are 79 C files, 27 header files and 17 ASM files, and the whole source repository weighs in around 4MB. The sources are about half of that, with the rest being what looks like various campaign scenario files, damage tables and aircraft stats, both in readable formats and compiled into binary equivalents, with some of the compiler tooling included in the repository. Those might be useful for modding and/or creating new missions. Several files contain C code while having odd extensions like .bk1/.ori/.mik – these seem backup/work copies of equivalent .c files for tweaking.
A look at the code
Here are some interesting takeouts from the Fleet Defender source code:
/*----------*/
/*
/* File: 3DObject.C
/*
/* Auth: Andy Hollis 11/25/87 <<<<< Amazing !!!
/*
/* Edit: Hacked yet again for F15 III, AWH - 8/92
/* AND YET AGAIN FOR F-14, MJM 4/93
/*
/* Routines to draw a 3D object. - NOT!!
/*
/*----------*/Most of the files in the repository are dated 1990-1994, with the oldest ones (containing trig function value tables) from 1989, but this appears to be the oldest date mentioned in the codebase, and the comment seems to indicate the Fleet Defender dev seemed appropriately impressed. The comment appears to confirm that the F-19 codebase has been reused in multiple projects over its lifetime, and parts of it made it into F-15 III and Fleet Defender. The references to “F-15 (III)” are the most numerous in the codebase, with a few stray ocurrences of “F-19/Stealth Fighter”, but surprisingly nothing about “F-117”. The file itself contains just 5 short routines seemingly related to 3D views (or not?).
//***************************************************************************
//*
//* AWG9.C
//*
//* Author: Mike McDonald (adapted from Bill Beckers F15-III:APG-70)
//*
//* Fleet Defender: F-14 Tomcat
//* Microprose Software, Inc.
//* 180 Lakefront Drive
//* Hunt Valley, Maryland 21030
//*
//***************************************************************************Again, it looks like the F-15 III radar (APG-70) code was tweaked and reused for AWG-9 radar in Fleet Defender. I don’t expect to find this in F-15 II though, whose radar is extremely simplistic and arcade-like.
//***************************************************************************
// FILE: GMAIN.C
//
// Fleet Defender - F14 Tomcat
// Project Manager: Scott Spanburg
// Revised by: Mike McDonald
//
// Adapted from F-15 Strike Eagle code by Sid Mieyer, Andy Hollis
//
//***************************************************************************
// [...]
char *F14CNUM[5] = { "F14R" };
char *F14SNAME[30] = { "Quality Assurance" };
char *F14SPER[30] = { "Vaughn Thomas" };
char *F14VNUM[5] = { "1.18" };
// [...]
main(argc, argv, envp)
int argc;
char **argv;
char **envp;
{
int z;
save_video_state();
InitOverlay(LoadOverlay("Mgraphic.exe","Fonts.F15"));
TurnOnGraphicsMode(0);
mclear_screen();
mprintf("**** DEBUGGING INFORMATION IS ON!!! - NOT FOR QA ****");
InitOptions();
LoadSoundConfig();
InitSound();
InitGraphicPages();
InitGraph('M');
// [...]
}
// [...]
MainGameLoop()
{
SetJoysticks(StickType);
KBInit();
do {
UpdateTime1();
UpdateTime2();
TakeInputs();
LocalCmds();
if ((--DisplayFrame)==0) {
GenDsp();
DoCockpitDisplays();
Messages();
}
ProcessInputs();
FLIGHT();
Stealth();
DoPlayerOnTheCat(); // keeps plane in sink with boat
AWG9();
TEWS_SYS();
UpdatePalette();
if (DisplayFrame==0) {
Flip();
DisplayFrame=Speedy;
}
} while (!BEND);
Release3DMemory();
MouseHIDE();
ClearPage(0, BLACK);
ClearPage(1, BLACK);
SndShutdown();
UnInitSpeech();
DumpLogFile();
GetRidOfKeyJoy();
}This contains the main function of the game and the main game loop itself. What’s interesting is that it was apparently adapted from F-15, and the fact that Sid is mentioned (who did not work on F-15 III) makes me think this was all the way from F-15 II. It’s not that big a deal because the function is rather simple, just calls into other functions and would be modified for Fleet Defender anyway, but it’s nevertheless cool to look at, and the routine names, structure definitions and other references might be useful. It’s interesting to see it init and load the mgraphic.exe video overlay driver, and I also remember setting the M mode value for VGA graphics in my code. The source code contains the name of Vaughn Thomas who was a tester at MPS during that time, but I don’t think that means he was the source of the leak - some developer probably created this build for Vaughn, gave the binaries to him for testing, and the source code was pilfered sometime at that point.
You can also see the ancient origins of the codebase in the fact that many routines are written in the pre-ANSI K&R style of C, with no return type (which defaulted to int), and local variables defined before the opening brace.
/************************************************************************
* *
* Project: Stealth Fighter(TM) *
* *
*<t> Flight Equations *
* *
* Author: Jim Synoski *
* Written: Jan 1988 *
* Last Editted: Jan 22,1988 *
* *
* Copyright (C) 1988 by MicroProse Software, All Rights Reserved. *
* *
************************************************************************/This comment is present in flight2.c and in views.c, although the latter with a later date of 1993. Both are calculation-dense routines seemingly for calculating flight dynamics and 3D view processing, and the fact that it seems they originated from F-19 makes me hope I can find traces of this code in F-15 II which would be super helpful.
/* File: Planes.c */
/* Author: Sid Meier */
/* */
/* Game logic for Stealth: enemy planes */
// [...]
// The Strategy:
// Planes have three overall plans:
// o PATROL means to fly amongst nearby enemy entries in the Rdrs list,
// quitting when "time" runs out. When this occurs, the nearest enemy
// base is chosen as destination and the plane flys home or just
// disappears. When near the base, landing is initiated. When landed,
// the plane is deactivated. PATROL is the default plane type.
// o LOITER is similar to PATROL, except that the target is an enemy
// base and never changes. This gives infinite touch-n-go's.
// Overriding factors include:
// o If he "pings" you, he comes after you until he loses you.
// o If "detected" planes will go after your last known position
// Slots in the planes array are as follows:
// o Last four are for CloseBase touch-n-gos
// [...]
/* File: Radars.c */
/* Author: Sid Meier */
/* */
/* Game logic for Stealth: enemy radars */
/* */
// [...]
// Radar detection check
detect(int i)
{
// [...]
// SAM WILL BE FIRED IF:
// 1) MISSILE IS ASSOCIATED WITH RADAR INSTALLATION
// 2) NOT OUT OF MISSILES
// 3) RADAR ALERT LEVEL IS HIGH ENOUGH
// 4) PLAYER IS NOT CLOSE TO FRIENDLY BASE
// 5) PLAYER IS WITHIN MISSILE ENVELOPE
// [...]
}
// [...]
// General radar detection algorithm for ground radars and planes
Rsignal(COORD x,COORD y,int z, int type,int *ang,int *dst,int isfriend,int *targetnum)Likewise, these two files credited to Sid and seemingly originating from F-19 seem like they might carry over to F-15 II. planes.c is over 6000 lines long! The radar handling logic looks to cover both SAMs and air-to-air, and I think it could be more useful than the advanced AWG-9 stuff.
stealth.c:495: if (FTicks<4) { /* Don't exceed 15 FPS */
stealth.c:499: TickDelay = 0; /* Don't run any slower than 3 FPS */I’m not sure if this means that the entire game is capped at 15 FPS, or is it just the stealth handling code running at that speed. Probably it’s the latter, but as I remember it, Fleet Defender isn’t very smooth even on fast machines.
Finally, some fun finds:
// THIS IS NOT MINE!!! I DID NOT DO THIS!!! I PLEDGE TO RIP THE HEART
// FROM THE INDIVIDUAL WHO DID! THIS IS JUST A WORSE VERSION OF GOTO!
// I WILL CHANGE THIS AT SOME LATER DATE - DON'T MESS WITH IT, DON'T CHANGE
// IT! THIS MEANS YOU!!! Thanks for your support - MJM
static jmp_buf resetmark;
// [...]
// THIS SUCKS!!! THIS WAS NOT MY IDEA - I REALIZE THAT THIS TAKES UP TO MUCH
// SPACE, BUT BLAM MIKE R.
animtype RioHead[15] =
{ //...
}The former comment applies to a global symbol used to execute longjmp() in awg9.c, the latter to some statically initialized arrays in riohead.c. Looks like developer frustration is a given on pretty much any software project. 😉
Great, now what?
Now, how do we go about figuring out if any of this code is actually present in F15 II, and if so, where? Fortunately, I recently developed some tooling for extracting routine signatures from binaries and locating them in other binaries, which I orignally used to see if any work I did for F-15 II’s START.EXE would carry over to EGAME.EXE. This time, I will use it to search for bits of F14.EXE in EGAME.EXE. I am a bit lucky here. Despite Fleet Defender’s relatively late release date of 1994, the game does not appear to use protected mode, or much 32-bit code either (except for a few minor sections in assembly files, still in real mode). It’s kind of surprising given that by that was the year Doom II came out, but for better or worse, Fleet Defender seems to run 16bit real mode code. If it had been rewritten to use protected mode, the code would likely be mostly useless to me.
The first thing I need to do then is build the code. There is a DOS-era makefile included in the source tree, but I ended up making my own to have a bit more control over the build process and integrate well with my tooling for wrapping MS C. Incidentally, Fleet Defender seems to have been compiled by MSC 7.0, but I need to build it with MSC 5.1 to have the code matching F-15 II as closely as possible. For that, I had to do a couple tweaks to the code, mostly with large, statically initialized arrays defined within functions that MSC 5.1 did not like, but it was a simple matter of moving them up to global scope. I also had to exclude some C files which used inline assembly which is not supported in MSC 5.1, but there were only a few and they contained a minuscule amount of code.
Unfortunately, my tooling cannot (yet) parse object files in the OMF format, only EXEs, so I also need to link it into an executable. Here we come to an unfortunate realization: the code is incomplete. Most of the missing functions are declared in the header library.h:
/*
╔═══════════════════════════════════════════════════════════════════════╗
║******************** MPS Labs Graphic Library *********************║
╟───────────────────────────────────────────────────────────────────────╢
║ File: Library.h ║
║ ║
║ Auth: David McKibbin ║
║ ║
║ Edit: dtm July 20, 1992 1:18 pm ║
║ ║
║ Note: HEADER definitions for MPSLIB?.LIB ║
║ ║
╟───────────────────────────────────────────────────────────────────────╢
║ Copyright (c) 1991 by MicroProse Software, All Rights Reserved. ║
╚═══════════════════════════════════════════════════════════════════════╝
*/
// [...]
/*************** USER/RESIDENT "C" prototypes ***************/
extern int OpenFile (char *file, int attrib); /* fileio.c */
extern void CloseFile (int fh);
extern int CreateFile (char *file);
// [...]
/*************** USER/RESIDENT "ASM" prototypes ***************/
extern void far InitSGF (char *palette); /* lzwio.asm */
extern void far InitSDF (void);
extern void far ReadSDF (char far *buffer, int count);
// [...]
/*************** Graphic Library prototypes ***************/
extern void far AddLine (char *RowBuff, int page, int x, int y, int count);
extern UWORD far AllocGraphicPage (int page);
extern UWORD far AvailSysMem (void);
// [...]
/*************** MISC Library prototypes ***************/
extern int far IsKey (void);
extern int far GetKey (void);
extern int far EchoGetKey (void);
// [...]
/*************** SOUND Library prototypes ***************/
extern int far SndSysSetup();
extern int far SndSounds();
extern void far SndShutdown();This looks quite familiar. I am pretty convinced that the “user/resident” declarations correspond to a bunch of common utility routines that I found while reconstructing F15SE2’s START.EXE, while the “Graphic/MISC/SOUND” prototypes are functions from the runtime-loaded overlay drivers, calls to which I can see all over the place in F15SE2. This library header file is a boon in itself, as I still haven’t figured out the arguments and/or the purpose of many of those functions. Establishing the relationship between the declarations and the routines in the code will not be straightforward, since the header file seems to have the graphics routines (which are the most numerous and important) listed in alphabetical order, but I’m sure I can figure it out eventually.
It actually makes sense for the library code to be missing; it was probably part of a different source tree, shared between multiple game projects at MPS, and one that the Fleet Defender devs did not need to mess with (most of the time?). For my needs, I just stubbed all of them out and moved on. This still left me with a bunch of unresolved symbols for some routines and data from the linker. I don’t know what these are and what happened to them, but they are also missing from the source code, so it’s stub time again:
// data
char far *TILECOLORS;
int Fencer;
int ag_msg;
char far *GREYBUF;
int MenuSpr1;
int *MenuSpr2, *MenuSpr3;
int *GroundObjectScale;
char far *GROUPCOLORS;
char **TRANSGREYPTRS;
char **GroundObjectNames;
long DESIGNATED_X;
long DESIGNATED_Y;
int SNDDETAIL;
char far *STAMPCOLORS;
int ACQ_PX, ACQ_PY;
int MSGDETAIL;
char far ***GROUPBUF;
volatile int MouseX;
volatile int MouseY;
volatile int Button;
int ag_msg_cnt;
char far ***STAMPBUF;
int MISSILETRAIL;
char **HRM_RANGES;
int WORLDDETAIL;
char far ***TILEBUF;
UWORD far *CrtMask;
int InFriendly;
int _acrtused; // workaround for unresolved symbol when linking without libc
// routines
long labs(long X) {}
void FlyGroupLine() {}
void FlyTileLine() {}
void FlyGroupine() {}
void FlyStampLine() {}
void FlyLine() {}
void LockAG () {}
char* strupr(const char* s) { return NULL; }
int isprint(char c) { return 0; }
void SetJoysticks (int i) {}
void ag_err_msgs () {}
void ChangeWeather() {}
void clip_rotmap() {}
char *WorldName() { return NULL; }
void OverlaySequencePoints() {}
void S2MapLine() {}
void _fmemcpy() {}
void rotate_pt() {}
void SMapLine() {}
void _fstrcat() {}
void _fstrcpy() {}
void GetRidOfKeyJoy() {}
void Draw3dGroundObject(int NUM, int X, int Y, int Z, int BlowUpFlags) {}
void Draw3dGroundObjects() {}That, along with the library function stubs got rid of the unresolved symbols. But the linker failed with a “fixup overflow” error in the data segment, meaning the size of the data exceeded 64k, and I’m building the code with the medium memory model (/AM), so there’s just one data segment. I was probably overly greedy in trying to cram as many of the source files into the exe as possible, and I could remove some, but I actually ended up using the /NOD[EFAULTLIB] linker option to avoid linking in the standard C library and its associated data, because I don’t actually need the resulting executable to be runnable. Doing that, however meant that I also had to stub out a bunch of libc routines to get it to link, but finally link it did, and I had my coveted f14.exe, built with MSC 5.1 with default flags.
The fact that the executable doesn’t run or have libc, poses a different problem. Because there’s no START from crt0, the entrypoint for the executable is fixed at 0:0. I need to walk the code with my mzmap tool to figure out where the routines begin and end, so I can extract their signatures for comparison with F15SE2, but if the main entrypoint is broken and control doesn’t really go anywhere, the walker will fail.
However, I do know where the routines are located, because the linker output a .MAP file when it created f14.exe, and the map file lists all public routines along with their entrypoint addresses. I implemented a new option to mzmap; --linkmap lets it ingest the linker map file as a set of “hints” for routine entrypoints, segment locations and data references as well. That adds a useful feature to mzmap, and while walking f14.exe, I found and fixed a bunch of pretty serious bugs in the walker, so I’m pretty happy that I took the effort to do so. At the end, I had a map file compatible with my tooling, with most of the f14.exe routine boundaries discovered.
Okay, now with the F-14 executable built and the map generated, time to extract the signatures. Or is it? My mzdup tool scans for duplicate routines given two exes and their maps. But I want it to be more flexible, so I ended splitting up the job, implementing mzsig. Given an exe and its map, plus some options to say what size of routines it should ignore, it will extract the signatures and save them to a file. This also lets me combine signatures from more than one executable in one signature file, which is pretty handy – remember that some F15SE2 routines have been built with debug flags enabled, so I need to build and extract signatures for at least two versions of f14.exe, one with /Zi and one without it.
The new version of mzdup ingests the signature file along with a exe/mapfile pair to look for matches in. I also changed the edit distance cutoff threshold from a hard value shared between all routines to a configurable ratio of the routine size, meaning bigger routines can have more differing instructions than smaller ones and still be considered matches.
Give us the damn results already
Well, okay, since you ask so nicely:
ninja@thinkpad:f15se2-re$ mzdup --verbose --minsize 5 --maxdist 20 ../f14/f14-ot.sig ../ida/egame.exe map/egame.map [...] Unable to find duplicate of _NCloudLine, 61 instructions, max distance 12 Unable to find duplicate of _NCloudLine2, 68 instructions, max distance 13 Unable to find duplicate of _NCloudLine3, 56 instructions, max distance 11 Unable to find duplicate of routine_1035, 60 instructions, max distance 12 Unable to find duplicate of _MakeBspDrawList, 16 instructions, max distance 3 Found duplicate of routine _TrgMul (9 instructions): routine_161/1000:3b2f/013b2f (9 instructions) with distance 1 Unable to find duplicate of _DPTRGMUL, 6 instructions, max distance 1 Unable to find duplicate of _MUL256DIV, 24 instructions, max distance 4 Found duplicate of routine _Icos (5 instructions): routine_158/1000:3b96/013b96 (5 instructions) with distance 1 Found duplicate of routine _Isin (5 instructions): routine_158/1000:3b96/013b96 (5 instructions) with distance 1 WARNING: Routine routine_158/1000:3b96/013b96 is a duplicate of _Isin and _Icos with equal distance Unable to find duplicate of routine_1037, 9 instructions, max distance 1 Unable to find duplicate of _TransScaleLine, 58 instructions, max distance 11 Unable to find duplicate of _TacTransScaleLine, 31 instructions, max distance 6 Processed 944 signatures, ignored 0 as too short Tried to find matches for 400 target exe routines (22524 instructions, 100%) Found 22 (unique: 20) matching routines (743 instructions,3% ) Unable to find 378 matching routines (97%) WARNING: Some routines were found as duplicates of more than one routine. This is possible, but unlikely. Try using a longer minimum routine size and/or lower distance threshold to avoid false positives. Saving code map (routines = 400) to map/egame.map.dup, reversing relocation by 0x1000
It was able to find matches for 20 unique routines from f14.exe built with default flags, which constitutes a meager 3% of F15SE2’s egame.exe. I tried with a version built with /Zi and it was similar. Increasing the maximum acceptable edit distance ratio does not seem to help much.
Great, so all that for nothing?
Depends on how you look at it. For me, it’s really all about the journey, and I could not have moved on with the reconstruction without settling this issue. Also, I’m still convinced the Fleet Defender codebase will come in useful, and getting familiar with it gave me a new source of insight and inspiration.
I also fixed a bunch of serious bugs in my tooling, further enabling it to work with more than just F15SE2, and implemented useful features which will become useful again when we decide to branch out to F-19 and F-117 in the future. All of this stuff is available in mzretools v1.0.0.
Additionally, I am still not convinced my signature search is a 100% reliable. For one, I’m reusing an enum for the instruction class (mov/sub/jump/…) which is an abstract representation of the opcode, and it retains some differences between instructions that I really should make more vague, like the fact that it uselessly distinguishes between call/jmp and their far equivalents. For memory operands, it remembers if an instruction used an 8-, or a 16bit offset for a data reference, which can also introduce differences into otherwise equivalent code. There could also be some serious bugs that make it miss matches for some other reason, who knows. Something must be working because when I search for signatures from F-19’s EGAME.EXE in F15SE2, it’s able to match 38% of the code with a threshold of 10% difference. When I increase the threshold to 30%, the matched code jumps up to 53%, but what I’m saying is that there still could be breakthroughs made on the tooling which would let it find more matches for f14.exe.
Finally, the leaked code itself introduces problems. Parts of it are disabled with #ifdef YEP/#endif - WTF??? Parts are commented out, with no way of telling whether they have been that way forever, or just disabled somewhere along the way. More work needs to be done to clean it up, generate more signatures and try again with different variants of the build.
With that said and done, I really should get to reconstructing egame.exe. I won’t make any promises this time, but the day is getting close…
