FLEXlm, FLEXcrypt

Introduction

This article was inspired by the tutorial by SiuL+Hacky on how to crack XprismPro 1.0. The above program ran on Linux, but my target ran on Windows 95/NT. The aim of this tutorial is to expand on some of the ideas in the first tutorial and to detail the differences encountered on the PC.

Tools required

FLEXlm programmers kit: http://www.flexlm.com
Hiew v5.66.
IDA v3.75.
W32Dasm v8.9.

Essay

1. Get hold of the flexlm programmers kit from www.flexlm.com. It needs a key to decrypt the file. S+H details how to crack it, or ask FLEXlm. Once you've got it installed the main files of interest are :-

lm_code.h :- This is where you want to put in all the target key information.
lm_client.h :- Contains useful function prototypes and error codes.
GenLic32.exe :- this program checks the keys and generates licenses for you.

I'd also recommend reading the HTML manual.

2. Find the easy information using lc_init().

Using W32Dasm, decompile your target application and the vendor daemon DLL it uses (e.g. lmgr326a.dll). Load up the target EXE and set break points wherever the lc_init() function is called. Run your target until it breaks. Looking at the prototype for lc_init in lm_client.h we see :-

lm_extern API_ENTRY lc_init lm_args((LM_HANDLE_PTR job, 
                                     LM_CHAR_PTR vendor_id, 
                                     VENDORCODE_PTR vendor_key,
                                     LM_HANDLE_PTR_PTR job_id));

job will be NULL as it's the first one.
job_id will be a pointer for the job structure to be filled by the function.

The things of interest for us are the Vendor_id and Vendor_key. Vendor_id is just a text string.

Again looking in lm_client.h we see :-

typedef struct vendorcode5 {
                            short type;    /* Type of structure */
                            unsigned long data[2]; /* 64-bit code */
                            unsigned long keys[4]; 
                            short flexlm_version;
                            short flexlm_revision;
                            char flexlm_patch[2];
                            char behavior_ver[LM_MAX_BEH_VER + 1];
                          } VENDORCODE5,  FAR *VENDORCODE_PTR;

#define LM_MAX_BEH_VER  "06.0"

In the above structure, data[0] = encryption seed 1 XORed with vendor key 5, data[1] = encryption seed 2 XORed with vendor key 5, keys[0..3] = vendor keys 1 to 4, behavior_ver[] = string containing FLEXlm version ( in this case = "06.0"). So all the stuff above will be pushed onto the stack prior to calling lc_init(). Looking at the disassembly we see :-

MOV ECX, DWORD PTR [EDI]
PUSH EDI
PUSH 00528DC8 <-- pointer to code structure.

* Possible StringData Ref from Data Obj ->"VENDOR"
PUSH 00525AE4 <-- pointer to vendor ID string.
PUSH ECX

* Reference To: LMGR326A.lc_init, Ord:0034h
CALL 004A9BD8

OK, so we've got the vendor ID string (in this case "VENDOR"), looking at memory address 00528DC8 we see the code structure :-

[00528DC8] - 00000004 ....
[00528DCC] - ab5e32e5 .?^. <-- seed 1 XORed with key5
[00528DD0] - 7bc6313d =1.{ <-- seed 2 XORed with key5
[00528DD4] - fc62965d ].l. <-- key 1
[00528DD8] - 853df75c \7=. <-- key 2
[00528DDC] - 2f324f23 #O:. <-- key 3
[00528DE0] - 1133e43b ;.3. <-- key 4
[00528DE4] - 00000006 .... <-- version and revision
[00528DE8] - 36300069 i.06 <-- patch and behaviour_ver
[00528DEC] - 0000302e .0.. <--

So we've got 4 keys and two XORed encryption keys.

4. Find your feature names. You need these to make up a working license file. Clear the breakpoints on lc_init() function calls and set breaks on calls to lc_checkout. Looking at lm_client.h again we see :-

lm_extern API_ENTRY lc_checkout lm_args((LM_HANDLE_PTR job,
const LM_CHAR_PTR feature,
const LM_CHAR_PTR version,
int nlic,
int flag,
const VENDORCODE_PTR key,
int dup));

The call looks like :-

PUSH 00004000
PUSH 00528DC8 <-- code structure
PUSH 00000000
LEA EAX, DWORD PTR [ESP+10]
PUSH 00000001
PUSH EAX <-- version
PUSH ECX <-- feature name
PUSH EDX

* Reference To: LMGR326A.lc_checkout, Ord:0022h
CALL 004A9BDE

OK, so we've got a feature name. Now have a look round and look for similar names, your target may use more than one.

5. Make your first license.dat file. Modify the lm_code.h file to contain the encryption seeds, and vendor keys 1 to 4. Set vendor key 5 to 0 for now. Run genlic32.exe, if it throws up an error message then you haven't typed in the keys correctly in lm_code.h. Enter the feature name as found above. Click on 'permanent' and 'run anywhere' so you don't need a server daemon running. Click on 'make license', fill in the filename license.dat and click 'Make license'. Have a look at the licence file, it'll look something like :-

FEATURE feature <- feature name
VENDOR <- vendor ID
1.000 <- version
permanent <- never expires
uncounted <- no need for a server
0AF0103F59E2 <- file checksum
HOSTID=ANY <- run on any machine

6. Finding vendor key 5. Like S+H, I didn't believe vendor key 5 could be unknown by the daemon, it had to be made on the fly. It appears to be made up of all 4 vendor keys and the vendor name. So it could be a checksum for the vendor info?. FLEXlm provides the 5 keys based on your vendor name, so they'll want to checksum it somehow. If you get keys 1 to 4 or the vendor id wrong in lm_code.h then genlic32.exe won't like it.

OK this bit is more tricky, but keep at it and you'll get there. Start W32Dasm again and load your target ready to read your nice new license file. Break on lc_checkout in your target EXE. Load up the daemon DLL (in the active DLLs window double-click on the daemon DLL). Start single-stepping through. There's lots of calls to undocumented functions, but keep stepping into them. Here's the edited highlights of how it went for me :-

Read and check the licence.dat file :-

l_validversion(), l_compare_version() check the version in the license file
l_date(),l_extract_date() extract the date from the checksum
( permanent gets converted into 1-jan-0 )
l_start_ok() start date is OK
l_host() OK 'cos it's ANY

Then we got into some functions with lots of XORs. This looks good as we want to XOR our two encryption seeds. We can see the keys 1,2,3 and 4 and the vendor ID getting read and XORed. Then we hit some code :-

CALL 10021160 <-- Get vendor key 5 in EBX.
ADD ESP, 0000000C
MOV EAX, DWORD PTR [EDI+04] <- Seed 1 from license.dat.
XOR EAX, EBX <-- Seed 1 XOR key5.
LEA ECX, DWORD PTR [EBP-58]
PUSH ECX
MOV DWORD PTR [EBP-54], EAX <-- Store Seed 1.
MOV EAX, DWORD PTR [EDI+08] <-- Seed 2 from license.dat.
MOV EDI, DWORD PTR [EBP+0C]
XOR EAX, EBX <-- Seed 2 XOR key5.
MOV DWORD PTR [EBP-50], EAX <-- Store Seed 2.
LEA EAX, DWORD PTR [EDI+54]
PUSH EAX
PUSH ESI

* Reference To: LMGR326A.l_extract_date
CALL 10004B98

So we've just got key5, and the XORed seeds1 and 2.

7. Make your final license.dat. In lm_code.h, replace your two seeds and key5. Run genlic32.exe and make your completed licence file.

Final Notes

As Siul+Hacky mentioned, the only security here is that of secrecy. Thanks to Siul+ for the initial hard work.

One of the first stages in analysing FLEXlm is to crack the encryption used to package it. FLEXcrypt is produced by Globetrotter Software Inc. and seems to share may commonalities with FLEXlm. This document analyses the FLEXcrypt en/de-cryption program to a stage that allows decryption of any FLEXcrypted package. This in turn leads into some interesting areas worthy of further investigation.

For a full explanation you'll need to download two files: flexlm5.12 and flexlm6.1. The site also contains many other FLEXcrypted files to practise on.

FLEXcrypt uses a rolling XOR based scheme to encrypt a file. It uses the same key on 8 bytes, recalculates the key, then continues through the file. The key is in some way based on the passkey you must find for correct decryption. The passkey is of the format xxxx-xxxx-xxxx-xxxx-nn where the x are hexadecimal numbers and nn ranges from 00 to 99 decimal. The hex numbers are in some way related to your PC ( hard disc number / network card number ).

The nn specifies which of 100 keys was used to encrypt the file. Only one nn value decrypts the file correctly. There is a table which stores 100 passkey related values in the decryption executable. There are many versions of FLEXcrypt available, but there is basically a 16bit and 32bit. Both use the same encryption method. 32bit introduces a file header with some sort of file/passkey check data included.

All FLEXcrypt files may easily be decrypted correctly using the 32bit decryption program. The decryption program is packaged with the install files and is usually called something like cryptwin.exe.

Command line options for decrypt.exe :-

-p xxxx-xxxx-xxxx-xxxx-nn : the passkey

-t Z *.fc : decrypt all .fc extension files to files with .Z extension

-d n : debug mode, n can take values 1,2,3 each displaying differing formats of debug info

Stage 1 : how to decrypt the files

Run the install.exe from the downloaded FLEXlm 5.12. This will extract all the files needed for the install. Run setup.exe

You'll be asked for the FLEXlm decryption key in the format xxxx-xxxx-xxxx-xxxx-xx. Cancel the install. We see a setup.ins file : InstallShield is here. Use isdcc to decompile the install script. Have a look for the text we saw earlier, xxxx-xxxx-xxxx-xxxx-xx, and we see :-

AskText("Please enter in your decryption key. It should be of the form xxxx-xxxx-xxxx-xxxx-xx . If you do not have this key, you can obtain it by calling Globetrotter Software at (408) 370-2800. If you do not have a key at this time, please press cancel to end t", "xxxx-xxxx-xxxx-xxxx-xx", string15);

Sprintf(lString1, "%scryptwin.exe", SRCDIR);
Sprintf(lString2, "Decrypting files, this may take some time");
function11(lString2, 1);
Sprintf(lString0, "-w %s  -p %s -t Z *.fc", SRCDIR, string15);

This decodes to :-

cryptwin.exe -p xxxx-xxxx-xxxx-xxxx-xx -t Z *.fc (where xxxx-xxxx-xxxx-xxxx-xx would be the numbers you type).

-p option is password, -t is convert *.FC into *.Z type.

So rename cryptw~1.exe to it's full title, cryptwin.exe, and try cryptwin.exe -p 0000-0000-0000-0000-00 -t Z *.fc. We end up with zero length Z files. Not too good. Cryptwin is 16-bit, so let's use IDA to dissassemble it. At this moment I'm interested in command line options so let's have a look.

Searching for -p gives us a function at loc_8B6_6C which looks like a command line parser. It seems to also accept -x -o -i -e and -d options. What's -d? debug? There's references to DEBUG in the exe, let's try it. A bit of experimenting shows there are 3 debug levels, with varying outputs, 1 writes a log file, 2 displays on the screen, 3 uses pop-up windows.

Stage 2 : Get a valid passkey

32bit's easier to debug than 16bit so let's abandon FLEXlm 5.12 for now and try FLEXlm 6.1. Clear the Windows\Temp directory and run the FLEXlm 6.1 install. When prompted for the key press cancel and copy all the files into another work directory.

Let's try cryptwin.exe -d 3 -p 0000-0000-0000-0000-00 -t Z *.fc again.

Again we end up with a zero length Z file. Dissasemble cryptwin.exe using W32Dasm. Load the program with the command line options -p 0000-0000-0000-0000-00 -t Z *.fc. Put a break-point on the reference to "%.4d-%.4d-%.4d-%.4d". Run to the break point then single step.

* Possible StringData Ref from Data Obj ->"%.4d-%.4d-%.4d-%.4d"

:00405E2F PUSH 0042B070
:00405E34 MOV ESI, 0042F360
:00405E39 PUSH ESI
:00405E3A CALL 0041E2E0
:00405E3F ADD ESP, 00000018

In this case there's a call to 0041E2E0 which returns ESI pointing to the processed string, which looks something like :-

[esi+00000000] - 7375
[esi+00000004] - -553
[esi+00000008] - 2-55
[esi+0000000C] - 21-5
[esi+00000010] - 871

This string appears to be related to machine specific items such as hard disk number and network card number. This string is the first four numbers of a potentially valid passkey. The last two digits are the key number used in the test. In this case 00.

OK, we have a key to try, it's 7375-5532-5521-5871-00. So let's try cryptwin.exe -d 3 -p 7375-5532-5521-5871-00 -t Z *.fc. Again, an error message and a zero file. We need to repeat the above process using keys 00 to 99 as the last two digits in the password. This would be a very slow process using W32Dasm. So modify the cryptwin program to generate a file with a name of each passkey.

Stage 2a: auto generate the passkeys

Copy your original cryptwin.exe to something else, I used dcrypt.exe. Using your favourite HEX editor, modify dcrypt.exe as follows :-

1). Just after ESI points to the passkey string, jump to some unused code space, replace :-

:00405E3F ADD ESP, 00000018
:00405E42 PUSH EDI
:00405E43 CALL 00405E54

With :-

:00405E3F ADD ESP, 00000018
:00405E42 JMP 00401828 (E9E1B9FFFF)

2). At this unused space, 00401828, replace the existing code with :-

:00401828 6A00 PUSH 00000000
:0040182A 6880000000 PUSH 00000080
:0040182F 6A01 PUSH 00000001
:00401831 6A00 PUSH 00000000
:00401833 6A03 PUSH 00000003
:00401835 68000000C0 PUSH C0000000
:0040183A 8BC6 MOV EAX, ESI
:0040183C 50 PUSH EAX

* Reference To: KERNEL32.CreateFileA, Ord:0030h

:0040183D E8CC760200 CALL 00428F0E <-- create the file.
:00401842 50 PUSH EAX

* Reference To: KERNEL32.CloseHandle, Ord:0017h

:00401843 FF15BC324300 CALL DWORD PTR [004332BC] <-- close the file handle.

* Reference To: KERNEL32.ExitProcess, Ord:006Ah

:00401849 FF158C324300 CALL DWORD PTR [0043328C] <-- end the program.

Now run the dcrypt.exe with the command line options -p 0000-0000-0000-0000-00 -t Z *.fc

You should get a file of name 7375-5532-5521-5871. Now write 2 batch files to run through 00 to 99 and store the results :-

test2.bat:
call test.bat 00
call test.bat 01 .... to .....
call test.bat 98
call test.bat 99

test.bat:
dcrypt -p 0000-0000-0000-0000-%1 -t cab data1.fc
rename ????-????-????-???? ????-????-????-????-%1

This gives 100 files with potential passkeys to try. Note that these passkeys will differ on different machines. So now we want to try each one and see when we get a valid result. DIR all the files into a text file and run a macro to get something like cryptwin -p 7375-5532-5521-5871-00 -t cab data1.fc, pause ... etc for each key. I checked the size of the decrypted file after each try. It's non-zero when it's been decrypted correctly. (further work: you may be able to use the debug error level to tell you when it's worked). You'll eventually end up with the correct key for your machine. Once you've got it you can then use it to decrypt all the fc files.

Stage 3: Repackage so you don't need the key again.

It would be wise to repackage the FLEXlm 6.1 so you don't need the key again. Delete the cryptwin.exe and the *.Z files then try installing again. You can now enter any passkey and FLEXlm 6.1 will install.

Stage 4: Try decrypting another package.

Let's try the passkey we got above with the 16 bit FLEXlm 5.12. No good. That's because the 16bit cryptwin.exe has a different set of keys stored in it's executable. OK, so let's try decrypting the 16bit FC files using the 32bit cryptwin. This may work as we know the key for the 32bit decrypter. Well, we get some Z files, no error messages, but the files are corrupt. Open up a Z file with your favourite hex editor and have a look. Hmmm, lots of repeating 8 byte patterns.

Looks like we've decrypted the file but with the wrong start value.

Stage 4a: A diversion ... partial analysis of file decryption

Time to look at how crypwin.exe decrypts the fc file. Open up the disassembled cryptwin.exe from FLEXlm 6.1, and look at 00402D1D (Some dissassembly has been removed) :-

1). Read the file and see if it contains the "FLEXcrypt Copyright" signature. If it does then this is a new 32bit generated file, else it's a 16bit file.

:00402D1D PUSH EBP
:00402D58 PUSH ECX
:00402D59 CALL 0041D430 <- fread "FlexCryp" from the file.

* Possible StringData Ref from Data Obj ->"FLEXcrypt Copyright (C) 1990-1997, "Globetrotter Software, Inc."

:00402D66 PUSH 0042AE70
:00402D6B PUSH ECX
:00402D6C CALL 0041F240 <- Compare the 2 strings.
:00402D71 ADD ESP, 0000000C
:00402D74 TEST EAX, EAX
:00402D76 JNZ 00402DF1 <- Jump if it's 16bit straight to decrypt file.

2). This is a 32bit file with a valid header. The analysis of this next section needs some more work, it may hold more clues. The code reads some sort of checksum from the file header. The first 8 bytes of data are decrypted. The data is then compared with something and a pass/fail is determined.

:00402D78 PUSH [EBP+08]
:00402D7B PUSH EBX
:00402D7C PUSH 00000068 <- 104 bytes of header data.
:00402D7E LEA ECX, DWORD PTR [EBP+FFFFFF24]
:00402D84 PUSH ECX
:00402D85 CALL 0041D430 <- read the header data.
:00402D8A ADD ESP, 00000010
:00402D8D LEA EDX, DWORD PTR [EBP-44]
:00402D90 PUSH [EBP+08]
:00402D93 PUSH EBX
:00402D94 PUSH 00000008
:00402D96 PUSH EDX
:00402D97 CALL 0041D430 <- read the first 8 bytes of data.
:00402D9C ADD ESP, 00000010
:00402D9F LEA ECX, DWORD PTR [EBP-1C]
:00402DA2 LEA EDX, DWORD PTR [EBP-44]
:00402DA5 LEA EAX, DWORD PTR [EBP-24]
:00402DA8 PUSH ECX
:00402DA9 PUSH EDX
:00402DAA PUSH EAX
:00402DAB CALL 004082C8 <- get an 8 byte key?.
:00402DB0 ADD ESP, 0000000C
:00402DB3 MOV ECX, 000000FF
:00402DB8 MOV EAX, DWORD PTR [EBP+14]
:00402DBB SUB ECX, EAX
:00402DBD PUSH ECX
:00402DBE PUSH EAX
:00402DBF PUSH EAX
:00402DC0 LEA ECX, DWORD PTR [EBP-38]

* Possible StringData Ref from Data Obj ->"%02x%03d%03d"

:00402DC3 PUSH 0042AE60
:00402DC8 PUSH ECX
:00402DC9 CALL 0041E2E0 <- sprintf first 8 bytes.
:00402DCE ADD ESP, 00000014
:00402DD1 LEA ECX, DWORD PTR [EBP-38]
:00402DD4 LEA EDX, DWORD PTR [EBP-1C]
:00402DD7 PUSH 00000008
:00402DD9 PUSH ECX
:00402DDA PUSH EDX
:00402DDB CALL 0041F200 <- string compare 8 bytes.
:00402DE0 ADD ESP, 0000000C
:00402DE3 TEST EAX, EAX
:00402DE5 JZ 00402DF8 <- passkey OK, continue to decrypt the file.
:00402DE7 MOV EAX, 00001001
:00402DEC JMP 00402EA6 <- fail, abort with error code.

3). Decrypt the file, 8 bytes at a time.

:00402DF1 MOV [EBP-08], 00000001
:00402E73 LEA EAX, DWORD PTR [EBP-1C]
:00402E76 LEA ECX, DWORD PTR [EBP-10]
:00402E79 PUSH EAX
:00402E7A LEA EDX, DWORD PTR [EBP-24]
:00402E7D PUSH ECX
:00402E7E PUSH EDX
:00402E7F CALL 004082C8 <- get an 8 byte key?.
:00402E84 ADD ESP, 0000000C
:00402E87 XOR EAX, EAX
:00402E89 MOV CL, BYTE PTR [EBP+EAX-2C] <- XOR 8 bytes.
:00402E8D XOR BYTE PTR [EBP+EAX-1C], CL <- with key.
:00402E91 INC EAX
:00402E92 CMP EAX, 00000008
:00402E95 JL 00402E89 <- loop 8 times.

Stage 5: XOR the incorrectly decrypted file

As mentioned above, it's as if we've used the wrong start point to decrypt the FC file. And from the analysis above we can see it's an XOR decryption. So how about finding some likely looking zeros in the file. XOR of zero with a key is zero. If we XOR the file with the value stored at a zero we may get a working file. Open up the html.Z that comes with FLEXlm 5.12 with your HEX editor. I spy lots of repeats at 40hex, in my case, d361f5a2006ebb12. Let's try XORing the whole file with d361f5a2006ebb12. I wrote a little C program to do this, (main parts only shown) :-

void main()
{
FILE *ReadFile, *WriteFile;
unsigned char xor_table[8]={0xd3,0x61,0xf5,0xa2,0x00,0x6e,0xbb,0x12};
ReadFile=fopen("test.x","rb");     /* Open up the project file to read
*/
WriteFile=fopen("test.z","wb+");   /* Open up the new project file to
write */
init=0;
i=0;
while (feof(ReadFile) == 0) /* Repeat until the end of the read file
is reached */
{  
  c=fgetc(ReadFile);
  if (init==1)
    c^=xor_table[i];
  i++;
  i&=7;
  init =1;
  fputc(c,WriteFile);
}
fclose(ReadFile);
fclose(WriteFile);
}

Run this on the files and they look like they should work, but they're still corrupt.

Stage 6: correct the first 8 bytes of the file

The problem is the first 8 bytes of the data are not correct. Further work: I think the first 8 bytes have something to do with the initial decrypt value. This doesn't pose a problem as FLEXcrypt is typically used to encrypt files with known headers. For an installshield Z file the first 8 bytes are 13,5D,65,8C,3A,01,02,00. For an installshield CAB file the first 8 bytes are 49,53,63,28,04,00,00,01. So I modified my XOR code to skip the first 8 bytes and replace them with the correct header :-

void main()
{
FILE *ReadFile, *WriteFile;
unsigned char xor_table[8]={0xd3,0x61,0xf5,0xa2,0x00,0x6e,0xbb,0x12};
unsigned char z_header[8] ={0x13,0x5D,0x65,0x8C,0x3A,0x01,0x02,0x00};
ReadFile=fopen("test.x","rb");     /* Open up the project file to read
*/
WriteFile=fopen("test.z","wb+");   /* Open up the new project file to
write */
/* Skip the first 8 bytes and write the Z file header */
for (i=0;i<8;i++)
{
  c=fgetc(ReadFile);
  fputc(z_header[i],WriteFile);
}
init=0;
i=0;
while (feof(ReadFile) == 0) /* Repeat until the end of the read file
is reached */
{  
  c=fgetc(ReadFile);
  if (init==1)
    c^=xor_table[i];
  i++;
  i&=7;
  init =1;
  fputc(c,WriteFile);
}
fclose(ReadFile);
fclose(WriteFile);
}

Run this and your encrypted files are now decrypted. As before, remove the cryptwin.exe and the FC files and your installation will now accept any passkey.

FLEXcrypt further work

Now we can decrypt ANY FLEXcrypted file. This includes the FLEXcrypt developers kit. So download this, decrypt it and we get a useful html manual and some source code. Have a look a mycode.c and cryptkit.h. It appears that the kit uses FLEXlm license data to encrypt the keys stored in the code. There's an array fc_keytab[256] stored in the cryptwin.exe which contains the 8 bytes keys to use XORed with vendorkey5. As we know from FLEXlm cracking, vendorkey5 is the hard one to find. And we also know it's calculated on the fly from the other vendor codes.

So maybe cryptwin.exe has the necessary license data stored in it's EXE?. Using your favourite HEX editor open up cryptwin.exe from FleLm 6.1. Have a look at offset 002A5C0, I think this is the fc_keytab table. Looking at the rest of the code it seems to have lots of FLEXlm code embedded in it. And we've cracked this before. So maybe, by analysis of the cryptwin.exe and the FC file it may be possible to determine which key will work, without all the effort above.

Conclusion

The above document shows how we can decrypt ANY FLEXcrypted file. Further work may show the essence of FLEXcrypt and it's relationship with FLEXlm. Thanks to the people out there who continue to reveal the light.

pilgrim