For a project involving a STM8S uC I have installed a C compiler that is protected with flexlm 9.5. I obtained a free license that allows code generation up to 16K. I can live with that, but I want to be able to install the software on a lot of PCs without having to mail the company my HOSTID and disk serial number for every new computer, or when a hdd/net/etc gets replaced.

Being unfamiliar with reversing, I started searching for the code part in the compiler executable that gets the disk serial number (a call to GetVolumeInformation in COMCTL32). It was very obvious that only the serialNb gets saved thus it must be the right location.
However when setting a breakpoint in that location, the executable never stops. I removed the license, reentered it in the License Finder but no breaking....

So my first guess was it might be that the disk serial number is not used when validating the license. Also in the limited license file, only a hostid is specified.
I noticed that netapi32.dll was imported, presumably to use NetBios to enumerate all the mac addresses. But again, putting a bp on the relevant fucntion call(s) didn't do anything.

So my question now is, how to proceed? Or am I missing something so that the functions never trigger? This is way outside my skills so some help would be appreciated.

(I read about searching and bp'ing for seed value 3D4DA1D6h but again it never breaks on them. Also, I wouldn't know how to generate a license from the seeds without the 9.5 sdk...)

Thanks

I think if you've never reversed before, you may want to start with normal targets. Otherwise, this will end up being a "roundabout" crack request. You wouldn't understand anything and having people getting it cracked for you is no fun.

For a beginner, Flexlm is hardly the way to go.

Have Phun

Thanks for your reply. You're right about starting with something easy, however I really want to end up with a working compiler.
Even if it does not lead me to a solution, I would appreciate it if someone could explain why the functions I mentioned are not (or don't seem to be) called.

They might be called, but not via normal means. FlexLM has put quite a few red herrings in there code over time, so you really needed to start when they did to keep up with all the changes and know whats fake and whats real.

You might find however that all this is moot - I've seen compiler vendors do all this checking with obfuscated code, useless code (calling functions and not caring about the result) and lots of time dependancies to check if the program is being debugged. However, at the end of it all they just set a "license is valid" flag and thats it, so you force the flag and it doesn't care about the license no more. In fact you can remove all license checking code as it only slows the program startup.

I'm not saying yours is going to be the same, but before you dive into the deep end and try to work out/fake the license system, there may be an easier route to take if you look out of the box.

CPCT,

There are more than a few free C compilers, but just for giggles try installing your compiler on another computer using the license you have, the disk serial number may just be a smokescreen.

SiGiNT

I had a tiny bit of success. One document mentioned the use of the flexlm error codes. I noticed that just after the License finder appears and is "canceled" the returning eax value was FFFFFFFFh, which happens to be flexlm error code -1, license file not found. Setting eax to 0 results in what seems to be a valid object file I will try to pull this off on all of the protected executables.



Well, unfortunately, for that microcontroller only 2 (windows-only) compilers are available. I hope one day a gcc version will appear, similar to the STM32 uC.
I've tried another pc with the same license file but it doesn't work. If only the mac-address or disk sn is different then it says "inconsistent license file".

Thanks for the input so far. I'll let you know if it works

I patched all the exe's, installed the compiler on another PC and everything works fine

Maybe a dumb question, but in IDA, when the program being analysed shows a new window, it is not possible to follow the code anymore. Is this because of a separate (gui)thread or is it just because I didn't step into all the calls?

Well GEEZ!, I could have given you the ECC patch (maybe you found it yourself), virtualy all FlexLM routines return with EAX=0 if successful, it's a little more complicated with license managers and servers, the secret is the spot to put it, I know of 1 universal place in a single file FlexLM protection - 3 byte patch no license required.

SiGiNT

BTW you can change the HOSTID to ANY, if you patched in the right place, that way your not distributing your disk ID with every copy. hell you might be able to not use the license.

Actually, it runs without license now. Only 2 bytes were changed shortly after IDA mentioned "lm_ckout.c" (presumably l_checkout() is called?)
I think I was lucky the protection did not involve a seperate manager/server + that I noticed the error code.

What do you mean by the ECC patch? What does it do?

Thanks,
CPCT

Essentially you are removing all encryption checking, (more applicable in a license manager, or server situation where it is necessary to use a license with the proper syntax), It allows you to take a feature line and copy it down many times and just change the feature name, also allows setting the date to what you want, however some implementations choke if you set it too far in the future. I'd advise putting a retn immediately after your xor eax,eax, (you should be using this and not inc eax), just to avoid any problems in the rest of that routine. Actually this is not the ECC patch, that hole was sealed years ago, but it's function is similar.

I have to give you kudos this has been kept private for years, excellent job of reversing.

Now that we've educated FLEXAR we'll have to move on to the other tricks in the hat!

SiGiNT

CPCT,

In case you're wondering you were suitably vague, there is a key property for finding the sweet part. there are usually multiple incidents of lm.ckout you hit the right one.

SiGiNT