I wrote new IDA plugin.

It loads obj and lib (COFF format) files signature to ida database..

it identifies so many labels more than flair signatures..

you can try..

(
i use it for flexlm cracking.. you can load lmgr.lib to ida database
)

Seems pretty useful. Thanks.

I downloaded and installed it, and now I'm curious as how it works?
Could you tell us a little bit of its internals?

thanks!

can you pls share the src code ?
it will be useful to us to add more into that. Thanks

If used correctly flirt signatures are created from libs and obj so i dont
see how this is any different

thanks .. still having the code is good to learn

FLIRT signature creation not possible for some situation, for example you can try to create flirt signature for flexlm libs..

my plugin is free... you can use if you wants...

thanks for all..

gerbay...

thanks a lot!!!
I'll take a look, and if possible, I'll try to contribute as a "thank you"

hehe,

gerbay

i did also not try it, just looked at the source. cool, its also a nice reference how to add fileformats to ida (for me, thank u )

regards, 0xf001

Hey,

@gerbay: Nice plug, i tested it with flexlm for example. I tried my sigs and also your plugin with the correct library. Seems that your plugin is recognizing more functions than the signature approach.

I will try it in future on other targets.

Cheers,

PAPi

Thanks for your codes!
It is very useful!

Thanks a lot, gerbay.
With this plugin, I have import MFC Static lib into a MFC static dll. It is very good. My MFC signatures files does not reveal many symbols.

Nice work gerbay! (and nice to see you're still around TQN)

Hi gerbay !
Sometimes your plugin causes IDA crash. Can I debug, find the bug and modify your code ?
Regards,

please, feel free the modifing my code..

if you find bug please post it the forum.

FLIRT should work with anything, just make a custom utility to parse the lib format, etc., to make .PAT files.

Sirmabus:

FLIRT falls over in many cases due to it's design (relies on first 32 bytes of function, no robust handling of referenced functions, no datapoint symbols, CRC16 hash has high collision rate, etc..).

gerbay's code does a straight binary match (excluding relocations) of lib functions, which will definately find more matches. It doesn't deal with 'identical-except-referenced-functions' or a couple of other edge-cases though.

pynary aims to provide a more flexible solution to this problem: http://code.google.com/p/openrce-snippets/wiki/pynary

For some ideas and comparisons check out some of the others that I know of:
1. IDACompare: Free w/source: http://labs.idefense.com/software/static.php#more_idacompare
2. BinDif: $1225USD, trial available too: http://www.zynamics.com/?page=bindiff
3. eEye Binary Diffing Suite: Free: http://research.eeye.com/html/tools/RT20060801-1.html

I didn't try them all very deeply but:

1. IDACompare: Appears to work fairly well. It uses some comparison methods beyond the typical binary matching.
Not as well as BinDiff (when it works, that is).

2. BinDiff: For starters I don't see why the $1225(USD) price tag. I don't think it justifies in this case paying more for a plugin then for the price of the primary tool it's self!
When it works (on relatively small DB's in my testing) it works well. It has a nice feature that allows you to "port" over one DB to the next, taking
comments with it, etc.
Because it's a commercial product and because of it's price, I think it's fair to judge it a bit more critically; I would expect a good product in
exchange for that kind of investment.
It has several problems making it useless in the main cases I was interested in. It crashes on large DB. Not only dose it crash, it leaves 100mbs of temp files around in an obscure directory (after many hours of running, and no user feedback). Bad user interface design; A design that's dependent on IDA's plugin situation is a problem since it (IDA) can't refresh it's window properly (no solution AFAIK).
Not knocking "Halvar", there are could be many technical/design issues behind the scenes. He's evidently done some sort of post graduate research work on the subject of pattern matching.
IMHO he needs to take this thing out of the academic into a more practical product. Apparently he mostly does malware research. If using BinDiff on small mallware samples the problems probably don't show up very often. But that would be no excuse for a viable commercial product..
I suppose you are paying for the algorithm(s) (and his research) underneath it, not for the fancy (but largely unworkable) exterior.

BinDif needs:
1. Better memory management that can handle any IDA DB you through at it. If algorithmically it can't then it needs to inform the user of it's limits
in the very least.. Perhaps it could switch to a different matching model for larger DB's in order to handle them. Not as accurate perhaps, but maybe something workable.

2. Crash handling, some sort of crash handling that informs the user what happened, and some way to fix or at least report it.
It should clean up all the temp files it creates on failure, etc.

3. A better UI that is not directly tied to an IDA Window. Any IDA plugin writer (like my self) knows there is problem with IDA's UI. Take any large/slow plugin and run it, then try to min then maximise IDA's window. You will see it didn't redraw because of it's modal nature with plugins, and, or some sort of thread starvation issues.
There is probably some way to fix this. Perhaps using the UI callback facility, or even hack'ish hijack part of IDA's screen handler, giving the UI thread more priority or something. I don't see why one can't just detach your own UI from IDA's.
IDACompare does this FYI. Use the plugin interface to do the real work, but launch a UI as different layer with a different thread, with it's own message pump, etc.
This way the user will still be in control. They can TAB in another program (like check their email) and come back and not see a messy un-redrawn window that leaves the user in a quandary wondering if it's running or if it's crashed. It can have a progress window and a cancel button (and a "save and come back" feature do to it's slow data intensive nature, would be nice).

Furthermore, the plugin could act more as an intermediary to put the real workload off on a separate process that can have it's own memory manager to solve other issues.

3. eEye: Haven't used it enough to comment. Although I vaguely recall results similar to IDACompare.


So I'd say if you want to make the ultimate IDA signature and DB porting utility study these other offerings.
Read Halvar's papers on the subject:
http://www.zynamics.com/?page=publications
and any similar publications.

Fix/avoid the problems of BinDiff, apply your own genius to it. That's a product we all want, and would be willing to pay our hard earned $$ for..

First of all, I don't think executable code identification signatures and executable diffing tools are the same. As soon as a function has been changed/updated, the former type will fail, while the latter will still work (and even point out the changes for you).

To keep track of available executable diffing tools, see the following CRCETL category:

http://www.woodmann.com/collaborative/tools/Category:Executable_Diff_Tools


To keep track of future updates of Gerbay's nice signature tool, the following CRCETL entry is for you (Gerbay, let me know if you want another name on it, I didn't find a real name created by you for it at all?):

http://www.woodmann.com/collaborative/tools/Advanced_obj_and_lib_IDA_signature_ripper

From October 2004 to roughly September 2006, as the first employee of Zynamics (known then as Sabre Security), I was the lead researcher and developer behind BinDiff. Although I'd prefer people not purchase BinDiff, I would like to respond to some of the points made here about my work.

Disclaimer: the opinions stated herein are my own and not those of Zynamics, on whose behalf I am not speaking. All statements made as facts are true to the best of my knowledge and do not contain any proprietary information. Indeed, this post is quite positive and beneficial to Zynamics.

It sounds as though Sirambus is talking about BinDiff versions v1.4-v1.8, which have been obsolete for about 20 months. I was a poor programmer working on a poor codebase at that time. With some programming experience under my belt, I rewrote the v1.99 codebase independently and from scratch, and Sabre began distributing that in April of 2006. Version 2.0 was eventually released in September of 2007. If Sirambus' criticisms are indeed based on obsolete versions, they may or may not be moot points.

>> 2. BinDif: $1225USD, trial available too: http://www.zynamics.com/?page=bindiff

I don't believe there's a trial available anymore. I believe they pulled it after it was cracked and distributed.

>> BinDiff: For starters I don't see why the $1225(USD) price tag. I don't think it justifies in this case paying more for a plugin then for the price of the primary tool it's self!

From the end user's perspective that's fair enough, but a small company in a niche market has got to make a living. We certainly didn't drive around in Bentleys sipping Moet when I worked there. (Although that may have happened after I left the company and Zynamics won EUR 100k in a contest for which they conveniently forgot to credit me -- and I'm told that they removed my name from the help/about screen in BinDiff v2 as well.)

>> It has a nice feature that allows you to "port" over one DB to the next, taking comments with it, etc.

I'm glad you like that feature -- I thought nobody used it :-) I'll digress for a minute. This is a point of pride for me. I got my job at Sabre Security after I analyzed and decompiled BinDiff v1.4 into working replica source code (this was before Hex-Rays). When I coded the comment porting feature in v1.6, I tested it by porting the comments off of my BinDiff v1.4 IDB onto an IDB for a later version. I named it "first_database_in_the_world_with_ported_comments.idb", and savored the moment, which remains one of my best technical achievements.

>> It crashes on large DB.

Although this happened quite a bit in v1.x due to memory leaks and other bugs, v1.99 was much more stable. I only recieved one bug report for v1.99 in the six months before I left the company, and it wasn't a crash. Your mileage may vary, of course, and I couldn't possibly make a claim about any code added to v2.0 after I left.

>> Not only dose it crash, it leaves 100mbs of temp files around in an obscure directory (after many hours of running, and no user feedback).

Sounds like v1.6 to me. The temp file situation was addressed in v1.99 (all temporary files are deleted after being read, so you will never see a buildup). Furthermore, v1.99 was substantially faster than v1.xx. With v1.99+, I doubt there's any scenario in which it would take "hours", let alone "many hours", to complete any diff. I can remember diffing acrord32.dll (60k+ functions) in under five minutes. I can't speak about any modifications made to the code after I left.

>> Bad user interface design; A design that's dependent on IDA's plugin situation is a problem since it (IDA) can't refresh it's window properly (no solution AFAIK).

IDA's GUI situation is indeed quite limited, but that's the price you pay for text-mode compatibility.

>> 1. Better memory management that can handle any IDA DB you through at it. If algorithmically it can't then it needs to inform the user of it's limits in the very least.. Perhaps it could switch to a different matching model for larger DB's in order to handle them. Not as accurate perhaps, but maybe something workable.

v1.99 scaled much better than v1.x in general. I remember diffing two 12k-function binaries with both v1.8 and v1.99. 40 minutes and 16mb of memory for v1.8 vs. 55 seconds and 13mb for v1.99. So unless they've slowed v2.0 down substantially compared to v1.99, this is probably a moot point, as v1.99 handled everything I threw at it with ease.

>> 2. Crash handling, some sort of crash handling that informs the user what happened, and some way to fix or at least report it. It should clean up all the temp files it creates on failure, etc.

As I said, I never recieved any crash reports for v1.99 while working there, and I don't know if something like this has been implemented in v2.0, so this may or may not be a moot point.

Any further questions about v2.0 should be directed to whoever's maintaining my code nowadays.

"don't think executable code identification signatures and executable diffing tools are the same"

Yes, if you had to categorize them.
And I suppose really to keep this thread on topic I shouldn't have posted this comparison here.
I hate spam as the next person, I just wanted to put my input on the subject since I feel the area is
one of the "holly grails" of RE.

In my mind, probably since I've been using such tools for so long, done my own research and work in the area, the lines
conceptually between the two are not as distinct.

Mainly, I see a BinDiff tool as not being a sig tool, but the concepts of a sig tool can be applied to (and is usually used by) one.

And you can to some degree use part of one for the other depending on your use.
Example, what is the official term for the updating a hack/modification from build of an executable to another?
Lets call it "UPDATE" (with out certification from the Ivory Tower).
Well you could use a signature tool to track parts of one build to the other and it comes in quite handy.
Although a BinDiff tool might be better since it might match partial things, plus it should show you knew things.

(Sorry admin, I've hijacked this thread more, but it's interesting topic?)

I meant not to slam the product, or anyone in particular.
IMHO doesn't live up to the hype (on the website), it's not a "done" product, and frankly it had to be said.
It was not just coming from my own experiences, but others that told me they had the same problems.
Should be noted it's the only commercial type AFAIK. And I understand and agree; it's a niche market.
I don't believe everything should be free.

I'm speaking of BinDiff v2.0.269.

I should really use specific cases to exactly document so it's just the "facts" to prove what I'm saying...
At least two that I tried on that failed. One is a 23mb DB, and the other a 26mb DB. In the second case it was particularly bad
because the origional had a PDF applied giving the IDA DB over 80k symbols (if that matters)!

The temp files were there after the crash, trust me. I did a "properties" on the BinDiff directory incidentally and saw it was increased by 100's of megabytes.
Also there were so many thousands of files it took several minutes to delete all of them (and that on a Raptor 10krpm drive).
There are obvious solutions to this so if/when it does crash either the crash handler deletes them, or on next invocation the plug-in having tracked, and, or, cataloged the previous run it knows to cleanup first.

Were you perhaps using the cracked version?

I used an eval version. I'm sure the cracked one has the same problem regardless.

I was going to have my company buy it for me.
But after using it, frankly I'm glad I didn't ask because of the issues I discussed.

Again, I'm sure it has it's uses. It probably works fine and even exceptional for malware researchers using small data sets/DBs et al.
But I expect more out of a commercial product, in particular because of the price.
Zynamics just needs to fix a few issues, test it with larger data sets, etc.

If any engineer of Zynamics wants to, they can PM me and I will help them fix it.

First of all, nice to see you on the board Rolf, feel free to stick around.

Just for the record, I love that feature! Extremely useful and nice when doing large scale analyses of executables (e.g. malware) and wanting to transfer all the work and knowledge to a new update of the code! I just wish it would support register names and some other types of names that are currently not supported too.

Regarding BinDiff, I think it's a good product at the core, and support for reported bugs is good and fast. I do think though that it is too unstable (our 2.0 versions have contained many different kinds of serious bugs, most of them not related to the core diffing engine though) and it is developed at too slow a rate for being such an expensive product (i.e. not even small feature requests from registered customers ever go through). And yes, my company has a valid license.

Sirmabus:

I'm familiar with most of the existing solutions and products, although I haven't had a chance to play with BinDiff itself, I've talked about it with a number of people close to it (including Ero and Rolf). I've also done a thorough survey of the existing academic research in the area and even invested quite a lot of time (together with others at my company) in developing _extensible_ algorithms for signature matching.

I aim to implement those algorithms as the first goal of pynary. Please feel free to help out

Eventually I'd like to be able to use the same framework basis and extensible sub-graph isomorphism algorithms to allow for version diffing as well as meta-data porting. I've also got lots of ideas for other powerful features (such as automated C++ class reconstruction).

I'm hoping to leverage the OpenSource reversing communities excellent minds to build a really powerful toolset. I suggest you take a glance at the ToDo list (http://code.google.com/p/openrce-snippets/wiki/PynaryToDo) for some ideas of where we're headed and where you can help out. If there are things you think should be added to the list, let me know.

dELTA: as I mentioned, that feature is planned for pynary as well.

Thanks for the update, and, if your haven't already, please consider adding your tools to our New Collaborative RCE Tool Library when its ready.

Regards,

Ah, the dangers of late-night posting. I'd say "that'll teach me", but it won't. I should have asked Sirambus which version he was using, and then I would have known not to reply because I've never used v2.0, only v1.99. I figured since the two shared a common codebase there couldn't be that much difference between them, but I was apparently mistaken. I'm sorry to hear about peoples' bad experiences with the product, and part of me wishes I could help you.

Still, I must say I am surprised to hear multiple people reporting speed and stability issues in v2.0. It's not an exaggeration to say that I spent weeks doing nothing but optimizing v1.99 for speed, and as for stability, I ran the thing tens of thousands of times without any crashes and did not have any crash reports from the field. Large databases were also not an issue -- I kept a corpus of pairs of binaries ranging from 25k functions to 100k functions, all of which were handled in 5-30 minutes without memory issues.

c1de0x said

It's in my best interest to correct this. I told c1de0x that I was unable to help him in any way with any binary-diffing related endeavors, not even to discuss it, although I wished him well.

Regarding speed and stability of the diffing, I've never had any problems with that with version 2.0, it's quite good indeed actually. Like I said, most problems and unprofessional traits are in all the other parts. It feels somewhat like a good core, wrapped up in a quickly thrown-together and badly maintained shell.

JMI: Will definitely do so as soon as it is ready for prime time.

Rolf: Sorry if I implied otherwise. I considered not mentioning that I had spoken with you about pynary (NOT binary diffing!) in general - as a binary analysis framework, but decided that since you were already participating in the discussion it'd be remiss. I apologize if my statement was not clear.

I also would like to clarify that pynary is *not* meant to be a replacement for BinDiff. Nor is it meant to be in any way a commercial endeavor. In fact, I've been seriously considering changing it from the open MIT license to something which restricts it's usage in commercial applications (share-alike), but am reluctant to do so if it means people will not want to contribute. I'd prefer it if that very important discussion takes place on the OpenRCE Forums though, as that is the official home of pynary.

In addition, I feel that it is important to mention that although I've drawn inspiration and ideas from Halver's (and other's) academic papers, the algorithms I (read: we, the pynary community) plan to implement are NOT directly derivative of those works. In most cases they are either independently invented or only loosely based on the freely available information. Although I very firmly believe in the opensource philosophy, I would never condone the usage of illegally acquired internal trade-secrets in the implementation of an open-source alternative to a commercial product.

At the end of the day, we're *ethical* hackers - right?

c1de0x

Hello! Ida seems to always crash while using this plugin to scan for sigs on msvcrt.lib. Any idea why and how to fix that?

I've been having the crash too. It's caused by the usage of the "&vector[0]" idiom, which fails if the vector is empty (i.e. vector[0] is illegal).

I attached my custom version with a fix for this crash (a length check) and some other modifications:
- prints the name of each recognized function/symbol
- marks recognized functions as library functions (cyan color)
- refrains from naming a block of code as a function if the start of that block is not also the start of a function (as recognized by IDA).

Attached it instead of replacing on the CRCETL since I'm not the original author of the plugin.

Thanks, will try it!

Great fix, I added a reference to it from the plugin's CRCETL entry:

http://www.woodmann.com/collaborative/tools/Advanced_obj_and_lib_IDA_signature_ripper

Yes, tested it and everything seems to be working just fine now.Thanks for a great fix!