I'm attempting to become the authority on Protection Plus. (The Nolan Blender of it if you will).

My plan is to make a replacement .dll will nice debugging facilities built in, etc. I have ALOT of the code written, all the replacement function stubs in place, and I have it working for target #1. I have picked up a couple of newer targets, and THEY don't like the format of my function declarations in the .DLL.

So, I'm asking for help with this.

I have searched and nothing specifically deals with this.

IDA spits out a function prototype from the dll like this:


According to the site over at SoftwareKey.com, this function has THIS prototype:


I've tried BOTH in my dll, and when one of my targets tries to use it, (It's an Excel sheet of all things!), it spits out "Bad DLL Calling Convention - Error 49".

In the VBA in the Excel sheet, it's declared like this:



And called like this:



(Sanitized to anonymize the target)

Anyone have any ideas of what I'm doing wrong?

On first glance it seems to only be caused by calling convention. Assuming that the prototype from the website is used in code and built by a default setup of MSVC/VS (ie: no /G param), it will be __cdecl.

I'm 99% certain that normal VB can only use stdcall, so logic would follow that VBA suffers the same restriction. This also logically confirms what IDA is showing you - the stdcall export.

So the problem you're having is that a VBA script can't call an exported function from your DLL? One thing I notice is that the IDA prototype (which uses the apparently corect stdcall) has an int in place of the LPTSTR param. Sure, you can store the pointer in the int if this was straight C code, but perhaps the problem is because VBA is in the mix here?

Actually Silver, I'm having the problem of the VBA script not being able to call the functions from the DLL, AND, I have a second target that just crashes, so I know I'm doing SOMETHING wrong.

Using the "factory" DLL, the VBA code works just fine, and that's what I disassembled to get the IDA prototype.

Went back and read some OLD SCHOOL windows programmings tuts, and found out that when you use __stdcall, you have to use a .def file to define the exports. (unlike __declspec(dllimport)).

Once I made the .def file, and changed the calling convention to __stdcall, it worked just fine.

JMI, you can kill this thread if you'd like.

Why do that? It might be of some help to someone else with the same or a similar problem.

Regards,

That's not entirely accurate. You can export stdcall without a .def file, but name mangling will ensue. The .def file prevents the name mangling because you're able to specify the explicit name for the export.

I guess VBA must have some extra detection system to find a name-mangled stdcall export in a DLL, because that error you got doesn't directly correlate to the problem. Interesting.

I use extern "C" to avoid the name mangling problems. The reason that I say that the .def file is required, is that when I set things up to use __stdcall without the .def file, VBA complained that it couldn't FIND the function in the .DLL instead of complaining that the calling convention was wrong.

JMI, I thought the subject matter was too simplistic for the ADVANCED forum.

FrankRizzo, you can't do that. When you compile __stdcall the exports are always decorated/mangled. You cannot extern "C" an __stdcall export without a def file and prevent name mangling. It's well documented, have a google for "prevent name mangling export def file stdcall" or some permutation thereof.

It's also extremely simple to check for yourself. Leave out the def file, keep the extern "C" with your __stdcall, then check your exports with depends.exe or anything else. You'll see your exports are still mangled.