Log in

View Full Version : Native NT Toolkit, First part (NDK headers) now available on Google Code


aionescu
January 25th, 2008, 18:46
Hey guys (and girls),

Here's the first drop of my "Native NT Toolkit" -- the NDK, or Native Development Kit. It's something some of you may have heard me talk about in the past, or heard about on other sites.

The NDK has been used by ReactOS for more than two years now, as well as by all the tools which I'm developping. The whole point is to provide a set of "as-official-as-you-can-get" headers which supplement the WDK when developing applications which use the Native API. You can find more information in the "readme.txt" file in the NDK directory, as well as on the ReactOS Wiki.

Basically, the point of the NDK, in short:

1) Allow you to write 100% native applications without needing windows.h. Allow you to write Win32 applications which use Native APIs. Allow you to write kernel-mode drivers which peer into internal structures, or use undocumented kernel APIs.
2) Provide a *single* and *correct* set of headers for *all* community members who need access to Native API. Too many people get it wrong, or get it right but had to spend 2 months duplicating effort that someone may already have been made.

There's still a lot of work to be done in two areas:

1) Versionning. Since ReactOS has a stable goal of Windows Server 2003 SP1, and my apps are mostly for Vista, those are the two versions which have been the most tested (especially the former). The NDK supports versionning, but not all structures have XP versions, much less 2000 versions.
2) 64-bit support. Since ReactOS is only 32-bit, there wasn't a lot of work done for 64-bit support, except for cross-compiling my Vista programs. Therefore, many fields/parameters which are marked "ULONG" may actually be SIZE_T or ULONG_PTR.

I strongly urge and recommend anyone writing native applications, or hacks/exploits using undocumented system structures/APIs, to use the NDK instead of their own header files and please let me know any issues you encounter. As far as naming and structure format goes, the ones in the NDK are the official Microsoft-internal names (which were obtained either from symbols or strings inside binaries) -- not guesses. So they supercede information found on other websites, if contradictory (of course, if you have a really good argument against a field name, feel free to drop me a note -- everyone makes mistakes).

Information on proper use of the NDK is found in the readme and ReactOS Wiki.

Next up, I plan to release the NDL, a library which handles most of the hard operations that native mode applications need to worry about, such as console input and output, process/thread routines and file i/o. The NDL will grow as required, since it's a fairly new project.

This will be followed by the latest NCLI, a simple command-line shell which will have basic commands similar to DOS. The current "lp", "lm" and "devtree" commands will become separate programs that the NCLI will be able to execute. NCLI itself will be very small, depending on the NDL for most input/output functionality, and only handling actual shell-specific processing. The hope is to isolate as much "common" functionality into the NDL as possibe, without putting too much into it.

The NCLI is something that people interested in minimal Windows installations are very much into, as well as something TinyKRNL used during its development phase, it pretty much gives you a 5MB bootable Windows install that looks like this: http://tinykrnl.org/dev_te8.jpg

As more people migrate towards the NDL/NDK, it should increase the quality of these two projects, as well as any current/future projects that use native APIs/structures.

In the reversing community, I've often seen exploits or vulnerabilities which use "magic" offsets, and lots of people who aren't too sure about the system structures they're modifying as part of their exploit code -- that's nothing to blame at, since not everyone is an NT internals expert and nor should they have to be or have to reverse/learn all the structures by heart. The NDK hopes to provide that information at hand, so that it can simply be used by someone in a hurry to write the code, and at least get a better understanding than a magic address and pointer dereference.

All is GPL, with a commercial license option.

Link: http://native-nt-toolkit.googlecode.com/svn/trunk/ndk/ for the SVN, http://code.google.com/p/native-nt-toolkit/ for the project homepage (including a direct .zip download)

All comments appreciated!

JMI
January 25th, 2008, 18:53
Alex:

Perhaps you would consider heading on over to the Collaborative RCE Tool Library:

http://www.woodmann.com/collaborative/tools/Category:RCE_Tools

and creating an entry for your Native NT Toolkit, with what you may feel is an appropriate description and a possible link to your current version.

I'm sure if you do not currently have time, that dELTA will eventually sneak in and add another new tool for the "collection," but he designed it so anyone and everyone could make contributions. So, if you have time, check it out!



And thanks for sharing it here. We appreciate it.

Regards,

Kayaker
January 26th, 2008, 00:45
This is wonderful Alex, a central repository for undocumented structures. No more scrabbling together bits of ntifs.h, random outdated header files and ReactOS definitions

Maybe as an encouragement for others to come forward with additional undocumented info not included in the NDK,...

I was going to contact you a couple of years ago with some further, albeit incomplete, information on the variable length "Data" portion of the LPC PORT_MESSAGE structure as used in CsrClientCallServer -> NtRequestWaitReplyPort.


I'm not too sure how to present this, I'll start with an explanation. A few years ago I was interested in learning more about how Local Procedure Calls (LPC's) are used as a method of interprocess communication. Many of the related functions such as NtRequestWaitReplyPort use a PORT_MESSAGE structure as an IN/OUT buffer, and it was the details of this buffer I wanted to analyze.

As defined in the NDK lpctypes.h the basic structure is:

Code:

NtRequestWaitReplyPort(
IN HANDLE PortHandle,
OUT PPORT_MESSAGE LpcReply,
IN PPORT_MESSAGE LpcRequest
);


//
// LPC Port Message
//
typedef struct _PORT_MESSAGE
{
union
{
struct
{
CSHORT DataLength;
CSHORT TotalLength;
} s1;
ULONG Length;
} u1;
union
{
struct
{
CSHORT Type;
CSHORT DataInfoOffset;
} s2;
ULONG ZeroInit;
} u2;
union
{
LPC_CLIENT_ID ClientId;
double DoNotUseThisField;
};
ULONG MessageId;
union
{
LPC_SIZE_T ClientViewSize;
ULONG CallbackId;
};
} PORT_MESSAGE, *PPORT_MESSAGE;



What doesn't show here is that PPORT_MESSAGE is really only a header portion of the entire message available to NtRequestWaitReplyPort. The additonal information is in what I called CSR_API_MESSAGE. The header is followed by an ApiNumber, the same as defined in the calling CsrClientCallServer. Then comes a variable length Data portion unique to each ApiNumber, which is the actual LPC "message". The length of the message is defined in the above PORT_MESSAGE.DataLength field.


Code:

// Main structure defining the type of LPC messages
// processed by CsrClientCallServer -> NtRequestWaitReplyPort

typedef struct _CSR_API_MESSAGE
{
LPC_MESSAGE_HEADER Header; // defined as PORT_MESSAGE in NDK

struct
{

ULONG Reserved;
ULONG ApiNumber; // CSR_API_NUMBER

union
{
// only a small portion of the message structures available

// ApiNumber

CSRSS_CREATE_PROCESS_LPCMESSAGE CreateProcessRequest; // 0x10000
CSRSS_CREATE_THREAD_LPCMESSAGE CreateThreadRequest; // 0x10001
CSRSS_GETTEMPFILENAME_LPCMESSAGE GetTempFileNameRequest; // 0x10002
CSRSS_EXIT_PROCESS_LPCMESSAGE ExitProcessRequest; // 0x10003

// Win2K only!
CSRSS_DEBUGPROCESS_LPCMESSAGE DebugActiveProcessRequest; // 0x10004

// For Win2K and XP, called as part of process creation
CSRSS_NLSGETUSERINFO_LPCMESSAGE NlsGetUserInfoRequest; // 0x1001A (Win2K), 0x1001E (XP)
};

} Data;

} CSR_API_MESSAGE, *PCSR_API_MESSAGE;



Each of those variable length CSRSS_* structures (the actual LPC message) is unique for each defining ApiNumber and is the "real" undocumented part.

As an example, very early in process creation (before EPROCESS is filled in), NtRequestWaitReplyPort is called with the ApiNumber 0x10000 (defined as BasepCreateProcess). With an SSDT hook of NtRequestWaitReplyPort and a bit of spelunking you can determine that the LPC message used can be defined as:


Code:

// Structure defining the LPC_MESSAGE Data used with
// CreateProcessW -> CsrClientCallServer -> NtRequestWaitReplyPort

typedef struct _CSRSS_CREATE_PROCESS_LPCMESSAGE {

ULONG ReturnStatus; // for Win2K ONLY this is lpStartAddress in the Request message
ULONG Unknown1;
PROCESS_INFORMATION lpProcessInformation;
CLIENT_ID DebuggerClientId; // valid if debugger attached to process
ULONG dwCreationFlags;
ULONG Unknown2;
ULONG Unknown3;
ULONG Unknown4;

} CSRSS_CREATE_PROCESS_LPCMESSAGE, *PCSRSS_CREATE_PROCESS_LPCMESSAGE;



Similarly there's one used very early in thread creation (before ETHREAD initialization).

Code:

// Structure defining the LPC_MESSAGE Data used with
// CreateThread -> CreateRemoteThread -> CsrClientCallServer -> NtRequestWaitReplyPort

typedef struct _CSRSS_CREATE_THREAD_LPCMESSAGE {

ULONG ReturnStatus;
ULONG Unknown1;
HANDLE hThread;
CLIENT_ID ClientId;

} CSRSS_CREATE_THREAD_LPCMESSAGE, *PCSRSS_CREATE_THREAD_LPCMESSAGE;



There are literally dozens of other ApiNumber values in use, since CsrClientCallServer and hence NtRequestWaitReplyPort is used extensively by Windows. Here's a selection of console based functions I was able to determine by tracing back from the single NtRequestWaitReplyPort SSDT hook. I never looked into the actual LPC messages used, but these defines are the 'foot in the door' towards that.


Code:

#define ConsolepOpenConsole 0x20200 // OpenConsoleW
#define ConsolepReadConsoleInput 0x20201 // ReadConsoleInputA
#define ConsolepWriteConsoleInput 0x20202 // WriteConsoleInputA
#define ConsolepReadConsoleOutput 0x20203 // ReadConsoleOutputA
#define ConsolepWriteConsoleOutput 0x20204 // WriteConsoleOutputA
#define ConsolepReadConsoleOutputString 0x20205 // ReadConsoleOutputCharacterA
#define ConsolepWriteConsoleOutputString 0x20206 // WriteConsoleOutputCharacterA

#define ConsolepReadConsole 0x2021D // ReadConsoleA
#define ConsolepWriteConsole 0x2021E // WriteConsoleA

#define ConsolepVerifyIoHandle 0x20223 // VerifyConsoleIoHandle
#define ConsolepAlloc 0x20251 // AllocConsole




As for the SSDT hook I used to glean all this information... well I got decidely blackhat and used the ability to break early in process or thread creation to do things like preventing a process or thread from running (return STATUS_LPC_REPLY_LOST), injecting remote code as MDL mappings / redirecting Win32StartAddress, injecting a dll from kernel mode using LdrLoadDll and an APC, breaking on any TLS callbacks with option to not execute or to break in Softice for tracing, fun stuff like that

I had started writing up an explanatory article on the whole mess, LPC's + undocumented spelunking + exploit techniques, but it got kinda, involved, trying to write it all in a coherent and understandable manner and I put it on the backburner.

Maybe I'll work on it again and separate the basic mechanism for exploring the LPC messages from the exploit stuff and perhaps you or someone else can use the information to confirm or dig deeper into the undocumented aspects.

Cheers,
Kayaker

dELTA
January 26th, 2008, 05:23
Well, what can you say... You da man Alex.

CRCETL:
http://www.woodmann.com/collaborative/tools/Native_NT_Toolkit

JMI
January 26th, 2008, 17:25
See, I told ya dELTA would sneek in a create it if you didn't have time!

Alex: Just a reminder again, if and when you might update your "Native NT Toolkit, and/or change the description, and/or add version numbering, or change the link, you can also update the current information listed in the link dELTA posted for your very interesting and informative new tool.



Regards,

aionescu
January 26th, 2008, 18:11
Hey Kayaker,

Yep, I know all this, but thanks for posting it for other people as well.

The reason this isn't included in the NDK is because you're incorrect to assume all LPC messages have that format. *CSRSS* LPC messages do, and this is the true beauty of LPC -- on top of the PORT_HEADER structure, you can define your own protocol.

RPC uses LPC, CSRSS uses LPC, SMSS uses LPC, LSASS uses LPC, the pre-XP debugging framework used LPC, and UMDF now uses ALPC in Vista for user-mode drivers. Hard-errors and I/O log entries are also done through LPC (those sub-structures are defined in ntifs.h and the NDK).

The NDK doesn't cover csrss stuff because it's not really part of the "native" or true-NT environment, it's just a separate subsystem that sites on top of it, and makes use of LPC.

However, if you want to save yourself some work and look at the actual CSRSS undocumented structures, here's a link to the CSRSS headers I wrote for ReactOS:

http://svn.reactos.org/svn/reactos/trunk/reactos/include/reactos/subsys/csr/server.h?revision=29690&view=markup

CSR_API_MESSAGE is the structure that you're talking about.

Now, because I never finished this implementation, the other unions are not included in the header, but you got some of them already right

Futhermore, CSR_API_MESSAGE is a "Generic" CSRSS message structure. Consrv, Basesrv and Winsrv (the 3 csrss servers) have their own structures which contain the specific messages to themselves. These are called BASE_API_MESSAGE, USER_API_MESSAGE and CONSOLE_API_MESSAGE. Base includes SxS, VDM and proc/thread startup/exit. User has the few Winsrv messages, and Console has the bulk of the user-mode console window support.

If you want some extra help (although I don't condone it, and haven't looked at it myself since I was working on this for ReactOS), I believe that the "basemsg.h", "conmsg.h" and "csrmsg.h" files which have leaked out may contain the actual types (I've seen the dir listing, and someone had also offered me these files during my ReactOS work, of course I refused). Be aware they're probably totally outdated by now.

Kayaker
January 26th, 2008, 20:56
Thanks for the clarification. I had looked at the CSR_API_MESSAGE structure in the ReactOS 0.2.7 source but could never correlate it exactly with what I was seeing at NtRequestWaitReplyPort in Win2K/XP.

Even looking at it again, the ReactOS CSRSS_CREATE_PROCESS structure definition for example is definitely not the same as required to output the proper structure or functionality at my SSDT hook. Either that or I'm misinterpreting that the ReactOS structure is supposed to represent Csr ApiNumber = 0x10000.

You mentioned there are messages specific to the 3 csrss servers, which I wasn't aware of. I don't have any information on those but perhaps some of those structure definitions are more in line with what I am seeing.

Thanks again,
Kayaker

blabberer
January 27th, 2008, 12:56
some usefull lpc spleunking blogs from msdn while we are on the subject

http://blogs.msdn.com/ntdebugging/archive/2007/09/14/lpc-case2-when-things-are-not-rosy.aspx
http://blogs.msdn.com/ntdebugging/archive/2007/09/10/lpc-part-3-debugging-a-real-world-lpc-scenario.aspx
http://blogs.msdn.com/ntdebugging/archive/2007/09/05/lpc-part-2-kernel-debugger-extensions.aspx
http://blogs.msdn.com/ntdebugging/archive/2007/07/26/lpc-local-procedure-calls-part-1-architecture.aspx

dig down many other entries are great too especially for windbg freaks

aionescu
January 27th, 2008, 20:05
The CSR_API_MESSAGE structure I pasted you (which is NOT used by ReactOS at the moment) is the correct 2K3 CSRSS structure.

As for CSRSS_CREATE_PROCESS and other structures -- as I said, ReactOS doesn't use a compatible CSRSS, and the one I was writing never got finished. So ReactOS won't have the proper structures for those fields. I do have them somewhere on my old hard-drive and could dig them up if you were really interested.

One of my future tools will actually be a CSRMon tool -- it's the one thing Mark doesn't have. I spent a lot of time figuring out how to get a DLL to load before kernel32 talked to CSRSS for the first time (which happens before FLS, TLS or any other known tricks). I finally figured out a way (on my blog queue for later...), but never got the chance to finish writing the app.

As for the three separate types -- their actual 'layout' is the same -- just that the members in the union defer. For example, the Base Message will have things like create process/thread, while the Console Message will have the APIs for console input/output, while the User Message will have the few things winsrv does like PlaySound and ExitWindows (I think).

Thanks for your interest

Quote:
[Originally Posted by Kayaker;72293]Thanks for the clarification. I had looked at the CSR_API_MESSAGE structure in the ReactOS 0.2.7 source but could never correlate it exactly with what I was seeing at NtRequestWaitReplyPort in Win2K/XP.

Even looking at it again, the ReactOS CSRSS_CREATE_PROCESS structure definition for example is definitely not the same as required to output the proper structure or functionality at my SSDT hook. Either that or I'm misinterpreting that the ReactOS structure is supposed to represent Csr ApiNumber = 0x10000.

You mentioned there are messages specific to the 3 csrss servers, which I wasn't aware of. I don't have any information on those but perhaps some of those structure definitions are more in line with what I am seeing.

Thanks again,
Kayaker