Hi! I've been seeking in vain for detailed information about AMD processors' supplementary debug facilities that are controlled by (undocumented, password protected!) MSR # C001_1024 "DbgCTLMSR2" and subsequent "DR0_Data_Match", "DR0_Data_Mask" & DR0_Addr_Mask (are there more of those ?)

Tantalisingly the register names which were leaked hint at how useful as a productivity tool the undocumented functions may be to programmers and reversers; unfortunately little more is available (the password itself is easily found by exhaustive search once the register numbers are known...). I've been trying to put those registers to work as a soothing since my dearest one passed away, but there is little hope of success without knowing at least the functions of the bits in the Control register (the low 8 bits are settable on my X32 Sempron 2400+ The registers /do/ affect DR0-based debugging, unfortunately I have not obtained more than the occasional crash or hang)... The data_match and Data_mask have full 32-bit width, while Address_match has the low 12-bits only settable (wild guess, an offset within "page" ?).

I find it unconceivable that such features be kept secret - especially so many years after they were introduced ! Doesn't someone in this select circle have access to the information, either from professional activity or personal reversing ?

It would be nice to have it published here either in the forum or the wiki.

Therefore we have set ourselves to solve this enigma and disclose our findings for the benefit of the community... Please keep visiting this forum thread and the results pages at my blog ("http://www.czerno.tk/") or the Collaborative RCE knowledge library ("http://www.woodmann.com/collaborative/knowledge/Super-secret_debug_capabilities_of_AMD_processors_!")

Best...

--
Czerno

Lots of info here: http://www.google.com/search?q=9C5A203A+OR+9C5A203Ah+OR+09C5A203A+OR+09C5A203Ah+OR+0x9C5A203A

Hi Disavowed! Sorry but your search suggestion is not very specific & yields nothing - the "secret" password is shared with a bunch of MSRs having nothing to do with extended debugging. You can bet I have searched, and not just Google! I even tried to remember the late Fravia's precious lessons in searching - to no avail. There is /no mention/ of this "thing" on the net, certainly not in AMD's publicly available PDFs, not even in the BIOS_Guides which are freely accessible for the Athlon 64 and later (while strangely they never released them for the K7).

The only place on the net it was mentionned before this thread is where I learnt about the existence of debugctlmsr2, etc. way back : an extended list of AMD documented and so-called undocumented MSR gathered by the author of the CBID tweaking utility, formerly at http://cbid.amdclub.ru/, now retrievable only from the web archive hosted at http://cbid.softnology.biz/html/undocmsrs.html.

I exchanged emails with the author a few years ago, he had no clues beyond the names and numbers for those MSRs As he wrote to me, he had gained knowledge of their existence by them being mentionned on a restricted AMD partners forum, but even there there was no further disclosure.

It is my guess AMD must have application notes which they communicated only under extremely strict NDAs. Though I can't accept nor even understand /why/ such tight secrecy applies - or, who knows, post 9/11 paranoia ?

Presumably /some/ developers received the specs yet beyond AMD's own, and tis my hope someone lurking here is able to clue us.

Guys, I've reversed this in part... breakpoints defined in DR0 can be made to fire only on data match (under optional mask), plus masking of any or all of 12 low address bits ! Works also for I/O break points, provided CR4_DE is set, of course !

I may start a tuto if some are really interested (which has not been clear until now). The above results were obtained with bit 1 (weight=2) of DebugCTLMSR2 set.

Interested folk, if any, might also help reversing further features. Bit 0 set definitely has an effect, but has only "locked up" the victim machine under the (limited) test conditions. Bits 2 to 7 haven't had any perceivable effect, need more testing.

Come on, please consider the challenge of completing the reversal...

Hardware conditional breakpoints? Sounds very interesting to me (and any debugger coder)! We would love to have your tutorial about this, and we could also store it in the CRCEKL.

PS.
I think the "Poll" thing might scare people away from the thread...

Very well, how do we add pages to the CRCEKL ? What I have at the moment is already opening very nice possibilities for inclusion in a debugger indeed. I was thinking about writing an add-on for ye old Turbo Debugger (they document the interface for HW based debugging). And this is only using control bit n°1, I still hope to reverse control bit zero in addition...

May I ask which particular debugger you have in mind, dELTA ? Excuse my question as I have not been following the scene for many years...

I've got another question by the way, do you know if there is some debugger, preferably with open source or at least open interfaces, which breaks into System Management Mode (using ACPI chipset facilties) ?


And I thought it would be good as an attention catcher :=)

The main page (http://www.woodmann.com/collaborative/knowledge) has some good info about how to use the CRCEKL, but here's a summary of the hopefully-not-too-complex process for you :

Traverse the catgeory tree to the left to find the most suitable category for your submission, and go to it (if there are several, pick one and see below for more info). (for you in this case I would e.g. suggest: http://www.woodmann.com/collaborative/knowledge/Category:Windows_Internals_Articles)
Click the button at the bottom of the selected category's page, which says it will let you add an item.
Enter all relevant info for your item in the presented form fields. And if there are more categories than one that you wanted the item to be listed in, select these in the "Item categories" section of this form.
Click the submit button, and enjoy the exposure and historical preservation of your item. And also feel free to update the entry for any updates you make to the item in question.

Sounds great indeed! Extending Turbo Debugger might be quite a practical waste though (except possibly for academic purposes of course), since very few people actually use that old relic anymore. Extending OllyDbg or Syser would be much more appreciated and useful to the community I think.


All I said is that it might be interesting for debugger developers to implement your findings in your products (if I understood you correctly). This should apply to any of them, but the big ones like OllyDbg etc would of course be extra nice if they got extended with this.


I must refer this question to the system low-level pro members, Kayaker? Intuitive guesses would be WinDbg or Syser though, but I really don't have any actual idea.

Thanks for the hints. Though, this has nothing to do with Windows or another existing category, could a new category be created, maybe "processors internals" ?

And, oh, I've now reversed bit zero of the control MSR ! In conjunction with bit 1, this simply reverses the condition for beakpoints on data, i.e. break on non match instead of on match !

I'm finding this stuff simply fantastic, and the most amazing is that AMD kept it under such strict secret.



Right, I want to publish what I can find and hope developers of free and or open debuggers go on and implement the stuff in their products.



SMM seems ideal for a powerful systems level debugger, having most of the capabilities of hardware ICE but without the high price onus !

...C U later !

Ah, sorry, my (stupid) mistake. Here you go:

http://www.woodmann.com/collaborative/knowledge/Category:X86_Internals_Articles

Is this applicable to both x86 and x64 btw?


Very cool, sounds like this will be a nice article for the CRCEKL indeed.

I can address the SMM question.

Debuggers - None that I know of, other than an actual ICE. The problem is that most modern BIOSes copy the code to SMRAM, and then set the D_LOCK (D_LCK?) bit, which stops you from being able to modify that memory. (It reverts back to video RAM).

There was an exploit on a VERY specific Intel MB, but that hole was patched and now if you happen upon that very specific board, there's no guarantee that you'll get one with the hole still open. (The exploit was a cache based exploit).

Now, if you were to get in before BIOS locked memory. You'd have to figure out how to plug your code in, and how to get it called.

Hi,

Maybe this source code will help (ps2+smm+apic HOOK):
http://www.debugman.com/read.php?tid=5562

Hi Czerno,

Nice to see you around again, up to your old (new?) tricks

I think the problem with working with SMM nowadays is that most new bioses probably lock the SMRAM control register to deter possible exploits such as the publicized SMM rootkits. (as FrankRizzo mentioned)

I don't know of any ad hoc debuggers which will work with SMM, other than a hardware based system such as Trace32

http://www.lauterbach.com/


If you *can* install your own SMI handler, "debugging" it (and presumably other aspects of the system) is fairly straightforward. You can communicate information over the COM port (WRITE_PORT_BUFFER_UCHAR / WRITE_PORT_UCHAR) using a serial cable and pick it up with something as simple as HyperTerminal. I had made good use of an old Win95 box for that.

Actually, I still have an article and example code on that...attached...

Am486® and ÉlanSC4xx Microcontrollers Simple Remote Debug Kernel CodeKit Software


Regards,
Kayaker

I expect it is, since the same interesting MSRs are listed on Vitaliy's (CBID) page in the X64 section. However as I don't own an X64 processor, I expect other members to check for that soon...



My BIOS doesn't lock it !... and if it did, it would be a simple matter to init/reset the processor programmatically and take control before the BIOS does - I'm sure you know the old trick (like, reprogram memory controller and/or A20-gate so invalid code will be fetched from address FFFFFFF0 upon reset, and catch the subsequent invalid Op exception). Maybe new Mobos have been protected against such hijacking now, but I doubt it somehow...

I'm ready to start an article but at the above URL it says


... and I don't see any listed subcategory to choose from ! Unless it's a browser compatibility problem (I tried 2! Scripts allowed) or PEBKAC of course, an administrator might have kindly to intervene.

Sorry, my mistake. It's fixed now, and I also added equivalent categories for x64 too, just in case.

Article now added !!! Preliminary, AS IS, no warranties... but USABLE imho.
Please contact me (yes, you, any you!) thru email, PM, or publicly in the forum, over technical inaccuracies, additions and so on.

Anyone is welcome to make simple corrections of form, grammar, formatting and the like as far as the software used for the knowledge base allows... Thank you!


http://www.woodmann.com/collaborative/knowledge/Super-secret_debug_capabilities_of_AMD_processors_!

Looks nice Czernobyl.

I did some minot fixups in the text, as you requested.

Looking forward to more details about this!

Thanks! ... How do you add formatting at the Wiki, by the way ?
Do you use HTML tags directly, or BBS-style tags, or something else ?



Yay, this is HUGE, I must stress : carries the power to create break conditions which otherwise were available only to owners of expensive hardware debuggers/ICEs, the price of which last time I checked would easily pay a dozen PCs...

My expectation is word about this will spread and the expanded capabilities be added to existing debuggers (open and commercial). First let's check my assumption, that this is available in all newer AMD procs. This is a spell for you all to test your AMD-based computers for compliance. Do I need to provide a small program which checked for the existence of the expanded debug MSRs, or would that be an insult to those who hang out here !

There isn't actually any way to apply special formatting to the text of entries in the collaborative libraries. The recommended thing for more complex articles is rather to submit them as a PDF file, with a text introduction/preamble in the entry's text.


Yeah, you can say that again, your CRCEKL article was Slashdotted yesterday (http://hardware.slashdot.org/story/10/11/12/047243/Hidden-Debug-Mode-Found-In-AMD-Processors), which blew our entire server out of the water for 24 hours of so.

So, more people than you and me apparently agree that this is very cool, yes.

So is that what happened to the server. Couldn't get it to load the Forums for most of the day.

Regards,

If you'd have read your "special email account" a bit more often, you would have been informed about what was going on a long time ago. Anyway, let's not hijack this thread any more with that topic.

Wow! That slash-dot effect! Sorry for indirectly putting this site down for awhile, though in a way it's had the effect I secretly hoped - I mean of course not crashing woodmann.com, but, this is all over the web now! Just google "Czernobyl + AMD" A plume of fame for this site and for my persona indeed : "an intrepid Polish (or Ukrainian?) hacker". Ha,ha! I can't help thinking how the late Fravia would have made fun of all the nonsense that's being spread, useless alarms about (non existent, AFAICT) security implications.



Why no, I'm waiting for you all to do your homework while me goes to rest - in particular, please check for the functions I have disclosed in as representative as possible a sample of AMD procs. I did the work on one lowly Sempron 2400+ (Thoroughbred die). Why do you think I chose /this/ board for unleashing my cute little findings ? What are you waiting for ? Go test /your/ AMDs !

Cheers

--
Czerno

Czerno:

dELTA was suggesting he and I not hijack the thread to talk about my not checking my email. Enough said about that.

Regards.

I grasped as much; what I'm wondering is why this thread is not (yet) being overflowed with all your test results. I chose this board to air my little bomb - out of a thousand places - because I believed that here, at least, it would meet with enthusiastic testers. Let me not be disappointed, the spirit of Fravia is blowing still isn't it ?

It's really not a tough matter to check for the debugctlmsr2 etc. features. Do people want to be spoon-fed a test program ? I didn't propse to write one out of respect for this community members ! I'll tell you a secret : perso, I do most of the quick and dirty tests under good old DOS debug (or rather the Freedos version, now maintained by Japheth). Just change some IDT vectors (at the minimum, 06 and the 0Dh one- GPF) so they point to Debug's routine for int3, and you can test the most arcane processor bits in less time than it costs to recount the results :=)

But whatever method you best like, go test and report results here for deity's sake !

Maybe we're all using Intel these days! (That's my excuse, at least...)

That's my excuse, too.
My last AMD was a K7. Never missed it dreadfully.
I have a slight dislike for AMD. Guess it's since they took over ATI.
Interesting reading, though.

Intel here as well. I too had a bad experience with an AMD processor several years ago and have been an Intel user since.

Yeah, Intel for years and years...

I guess a test program (preferably "open source" though, to avoid paranoia) might help you get more results from the few people who use AMD today anyway.

You may be right, people might want to be spoon fed nowadays. Well, I shall make a small test program, time allowing, only don't hold your breath : I'm busy with other priorities ATM

well... I have several AMD CPU's, I think that I might do some tests on that. However I never did a low level reverse engineering. But I think that i have enough knowledge to do at least some tests.

I have some AMD Athlon/Sempron and a Phenom XII (mayble useless). maybe there will be a Athlon 64 X2 but I'm not sure.

Hi,

I tested on my Athlon XP-M 2800+ and I didn't succeed.

What I did :

- write a kernel module to read/write the 4 "secret" MSR registers => I works with the password in ESI
- set the registers to :
DbgCTLMSR2 <= 2
DR0_Data_Match <= 0xaabbccdd
DR0_Data_Mask <= 0
DR0_Addr_Mask <= 0xfff
- write a little program which set a variable to 0xaabbccdd
- run it under gdb, and set a watchpoint to an address near the variable address (e.g. if variable is at 0x080482c4, I set the watchpoint to 0x080482c0)
=> there is no break when I run my program I checked what debug register was in use and it was DR0.

Any solution ?

Thanks in advance !

Is ESI a typo ? You want the "password" in EDI instead !



Please ask again if you still are having problems after checking the obvious.
Also if it doesn't seem to work as expected, please dump the contents of DR7 after they're set by GDB...

Yes, it's a typo I tried with and without setting the password, and there's a GPF when not setting it.

I don't have my Athlon right now, I'll give you the answer about DR7 this evening or tomorrow.

Thanks !

Good!



Yay, but first pause to read this : I've just done a few more test cases and, behold! my (preliminary, remember) writing is inaccurate - and you're right, I mean, for the DATA BP with address mask to work (for exception 1 to occur), we need CtlMSR C001_1024 = ZERO, not 2 !
And then of course, non-zero mask(s).

BTW as you wrote, you entered a test value as DATA_MATCH in your own test, but as long you let DATA_MASK=0 the value won't be significant. At least such is my understanding ATM, I need be prudent in my assertions ;=)

My first conclusions were much too hasty and centered on instruction breakpoints. Well, I don't regret publishing unfinished work anyway, since now fine people (like you) are going to complete the sleuthing instead of me alone; and as I said already, I have other matters to deal with concurrently

We'll be updating the Wiki as the collective knowledge develops. There needs to be done a systematic (and tedious) combinatorial trial of all the cross-settings of CtrlMSR (at least bits 1-0), types of BP set in DR7, and null vs. non-null address and data mask. Maybe surprises still in waiting...

Sounds interesting

I wanted to test tonight, but I forgot my usb key with the kernel module at work ... and as I'm to lazy to write a new one, I'll test tomorrow !

Well, it works as intended with instruction breakpoints, but not with data breakpoints.

Instruction execution BPs with address mask are neat indeed.

Did you try breakpoints on port I/O ? They too seemed to work great in my prelim test, UIAM you can combine address mask (for a range of ports) and data mask (for filtering by value) and finally bit zero of the Control mask which inverts the test condition on value. This alone is invaluable for hardware driver testing as well as for some reversing jobs !

Data BPs... are the muddy question. I thought memory data BPs would "naturally" work in the same way as IO ones, which I reflected in my writing . This as you found is not the case They should be investigated further, possibly toggling new bit(s) in the Control MSR. The possibility that memory data BPS are not (yet) implemented / or buggy (but could work on later dies?) should be considered only after more trial of all possibilities IMHO.

Your Athlon is a K7, like mine, right ? What's the CPUID ? I'd love people with recent K8, K10... to test the functions thoroughly.

Also, the AMD PR spokesperson has commented that this is "not secret", and unless he or she was delivering BS, the relevant MSRs are no more PW protected in newer cores - so AMD should be taken at its words and someone, preferably with a work relation to the firm, could ask for a pointer to the relevant datasheets !

Have a good w/e you all, and please rip me those AMD open, we want to see their guts spread on the bench :=)

Just to update you all, I'm advancing, now focusing on the IN/OUT break points.
AMD rocks (in spite of its own P.R.) !!! There turns out to be quite more than we thought.

The Wiki is NOT yet 's been updated, but just so you faithful readers got an idea - repeatable results :

- expanded port address match is ALWAYS performed (unless of course address_mask is 0, signifying match exact port), even if control=0. We can match a whole range of ports, up to 4096 !

- Ctrl=0 (the default) : break on both IN or OUT. Use expanded address match, DON't use data match.

- Ctrl=2 (resp. 3) : break on OUT only ! DO data match (resp. non-match)

- Ctrl=8 (resp.9) : Idem except break on IN only !

- Ctrl=1 : OUT to ANY port, DO data match (this case to be checked further).

- Memory BPs: data match does work here too (only on WRITE to memory). See details in the knowlege library, and you must check as possible and report success or otherwise - absolutely do!

....

'gator....

Not of practical interest.

Wiki updated again, now at revision 0.5. We're seing the light, and yes, memory write breakpoints w/ data_match work perfectly ! The option to break when a particular value is written to any memory or port is tremendous indeed (and should immediately arouse interest from any real man, uh, reverser)

Note my blog http://blogs.mail.ru/mail/czernobyl/ ("http://blogs.mail.ru/mail/czernobyl/") has a nicer version of the write-up with some formatting and eye-candy.

@Fab: Would you double-check ? Note that (I'm not used to gdb) depending on the conditions tested and how the debugger works, it might be better to set the Debug Registers yourself (using that module of yours, maybe...) Relying on the debugger itself, when said debugger is unaware of the new breakpoint conditions, might cause difficulties and headaches. I'm thinking of cases where the debugger's int 1 routine will be invoked recursively because of debugger being unaware of the way to properly unarm expanded BP0.

@Indy " Not of practical interest. "

Praytell, are you talking of the Poll, the thread subject, or both ? Without a hint of what you mean, your contribution to the thread is not of great practical interest either. Be my guest anyway!

Very nice progress Czernobyl! Are you still discovering new things?

Hello,

The thread has been apparently idle for the last couple of days... Weird I haven't received any response from Czernobyl, either, yet I would like to share a tiny tool of mine, capable of monitoring and modifying the secret MSRs' values.

Even though the most involved people (are there any btw?) must have prepared the testing environments on their own, I believe this utility might come in handy for the less kernel-familiar.

AMD_dbg is a small application for Microsoft Windows XP - 7, consisting of two major parts: a user-mode client (displaying a simple TUI for user interaction), and a kernel-mode /server/, actually performing the RDMSR and WRMSR operations. Currently, only 32-bit architecture is supported, and there are no plans of porting this one to x86-64. However, as the program is released together with its source code, you're free to introduce your own improvements / ports, as long as it is compliant to the license (check the LICENSE file for details).

http://j00ru.vexillium.org/dump/amd.png

The package can be downloaded from http://j00ru.vexillium.org/dump/amd_dbg.zip ("http://j00ru.vexillium.org/dump/amd_dbg.zip"). (ZIP, 44kB)

I have performed a couple tests based on czerno's write-up, and the AMD functionality actually works as desired (you can verify the information, by setting the MSRs to custom values, and then launching a standard debugger, like ollydbg).

If there are any bugs that appear to be extremely annoying, please contact me via e-mail or PM. Thanks for your attention

Happy reverse engineering!

Thanks j00ru!, very nice job!, and thanks to Czernobyl for discovering this.

This tool seems to work pretty well in my AMD Athlon Neo 64 MV-40

I received your mail J00ru, sorry for not responding immediately. Life can be a bitch, you know. Whatever, yours looks like a very, very fine tool, I'm eager to give it a test run here ASAP; and sure hope more people will be willing to check expanded debug operations thanks to you.



Wow, good news ! What was the tested CPU model exactly ?

Sure dude, no problem with that.



Yeah, hopefully this is going to be the consequence ;>
Have you given it a test, already?




It's like AMD Phenom II X4 810 ;>

Just a super quick one, but yes it does work under Windows 2k even ! Good work.



Very well ! If it's, like, confirms the hidden features have remained unchanged across proc generations, it will be worth the pain to make them available to a larger public of developers. And hopefully AMD won't destroy its own work, maybe even it will be willing to make the things official and documented, who can tell ?

Thanks, that's just what I wanted to see ;D



Yeah, that's my point of view, as well. Soon I am going to launch some tests on a wider number of AMD processors ;>

BTW - do you think the app is worth being put into CRCEKL, supplementing the existing AMD research category?

Ah this thread makes me regret I have a house full of 20 Intels and not a single AMD to be found. Brb, time to go digging through the garage

Most interesting, I should really get my ass out of those VM's and do some proper reversing it seems. Nice work, seems it's also right here on my AMD Phenom X4 9950 black edition.

Yeah, sure, of course it is, here you go:

http://www.woodmann.com/collaborative/tools/AMD_dbg

Don't know about any "AMD research category" though, so I added it to some other suitable ones as a start anyway.

Update this entry as you please, as usual (and thanks for the nice tool, as usual too).

Thanks!
Well, the "AMD research category" was apparently a product of my imagination... ignore it

When defining processor features orthogonality is often desirable,although it may sometimes conflict with other design goals... Today I added a note on this subject as it applies to reverse engineering the expanded debug features, that I suggest the interested reader had a look at; there is certain lack of orthogonality which might in part be due to incomplete or partial implementation of a design, at least on my old processor model. And here comes your part, in the note I am suggesting ideas of things which while they do not work on my old Sempron, might be working on your newer processors... Please feel free to add your own debugging notes at the bottom of the page :
Super-secret_debug_capabilities_of_AMD_processors_! ("http://www.woodmann.com/collaborative/knowledge/Super-secret_debug_capabilities_of_AMD_processors_!")