Log in

View Full Version : Physical to virtual address translation


SynApsus
November 2nd, 2004, 19:12
Hello, I'm back with my annoying questions !
( thanks for all your answers about Low-level Graphics Programming )
As usual, my question is about Ring0.
I'm coding a driver and I need, to access I/O APIC tables, to translate the PHYSICAL address of this table ( mapped in physical memory at 0xFEC00000 )
into a VIRTUAL or LINEAR one ( 0008:xxxxxxxx ) which I could use to access my values.
On my computer, this address is mapped to 0008:0FFD06000h , but it is NOT the same for all computers. Note that there are LOTS of ways of translating a virtual address into a physical one ( ntoskrnl undocumented functions, examples around the web, sysinternals, cr3 register and PTE/PDE paging, and so on )
But I never found a method to retrieve the virtual from a physical. SoftICE does it with the PHYS command ( which returns, from time by time, more than one virtual addresses for the same physical one ) but due to the extreme poverty of this command's output, I had not been able to locate it in my sice's IDB. And the other sice's command I analysed, which translate the FEC00000h to a virtual address ( there are, you can find them by -deeply -analysing the intobj and irq commands in ntice.sys )
There is a ntoskrnl ( undoc ) function too ( MmGetVirtualForPhysical ), but if she sometimes returns the good values ( the Cr3 value for example ) with FEC00000 it returns just shit. And I did not understand this function too, even with the leaked source code of my favourite OS.
If someone could help me ?

Kayaker
November 3rd, 2004, 01:53
Hi SynApsus,

Hey, this isn't an annoying question... Crack requests, those are annoying.
You seem to indicate MmGetVirtualForPhysical sort of works but doesn't work? The PHYSICAL_ADDRESS value is 64 bit, which you can see from ntoskrnl.

From ntddk.h:

NTKERNELAPI
PVOID
MmGetVirtualForPhysical (
IN PHYSICAL_ADDRESS PhysicalAddress
);

In ntoskrnl, there are 2 dwords pushed to call the function, which indeed returns with a retn 8.

mov eax, cr3
mov [ebp-10h], eax
xor eax, eax
xor edi, edi
push eax
push dword ptr [ebp-10h] ; PhysicalAddress
call MmGetVirtualForPhysical


Perhaps the problem you have in address translation is related to the problem discussed here in relation to MmHighestPhysicalPage?

Playing with Windows /dev/(k)mem
http://www.phrack.org/show.php?p=59&a=16


Kayaker

SynApsus
November 3rd, 2004, 09:39
Yes I saw this in my ntoskrnl code analysis. Of course I pushed a 64 bit value !
Ah, note that the MmGetVirtualForPhysical function does destroy some of the values on the stack, that's why I use it after some stack saving ( 3 lame push 00 for example )
And for the article of CrazyLord, I read it some months ago, as I wasn't interested by the physical memory problem. I just read it one more time, but there is nothing interesting... He wants, as usual, to translate virtual into physical address. His MmHighestPhysicalPage problem isn't very useful to me, because this value is 77ef0000.... a page is 4096 bytes long, and I'm far from it with my FEC00000 ! Nope, I want to know if there is a better method, because I'm scared this is not a very good solution.

Kayaker
November 3rd, 2004, 14:47
What I was wondering by the reply, is there any physical address where the high part of that quad word is anything but 0? You have a physical address apparently higher than MmHighestPhysicalPage. Would this be for example a correct declaration for it or not for MmGetVirtualForPhysical?

typedef LARGE_INTEGER PHYSICAL_ADDRESS;

PhysicalAddress.LowPart = (ULONG) 0xFEC00000;
PhysicalAddress.HighPart = (LONG) 0x00000000;

Neitsa
November 3rd, 2004, 20:43
Hello,

Synapsus, I don't know how to help you exactly (I mean I can't provide you more information than you have) but if you haven't done it already you may take a look at "Undocumented windows 2000" chap 4, where there is an explanation of Linear to Physical translation (with some source code if I remember correctly).

There's also very helpfull informations in "Inside Win 2k" chap 7 => Address translation.

Sorry if I couldn't help you more than that...

Regards, Neitsa.

Kayaker
November 3rd, 2004, 21:55
That's excellent Neitsa. The Schrieber book indicates what the high part of PHYSICAL_ADDRESS might be used for in the explanation of MmGetPhysicalAddress.
"On most i386 systems, the upper 32 bits will be always zero. However, on systems with Physical Address Extension (PAE) enabled and more than 4 GB of memory installed, these bits can assume nonzero values"

And Inside Windows explains PAE
"..there is a special version of the core kernel image (Ntoskrnl.exe) with support for PAE called Ntkrnlpa.exe. (The multiprocessor version is called Ntkrpamp.exe.) To select this PAE-enabled kernel, you must boot with the /PAE switch in Boot.ini."

Assuming no PAE or other reason for how MmGetVirtualForPhysical should be called, Synapsus may have found a limitation of the function.

crUsAdEr
November 3rd, 2004, 22:36
Kayaker... u forgot to tell him about the Command table in sice so he can locate the Phys code in 2 second

Kayaker
November 3rd, 2004, 22:43
Hi Crusader, yeah I did. I was waiting for you to provide your idc script for parsing the NamesIndex and CommandIndex to make it automatic

SynApsus
November 5th, 2004, 05:41
Hmmm... my processor is non-PAE. I have already read this book ( Undeocumendted Win2k ) and I found nothing interesting, only this fucking Linear to Physical translation ! Lol I know it by heart now, but it doesn't help me much !
And of course Kayaker, I tried MmGetVirtualForPhysical with different values for the HighPart of my PHYSICAL_ADDRESS ( 008, 0000, and other descriptors ). Doesn't work.
If you have another idea... because I have tested A LOT of different techniques before posting, and in first the ones you said. Forget MmGetVirtualForPhysical, but if someone could explain me it's source code...
I could perhaps write my own function. Have only the NuMega developers found the answer to this translation problem ? I think not, this should be done by any of us.
And about this command table... I'm VERY interested, although analysing SoftIce is... a lot much difficult than ntoskrnl.exe ! ( not debuggable ) But I'll have to break my ( lame ) limits !

Thanks, if you have another idea...

Opcode
November 5th, 2004, 08:36
Try to do some research about the MmPfnDatabase.
Maybe it helps you

Regards,
Opc0de

SynApsus
November 7th, 2004, 06:25
That's exactly what I have done, because I found your answer very interesting OpCode. But the result of my research is that the MmPfnDatabase structure IS the one used in the function MmGetVirtualForPhysical. Doesn't work !
Here is the display in my IDA for this :
Code:
MmGetVirtualForPhysical proc near

PhysicalAddress = PHYSICAL_ADDRESS ptr 4

mov edx, dword ptr [esp+PhysicalAddress+4]
push esi
mov esi, dword ptr [esp+4+PhysicalAddress]
push 0Ch
pop ecx
mov eax, esi
call _allshr
mov ecx, VirtualAddress
; PfdBase = MmPfnDatabase
lea eax, [eax+eax*2] ; multiplies eax by 3, we can access
; to his entry inside of the Pfn table
; ( PfnEntry is 24 bytes in size )
; with an [ecx+eax*8]
and esi, 0FFFh ; esi = PhysicalAddress.LowPart
mov eax, [ecx+eax*8+4]
; mov eax, MiGetVirtualAddressMappedByPte(Pfn->PteAddress)
; where MiGetVirtualAddressMappedByPte = VirtualAddress ( = PfdBase )
; ( info from the leaked windows sources )
; In fact :
; - we find the entry corresponding to the physical addr in the
; page frame directory ( pfd ) ( and eax=pfn, the page frame number )
; - and we get the member *pteEntry . ( PteAddress )
;
; typedef struct PfdEntry {
; DWORD NextPage,
; void *PteEntry/*PpteEntry,
; DWORD PrevPage,
; DWORD PteReferenceCount,
; void *OriginalPte,
; DWORD Flags;
; } PfdEntry_t;
;
shl eax, 0Ah ; then multiplies it by 2^10 ?! dunno why
add eax, esi ; and add the low part ( offset )
pop esi
retn 8

MmGetVirtualForPhysical endp


It seems that the MmPfnDatabase is a too little table in order to translate this addresses, and is only used for very specific pages ( the pfd itself, that's why cr3 value works ) and the swapped pages.
But perhaps I am wrong... Another idea ?

Kayaker
November 7th, 2004, 23:12
For what it's worth, I encountered the same problems with MmGetVirtualForPhysical as you SynApsus. Perhaps it is time to look towards the Softice PHYS command for answers. To save describing the procedure yet again I wrote up a little summary here

Setting up IDA for analysing Softice functions
http://woodmann.com/forum/showthread.php?p=41319#post41319

Cheers,
Kayaker

bilbo
November 9th, 2004, 03:23
Quote:
[Originally Posted by Kayaker]Perhaps it is time to look towards the Softice PHYS command for answers


Well, the algorithm used by Softice is rather intuitive. It scans all the PTEs related to the current 4GB virtual space, starting from the current Page Directory pointer (CR3 register). For every PTE, it compares the physical address associated to it with the physical address we want to map. As soon as we find a match between the two physical addresses, we have found a new virtual address translating the physical one.
In the case that all PTEs are 4K sized, the number of scanned PTE will be 1024 (PDE entries) * 1024 (PTEs per PDE): about one million!
Let's see the code in detail (DS 3.1):
Code:

;
; phys2virt(EAX=address, ESI=callback)
; callback is called with: EDX=passed context, EAX=translated virtual,
; and can be called more than once
;
byte_1806B db 0 ; global parameter: stop_at_first_translation
sub_1806C proc near

var_4 = dword ptr -4

pusha
mov ebp, esp

sub esp, 4
and eax, 0FFFh ; offset in page
mov [ebp+var_4], eax ; initialize translated virtual

; save_bios_pte_contents()
; see below
call sub_181FD

mov eax, [ebp+1Ch] ; saved EAX after PUSHA: physical address
and eax, 0FFFFF000h ; start of page
mov edx, eax ; physical address to search

; EDI <- virtual address of Page Directory
mov eax, cr3 ; physical address of Page Directory
mov edi, eax
add edi, dword_E15FF ; mapped_zero_phys (see below)

mov dword_115A06, 0 ; cached_selector_start
mov dword_115A02, 0FFFFFFFFh ; cached_selector_end

mov ecx, 400h ; number of PDE's in Page Directory

; outer loop: a loop per PDE
loc_180AD:
; EDI = physptr_to_PDE + mapped_zero_phys
; peekphyslong(EDI=phys+mapped_zero_phys)
call sub_18286
jb loc_18168 ; break if ko

or eax, eax
jz loc_18157 ; null PDE: continue

test eax, 1
jz loc_18157 ; not present PDE: continue

test eax, 80h ; page directory size
jz short loc_18103

; page directory size is 4M: mask page base address
; (physical address of the first byte of a 4K page)
and eax, 0FFC00000h
cmp edx, eax ; CMP passed physical
jb short loc_18157 ; continue if passed physical is lower

add eax, 400000h ; end of 4M page
cmp edx, eax ; CMP passed physical
jnb short loc_18157 ; continue if passed physical is bigger

; found one virtual
sub eax, 400000h ; go back to start of 4M page
; EAX = bias_in_page = passed physical - physical start of page
sub eax, edx
neg eax

; EAX = current translated virtual + bias_in_page
add eax, [ebp+var_4]

push edx
mov edx, [ebp+14h] ; restore saved EDX
call [ebp+4h] ; saved ESI after PUSHA: callback
pop edx

; see if we must continue or break
cmp ds:byte_1806B, 1 ; global parameter: stop_at_first_translation
jz short loc_18168 ; break
jmp short loc_18157 ; continue

; page directory size is 4K
loc_18103:
; mask PTE lower bits
and eax, 0FFFFF000h
add eax, dword_E15FF ; mapped_zero_phys

push ecx
push edi
mov edi, eax

mov ecx, 400h ; number of PTE's
xor esi, esi ; init bias to add to translated virtual (4K per PTE)
; inner loop
loc_18119:
; peekphyslong(EDI=phys+mapped_zero_phys)
call sub_18286
jb short loc_1814A ; continue inner loop if ko

test eax, 1 ; present bit
jz short loc_1814A ; continue inner loop if PTE not present

and eax, 0FFFFF000h ; physical address of first byte in 4K page
cmp eax, edx ; CMP passed physical
jnz short loc_1814A ; continue inner loop

; matching found
mov eax, [ebp+var_4] ; update translated virtual
; add bias to base translated virtual
or eax, esi

push edx
mov edx, [ebp+14h] ; restore saved EDX
call [ebp+4h] ; saved ESI: callback
pop edx

cmp ds:byte_1806B, 0 ; global parameter: stop_at_first_translation
jz short loc_1814A ; continue inner loop
; done with inner loop
pop edi
pop ecx
jmp short loc_18168 ; break from outer loop

; continue inner loop
loc_1814A:
add esi, 1000h ; 4K more to base translated virtual
add edi, 4 ; update physical PTE pointer
loop loc_18119 ; decrement ECX (number of PTEs)
; done with inner loop
pop edi
pop ecx

loc_18157: ; continue to next PDE (outer loop)
add [ebp+var_4], 400000h ; update translated virtual (+= 4M)
add edi, 4 ; update physptr to PDE
dec ecx ; update remaining PDEs
jnz loc_180AD ; continue

; done with outer loop
loc_18168:
; restore_bios_pte_contents()
call sub_18217
mov esp, ebp
popa ; matches initial PUSHA
retn
sub_1806C endp ; sp = 20h


Some explanations are required regarding mapped_zero_phys and save_bios_pte_contents(), before analyzing the tricky subroutine peekphyslong(), which will map a physical address (the current PTE pointer) to a virtual one, in order to read it.

At init time a MmMapIoSpace(0, 0, 0x100000, 0) is executed (the first two zeros are the physical address to map), and the return value, a virtual address, is saved in mapped_zero_phys: this is the start address of a virtual range mapping the physical range 0-100000. Furthermore, a particular PTE is located which initially maps the BIOS space (F0000-100000): its virtual address is mapped_zero_phys+0xF0000; we'll call it mapped_bios. Its virtual PTE pointer is calculated:
C0000000 (start of consecutive PTEs) + (mapped_bios/4K) * 4 bytesperPTE; we'll call it pBIOS_PTE.
Now we can understand how peekphyslong() works. We have a physical address; we build a new PTE with it; we replace this PTE into pBIOS_PTE; we can now read the 4K page at virtual address mapped_bios. Before and after this operation we issue save_bios_pte_contents() and restore_bios_pte_contents().
Let's see the code:
Code:

;
; peekphyslong(EDI=phys+mapped_zero_phys)
;
sub_18286 proc near
push ebx
push edi
; subtract back mapped_zero_phys
sub edi, dword_E15FF
mov ebx, offset sub_6D3E3 ; callback: fastpeeklong(EDI=address)
; map_phys_to_virt_and_do_callback(
; EDI=physaddr, EBX=callback)
call sub_18231
pop edi
pop ebx
retn
sub_18286 endp

;
; map_phys_to_virt_and_do_callback(EDI=physaddr, EBX=callback)
; callback is called with mappedvirt in EDI
;
sub_18231 proc near
push ecx
push edx
push edi
mov ecx, edi ; physaddr

mov edi, dword_D9EB1 ; pBIOS_PTE (it holds F017B: phys F0000)
or edi, edi
jnz short loc_18248

; pBIOS_PTE null: simply add mapped_zero_phys and call callback
; never follow this way!
add ecx, dword_E15FF ; mapped_zero_phys
jmp short loc_1827E ; call callback(EDI<-ECX mappedvirt)

loc_18248:
; pBIOS_PTE not null:
; build 4K PTE
mov edx, ecx
and edx, 0FFFFF000h ; physaddr, start of page
or dl, 7 ; set low pte bits: userpriv|readwrite|present
cmp dword_E4CE4, edx ; saved_bios_pte
jz short loc_18272 ; jump if we have already set it

; set built PTE in pBIOS_PTE
push eax
; pokelong_(EDI=addr, EAX=value) must return !CY
mov eax, edx
call sub_6D5B4
pop eax
jb short loc_18282 ; return if ko

mov dword_E4CE4, edx ; update saved_bios_pte

; flush cache
mov edx, cr3
mov cr3, edx

; passed physaddr: add page offset
loc_18272:
and ecx, 0FFFh
add ecx, dword_D9EB5 ; add mapped_bios to convert to virtual

; call callback(EDI<-ECX)
loc_1827E:
mov edi, ecx
call ebx

; return
loc_18282:
pop edi
pop edx
pop ecx
retn
sub_18231 endp


I hope someone could follow this (rather obfuscated) explanation. Sorry, mates, but I couldn't find easier words.

Best regards, bilbo

SynApsus
November 9th, 2004, 06:25
Aha ! Thanks Bilbo !
Yes that is the code I found for the PHYS command, by analysing the IRQ command and the init of ntice. But I was not sure it was this function, and I had not been able to analyse it...
Thanks for this great piece of knowledge, I'll use it, write a tut and a proggie with many thanks !

Regards and respect, Syn

Opcode
November 9th, 2004, 06:28
Very useful information!

Thnx bilbo!

SynApsus:
Continue to make more "annoying questions" !!!

Regards,
Opcode

SynApsus
November 13th, 2004, 17:45
Just a last question, for Bilbo : in what way are the Save_Bios_Pte_Content and Restore_bios_Pte_Contents calls useful ?
Or can I build my proc without them ?
Because as I was trying to rip the code from ntice to my prog, there was no problem with translating the main peekphyslong code and around, but for Save_Bios_Pte_Content it's nearly impossible : really a LOT of constants and values used, and it would take entire days.

bilbo
November 15th, 2004, 03:45
AFAIK, the routines I called Save_Bios_Pte_Contents and Restore_Bios_Pte_Contents are not necessary.
I understand them as a sort of cache for the last mapped address: simply to avoid consecutive mappings for the same memory page.
Regards, bilbo

SynApsus
November 20th, 2004, 20:06
Hmm, I try I try... but I don't seem to be able to fix my problem using this; it's too complex for the time I can spend on my computer nowadays ( very hard @ school ).
I'll see it later... thanks anyway.

bilbo
November 23rd, 2004, 02:35
Quote:
[Originally Posted by SynApsus]Hmm, I try I try... but I don't seem to be able to fix my problem using this

By the way, what is your problem? Have you some code to share?
Best wishes for you school results. bilbo