nikolatesla20
June 1st, 2004, 14:11
I ripped this method from Microsoft's own Virtual PC product:
I installed VirtualPC, then installed Win2K on it, and then installed VirtualPC inside of VirtualPC. It installs fine but when run it comes up and says "Virtual PC cannot be run inside another virtual machine". Naturally I wanted to turn this off.
After some hunting, I found the code responsible for this: (from Olly)
As you can see, the code performs a CMPXCHG8B EAX instruction, which causes an exception. A real processor handles this exception by going to the exception handler. The virtual processor does not call the exception handler, and thus sets EAX to 1, which causes the program to know something isn't right. The exception handler goes ahead and runs the Virtual software normally.
This could be a good technique for a program to detect if it's running in Virtual PC or not. Just set up an exception handler, and perform a CMPXCHG8B EAX instruction, and if your exception handler never gets called, you are in a virtual machine (at least a Virtual PC machine). I believe you have to hard code the CMPXCHG8B EAX instruction however, since it won't assemble with most compilers (because it's invalid). The bytes are: OF C7 C8.
-nt20
I installed VirtualPC, then installed Win2K on it, and then installed VirtualPC inside of VirtualPC. It installs fine but when run it comes up and says "Virtual PC cannot be run inside another virtual machine". Naturally I wanted to turn this off.
After some hunting, I found the code responsible for this: (from Olly)
Code:
004C1680 55 PUSH EBP
004C1681 8BEC MOV EBP,ESP
004C1683 6A FF PUSH -1
004C1685 68 B0BB5C00 PUSH Virtual_.005CBBB0
004C168A 68 58605700 PUSH Virtual_.00576058 <-- SE handler
004C168F 64:A1 00000000 MOV EAX,DWORD PTR FS:[0]
004C1695 50 PUSH EAX
004C1696 64:8925 00000000 MOV DWORD PTR FS:[0],ESP
004C169D 83EC 08 SUB ESP,8
004C16A0 53 PUSH EBX
004C16A1 56 PUSH ESI
004C16A2 57 PUSH EDI
004C16A3 8965 E8 MOV DWORD PTR SS:[EBP-18],ESP
004C16A6 C745 FC 00000000 MOV DWORD PTR SS:[EBP-4],0
004C16AD 33C0 XOR EAX,EAX
004C16AF 33D2 XOR EDX,EDX
004C16B1 33DB XOR EBX,EBX
004C16B3 33C9 XOR ECX,ECX
004C16B5 0FC7C8 CMPXCHG8B EAX ; Illegal use of register
004C16B8 0100 ADD DWORD PTR DS:[EAX],EAX
004C16BA C745 FC FFFFFFFF MOV DWORD PTR SS:[EBP-4],-1
004C16C1 B0 01 MOV AL,1
004C16C3 8B4D F0 MOV ECX,DWORD PTR SS:[EBP-10]
004C16C6 64:890D 00000000 MOV DWORD PTR FS:[0],ECX
004C16CD 5F POP EDI
004C16CE 5E POP ESI
004C16CF 5B POP EBX
004C16D0 8BE5 MOV ESP,EBP
004C16D2 5D POP EBP
004C16D3 C3 RETN
004C16D4 B8 01000000 MOV EAX,1
004C16D9 C3 RETN
004C16DA 8B65 E8 MOV ESP,DWORD PTR SS:[EBP-18]
004C16DD C745 FC FFFFFFFF MOV DWORD PTR SS:[EBP-4],-1
004C16E4 32C0 XOR AL,AL
As you can see, the code performs a CMPXCHG8B EAX instruction, which causes an exception. A real processor handles this exception by going to the exception handler. The virtual processor does not call the exception handler, and thus sets EAX to 1, which causes the program to know something isn't right. The exception handler goes ahead and runs the Virtual software normally.
This could be a good technique for a program to detect if it's running in Virtual PC or not. Just set up an exception handler, and perform a CMPXCHG8B EAX instruction, and if your exception handler never gets called, you are in a virtual machine (at least a Virtual PC machine). I believe you have to hard code the CMPXCHG8B EAX instruction however, since it won't assemble with most compilers (because it's invalid). The bytes are: OF C7 C8.
-nt20
