View Full Version : ntfs bootloader question...
Hi reteam,
I have completely reversed the ntfs bootloader code on my xp partition (with the aid of the Linux-NTFS documentation (http://linux-ntfs.sourceforge.net/ntfs/)), and was wondering if someone with access to the sources for the ntfs bootcode could answer a question regarding some of the code I have reversed...
Since I don't have any sources myself, I'll have to describe the code in pseudo-assembly language (with pseudo-names too :blink: ):
* *(initial bootsector code that loads the rest of bootstrap code...)
0D00:026A
* *call GetInt13SectorCount
*; (calculations for BytesPerCluster, BytesPerRecord, SectorsPerRecord...)
* *call Build$MFTRecordDescriptorList *; <- Question regarding this routine below...
*; (initialize $index_root, $index_allocation, $bitmap (and other) buffer pointers...)
*; (proceed to locate and load ntldr)
Okay so I understand the function "Build$MFTRecordDescriptorList" builds an array/list describing the location on disk of the '$MFT' base file record and all its extents (if any). It first reads the base file record, and proceeds to look for an $attribute_list attribute. If it found one, it looks to the unnamed $data attribute entries inside $attribute_list, and if more than one is found, it locates the file record extents described by these additional $data attributes, and adds their disk location information to the array/list. What I don't understand is why it skips the first $data attribute found in the $attribute_list attribute?
According to the linux-ntfs docs, all attributes in an $attribute_list describe file record extents belonging to the base file record that has the $attribute_list. The code is locating additional extents of the $MFT record so other functions can perform VCN to LCN translation, but I don't know why it skips the first $data attribute found...
Are there any comments in the sources that describe exactly why this code works the way it does? Can someone who has access to the sources provide me with some info regarding this function? I have it fully reversed, so if you need a better description of the function I'm referring to, please let me know and I'll post some more pseudo-assembly...
AndreaGeddon
06-04-2005, 07:18 AM
Originally posted by rwid@Jun 3 2005, 07:32 AM
* *(initial bootsector code that loads the rest of bootstrap code...)
0D00:026A
* *call GetInt13SectorCount
; (calculations for BytesPerCluster, BytesPerRecord, SectorsPerRecord...)
* *call Build$MFTRecordDescriptorList ; <- Question regarding this routine below...
; (initialize $index_root, $index_allocation, $bitmap (and other) buffer pointers...)
; (proceed to locate and load ntldr)
What I don't understand is why it skips the first $data attribute found in the $attribute_list attribute?
well first i would advise you to consult the book "File system internals" of Rajeev Nagar, it's a good book. Unluckily linux ntfs documentation is poor and imprecise, Nagar's book is better but contains some imprecisions too.
After this, you are in 0D00:0200 + 6A, so you are now in the part where the bootloader has relocated itself, and is now executing file system specific code. The code you are viewing reads ntldr and maps it in memory at 2000:0000, as you correctly say it searches attribute list and reads $data from it. I don't see the first attribute being skipped, maybe i misunderstood your problem? If you can post more specific code i hope i can help you :)
Bye!
AndreaGeddon
Ewell first i would advise you to consult the book "File system internals" of Rajeev Nagar, it's a good book.
Thanks for the reference (the book costs a bit though :( )
I found the linux-ntfs docs described the structures needed to understand and reverse the ntfs boot code binary (though I understand the docs are somewhat incomplete).
First here's my understanding of the function I call Build$MFTRecordDescriptorList.
A file record is made up of a Base File Record (BFR) and zero or more record extents (located through an $ATTRIBUTE_LIST attribute). In order to locate any file record on disk, we look to the unnamed $DATA attribute in the $MFT file record. If the $MFT file has an $ATTRIBUTE_LIST, then we must locate the file record extents that contain any additional unnamed $DATA attributes, before we can find any other file records.
So this function builds a list describing the location on disk of the $MFT BFR and any extents holding unnamed $DATA attributes. The list is used in translating Virtual Cluster Numbers (VCNs) within the $MFT file's unnamed $DATA attribute data to Logical Cluster Numbers (LCNs), therefore allowing us to locate the LCN on disk of any file record in the MFT. The list can describe fragmented file record extents.
Below is the disassembly for the function I'm referring to (from my ida database file). The symbol names I have used may not be all that helpful (the comments might not be either).
(Note seg000 would actually be segment 0D00 in memory. The disassembly comes from a binary image of the first 16 logical sectors of my XP boot partition.)
seg000:06D0; ¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦ S U B R O U T I N E ¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦
seg000:06D0
seg000:06D0
seg000:06D0 Build$MFTRecDescList proc near * * * * ; CODE XREF: Main+56p
seg000:06D0 * * * * * * * * * * push es
seg000:06D1 * * * * * * * * * * push ds
seg000:06D2 * * * * * * * * * * pushad
seg000:06D4 * * * * * * * * * * mov eax, 1
seg000:06DA * * * * * * * * * * mov ds:NumberOf$MFTRecords, eax
seg000:06DE * * * * * * * * * * mov eax, ds:MemoryAllocOffset
seg000:06E2 * * * * * * * * * * add eax, ds:BytesPerRecord
seg000:06E7 * * * * * * * * * * mov ds:pRecordBufferForLCNtoVCN, eax
seg000:06EB * * * * * * * * * * add eax, ds:BytesPerRecord
seg000:06F0 * * * * * * * * * * mov ds:pEndOf$MFTRecordList, eax
seg000:06F4 * * * * * * * * * * mov eax, dword ptr ds:LCNof$MFT
seg000:06F8 * * * * * * * * * * movzx ebx, ds:SectorsPerCluster
seg000:06FE * * * * * * * * * * mul ebx
seg000:0701 * * * * * * * * * * mov ebx, ds:pEndOf$MFTRecordList
seg000:0706 * * * * * * * * * * mov [bx], eax * * *; Add LSN to $MFT record descriptor list.
seg000:0709 * * * * * * * * * * mov ds:LSNToRead, eax
seg000:070D * * * * * * * * * * add bx, 4
seg000:0710 * * * * * * * * * * mov eax, ds:SectorsPerRecord
seg000:0714 * * * * * * * * * * mov [bx], eax * * *; Add sector count to $MFT record descriptor list.
seg000:0717 * * * * * * * * * * mov ds:SectorReadCount, ax
seg000:071A * * * * * * * * * * add bx, 4
seg000:071D * * * * * * * * * * mov ds:pEndOf$MFTRecordList, ebx
seg000:0722 * * * * * * * * * * mov ebx, ds:MemoryAllocOffset
seg000:0727 * * * * * * * * * * push ds
seg000:0728 * * * * * * * * * * pop es
seg000:0729 * * * * * * * * * * call ReadSectorsIntoMemory; Read $MFT BFR into memory.
seg000:0729
seg000:072C * * * * * * * * * * mov edi, ebx
seg000:072F * * * * * * * * * * call FixUpdateSeqArrayWords
seg000:072F
seg000:0732 * * * * * * * * * * mov eax, ds:MemoryAllocOffset
seg000:0736 * * * * * * * * * * mov ebx, $ATTRIBUTE_LIST
seg000:073C * * * * * * * * * * mov ecx, 0
seg000:0742 * * * * * * * * * * mov edx, 0
seg000:0748 * * * * * * * * * * call FindAttributesHeader
seg000:0748
seg000:074B * * * * * * * * * * or *eax, eax
seg000:074E * * * * * * * * * * jz *$MFTRecordDescListIsComplete
seg000:074E
seg000:0752 * * * * * * * * * * mov ebx, eax
seg000:0755 * * * * * * * * * * push ds
seg000:0756 * * * * * * * * * * pop es
seg000:0757 * * * * * * * * * * mov edi, ds:p$ATTRIBUTE_LIST
seg000:075C * * * * * * * * * * call ReadAttribData
seg000:075C
seg000:075F * * * * * * * * * * mov ebx, ds:p$ATTRIBUTE_LIST
seg000:0764
seg000:0764 FindFirst$DATA: * * * * * * * * * * * *; CODE XREF: Build$MFTRecDescList+A2j
seg000:0764 * * * * * * * * * * cmp [bx+ATTRIBUTE_LIST_Entry.Type], $DATA
seg000:076B * * * * * * * * * * jz *FindNext$DATA
seg000:076B
seg000:076F * * * * * * * * * * add bx, [bx+ATTRIBUTE_LIST_Entry.RecordLength]
seg000:0772 * * * * * * * * * * jmp short FindFirst$DATA
seg000:0772
seg000:0774; ---------------------------------------------------------------------------
seg000:0774
seg000:0774 FoundAnother$DATA: * * * * * * * * * * ; CODE XREF: Build$MFTRecDescList+194j
seg000:0774 * * * * * * * * * * push ebx
seg000:0776 * * * * * * * * * * mov eax, dword ptr [bx+ATTRIBUTE_LIST_Entry.BFRofAttribute]
seg000:077A * * * * * * * * * * mul ds:SectorsPerRecord
seg000:077F * * * * * * * * * * push eax * * * * * ; Preserve VSN of record/fragment.
seg000:0781 * * * * * * * * * * xor edx, edx
seg000:0784 * * * * * * * * * * movzx ebx, ds:SectorsPerCluster
seg000:078A * * * * * * * * * * div ebx
seg000:078D * * * * * * * * * * push edx
seg000:078F * * * * * * * * * * call GetLCNofRecordFromVCNinMFT
seg000:078F
seg000:0792 * * * * * * * * * * or *eax, eax
seg000:0795 * * * * * * * * * * jz *DiskReadError
seg000:0795
seg000:0799 * * * * * * * * * * mov ecx, ds:SectorsPerRecord
seg000:079E * * * * * * * * * * movzx ebx, ds:SectorsPerCluster
seg000:07A4 * * * * * * * * * * mul ebx
seg000:07A7 * * * * * * * * * * pop edx
seg000:07A9 * * * * * * * * * * add eax, edx * * * ; eax = LSN of record.
seg000:07AC * * * * * * * * * * mov ebx, ds:pEndOf$MFTRecordList
seg000:07B1 * * * * * * * * * * mov [bx], eax * * *; Add LSN to $MFT record descriptor list.
seg000:07B4 * * * * * * * * * * add bx, 4
seg000:07B7 * * * * * * * * * * movzx eax, ds:SectorsPerCluster
seg000:07BD * * * * * * * * * * sub eax, edx * * * ; eax = Sector count from LSN to the end of the cluster.
seg000:07C0 * * * * * * * * * * cmp eax, ecx
seg000:07C3 * * * * * * * * * * jbe @@1 * * * * * *; Jump if sector count is less or equal to sectors
seg000:07C3 * * * * * * * * * * * * * * * * * * * *; per record.
seg000:07C3
seg000:07C7 * * * * * * * * * * mov eax, ecx * * * ; If sector count is larger than sectors per record,
seg000:07C7 * * * * * * * * * * * * * * * * * * * *; then use sectors per record as sector count.
seg000:07CA
seg000:07CA @@1: * * * * * * * * * * * * * * * * * ; CODE XREF: Build$MFTRecDescList+F3j
seg000:07CA * * * * * * * * * * mov [bx], eax * * *; Add sector count to $MFT record descriptor list.
seg000:07CD
seg000:07CD CheckIfHaveRecord: * * * * * * * * * * ; CODE XREF: Build$MFTRecDescList+156j
seg000:07CD * * * * * * * * * * * * * * * * * * * *; Build$MFTRecDescList+179j
seg000:07CD * * * * * * * * * * sub ecx, eax * * * ; ecx = Number of remaining sectors of record to locate.
seg000:07D0 * * * * * * * * * * pop edx * * * * * *; edx = VSN of record/fragment.
seg000:07D2 * * * * * * * * * * jz *HaveRecord
seg000:07D2
seg000:07D6 * * * * * * * * * * add eax, edx * * * ; eax = VSN of next record fragment.
seg000:07D9 * * * * * * * * * * push eax * * * * * ; Preserve VSN of next record fragment.
seg000:07DB * * * * * * * * * * xor edx, edx
seg000:07DE * * * * * * * * * * movzx ebx, ds:SectorsPerCluster
seg000:07E4 * * * * * * * * * * div ebx
seg000:07E7 * * * * * * * * * * push ecx
seg000:07E9 * * * * * * * * * * call GetLCNofRecordFromVCNinMFT
seg000:07E9
seg000:07EC * * * * * * * * * * pop ecx
seg000:07EE * * * * * * * * * * or *eax, eax
seg000:07F1 * * * * * * * * * * jz *DiskReadError
seg000:07F1
seg000:07F5 * * * * * * * * * * movzx ebx, ds:SectorsPerCluster
seg000:07FB * * * * * * * * * * mul ebx * * * * * *; eax = LSN of next record fragment.
seg000:07FE * * * * * * * * * * mov ebx, ds:pEndOf$MFTRecordList
seg000:0803 * * * * * * * * * * mov edx, [bx]
seg000:0806 * * * * * * * * * * add bx, 4
seg000:0809 * * * * * * * * * * add edx, [bx] * * *; edx = LSN following last record fragment.
seg000:080C * * * * * * * * * * cmp edx, eax
seg000:080F * * * * * * * * * * jnz FragmentIsNotContiguous
seg000:080F
seg000:0813 * * * * * * * * * * movzx eax, ds:SectorsPerCluster
seg000:0819 * * * * * * * * * * cmp eax, ecx
seg000:081C * * * * * * * * * * jbe @@2 * * * * * *; Jump if sectors per cluster is less or equal to
seg000:081C * * * * * * * * * * * * * * * * * * * *; sectors per record. (Use sectors per cluster for
seg000:081C * * * * * * * * * * * * * * * * * * * *; sector count)
seg000:081C
seg000:0820 * * * * * * * * * * mov eax, ecx * * * ; If sectors per cluster is larger than sectors per
seg000:0820 * * * * * * * * * * * * * * * * * * * *; record, then use sectors per record as sector count.
seg000:0823
seg000:0823 @@2: * * * * * * * * * * * * * * * * * ; CODE XREF: Build$MFTRecDescList+14Cj
seg000:0823 * * * * * * * * * * add [bx], eax * * *; Add sector count to last fragments sector count in
seg000:0823 * * * * * * * * * * * * * * * * * * * *; the $MFT record descriptor list.
seg000:0826 * * * * * * * * * * jmp short CheckIfHaveRecord
seg000:0826
seg000:0828; ---------------------------------------------------------------------------
seg000:0828
seg000:0828 FragmentIsNotContiguous: * * * * * * * ; CODE XREF: Build$MFTRecDescList+13Fj
seg000:0828 * * * * * * * * * * add bx, 4
seg000:082B * * * * * * * * * * mov ds:pEndOf$MFTRecordList, ebx
seg000:0830 * * * * * * * * * * mov [bx], eax * * *; Add LSN of fragment to $MFT record descriptor list.
seg000:0833 * * * * * * * * * * add bx, 4
seg000:0836 * * * * * * * * * * movzx eax, ds:SectorsPerCluster
seg000:083C * * * * * * * * * * cmp eax, ecx
seg000:083F * * * * * * * * * * jbe @@3 * * * * * *; Jump if sectors per cluster is less or equal to
seg000:083F * * * * * * * * * * * * * * * * * * * *; sectors per record. (Use sectors per cluster for
seg000:083F * * * * * * * * * * * * * * * * * * * *; sector count)
seg000:083F
seg000:0843 * * * * * * * * * * mov eax, ecx * * * ; If sectors per cluster is larger than sectors per
seg000:0843 * * * * * * * * * * * * * * * * * * * *; record, then use sectors per record as sector count.
seg000:0846
seg000:0846 @@3: * * * * * * * * * * * * * * * * * ; CODE XREF: Build$MFTRecDescList+16Fj
seg000:0846 * * * * * * * * * * mov [bx], eax * * *; Add sector count to $MFT record descriptor list.
seg000:0849 * * * * * * * * * * jmp short CheckIfHaveRecord
seg000:0849
seg000:084B; ---------------------------------------------------------------------------
seg000:084B
seg000:084B HaveRecord: * * * * * * * * * * * * * *; CODE XREF: Build$MFTRecDescList+102j
seg000:084B * * * * * * * * * * add bx, 4
seg000:084E * * * * * * * * * * inc ds:NumberOf$MFTRecords
seg000:0853 * * * * * * * * * * mov ds:pEndOf$MFTRecordList, ebx
seg000:0858 * * * * * * * * * * pop ebx
seg000:085A
seg000:085A FindNext$DATA: * * * * * * * * * * * * ; CODE XREF: Build$MFTRecDescList+9Bj
seg000:085A * * * * * * * * * * add bx, [bx+ATTRIBUTE_LIST_Entry.RecordLength]
seg000:085D * * * * * * * * * * cmp [bx+ATTRIBUTE_LIST_Entry.Type], $DATA
seg000:0864 * * * * * * * * * * jz *FoundAnother$DATA
seg000:0864
seg000:0868
seg000:0868 $MFTRecordDescListIsComplete: * * * * *; CODE XREF: Build$MFTRecDescList+7Ej
seg000:0868 * * * * * * * * * * popad
seg000:086A * * * * * * * * * * nop
seg000:086B * * * * * * * * * * pop ds
seg000:086C * * * * * * * * * * pop es
seg000:086D * * * * * * * * * * retn
seg000:086D
seg000:086D Build$MFTRecDescList endp
seg000:086D
seg000:086E
See FindFirst$DATA label above. Here the function locates the first unnamed $DATA entry in the $ATTRIBUTE_LIST attribute of the $MFT BFR. When found, it jumps to FindNext$DATA which looks for another $DATA entry, and if found only then does it jump to FoundAnother$DATA where the second $DATA entry (and any further $DATA entries) are then used to locate their their respective file records, and have those records (possibly in fragments) added to the list (in the form of an LSN dword and a NumberOfSectors dword).
Why is the first $DATA entry in an $ATTRIBUTE_LIST skipped here? Does this code assume that the first $DATA entry references the Base File Record, which has already been added to the list? Why would the first one be needed if it is assumed to always reference the BFR, where a $DATA attribute in the BFR (outside of the $ATTRIBUTE_LIST) must therefore exist?
I'm not sure I'm explaining myself properly, or if it is clear what I am asking... sorry :unsure:
Thanks for your last reply AndreaGeddon, I've read your article 'UNDERSTANDING WINDOWS 2K SOURCES (part 1)' - it's given me a few symbolic names to use as I'm reversing the osloader.exe portion of ntldr :) .
I've been wanting to learn how to write a bootloader (with ntfs read support) and basic protected mode code (eg video/disk/keyboard support through BIOS calls made using mode-switches or V86 mode) and I thought I might learn a great deal by studying the windows binaries (ntfs bootsector & ntldr). If I get stuck on any parts reversing the code, could you answer other questions I might have?
Thanks
rwid.
.
AndreaGeddon
06-05-2005, 09:29 AM
Thanks for the reference (the book costs a bit though
maybe you can find the ebook :P
Why is the first $DATA entry in an $ATTRIBUTE_LIST skipped here?
which os are you looking at? My code is different, thats why i couldnt understand your problem. That is, in your code i see what you say, my code (nt4) instead looks like this:
FindFirst$DATA:
* * * * * * * * * * cmp [bx+ATTRIBUTE_LIST_Entry.Type], $DATA
* * * * * * * * * * jz *FindNext$DATA
* * * * * * * * * * add bx, [bx+ATTRIBUTE_LIST_Entry.RecordLength]
* * * * * * * * * * jmp short FindFirst$DATA
FindNext$DATA:
* * * * * * * * * * cmp [bx+ATTRIBUTE_LIST_Entry.Type], $DATA
* * * * * * * * * * jne $MFTRecordDescListIsComplete
* * * * * * * * * * ...
then there is the loop that scans the $data entries to read each referenced file record segment, so here the first $data is not skipped. I too don't understand why the first $data is skipped in your code. However if i recall well usually for every file ONE $data is present and holds informations on VCNs to load frs. I am searching among docs but i can't find why first $data is skipped
I've been wanting to learn how to write a bootloader (with ntfs read support) and basic protected mode code (eg video/disk/keyboard support through BIOS calls made using mode-switches or V86 mode) and I thought I might learn a great deal by studying the windows binaries (ntfs bootsector & ntldr). If I get stuck on any parts reversing the code, could you answer other questions I might have?
well for boot code i can advise you the john fine boot, its fast and handy and is easy to understand. For the other things
(peripheral io) you can read intel manuals, every information you need is there :)
If you ave any question you can of course post :)
Bye!
AndreaGeddon
maybe you can find the ebook :P
I'll try though I'm not that resourceful
which os are you looking at? My code is different, thats why i couldnt understand your problem. That is, in your code i see what you say, my code (nt4) instead looks like this:
FindFirst$DATA:
* * * * * * * * * * cmp [bx+ATTRIBUTE_LIST_Entry.Type], $DATA
* * * * * * * * * * jz *FindNext$DATA
* * * * * * * * * * add bx, [bx+ATTRIBUTE_LIST_Entry.RecordLength]
* * * * * * * * * * jmp short FindFirst$DATA
FindNext$DATA:
* * * * * * * * * * cmp [bx+ATTRIBUTE_LIST_Entry.Type], $DATA
* * * * * * * * * * jne $MFTRecordDescListIsComplete
* * * * * * * * * * ...
The code I am viewing/reversing came from my windows xp ntfs partition. So it's written for the nt5 ntfs version. Why it's different from the nt4 code is puzzling me! :blink: The only logic I can see in the nt5 code skipping the first $data is that it assumes the 'BFRofAttribute' member of the first $data is the $MFT base file record itself. If this is the case, there must always be a $data attribute in the $MFT base file record even though it's got an $attribute_list with $data(s) as well - the first $data in $attribute_list just references this base file record, which is already processed in the code before scanning $attribute_list.
Or maybe its a bug in the nt5 ntfs bootloader :rolleyes:
In any case I'm writing an ntfs bootloader for xp, so I'll have to use the same method the xp ntfs bootloader uses, without exactly understanding why it does what it does (the first $data skip thingy I was wondering about).
However if i recall well usually for every file ONE $data is present and holds informations on VCNs to load frs.
So part of the $data attribute is always present as an ordinary attribute in the BFR even though the record may contain an $attribute_list with $data too? As I said above, does the first $data reference the BFR which it is in?
I am searching among docs but i can't find why first $data is skipped.
Neither can I!
well for boot code i can advise you the john fine boot, its fast and handy and is easy to understand.
Thanks I'll look at it.
Partly the reason I'm looking into windows boot code is that I can see what bios calls *windows* relies on during the boot phase. I figured this way I can avoid using bios functions that may be unsupported in some other bios. Also as I'm now looking into ntldr disassembly, I see how windows is calling bios functions from protected-mode - it switches back to real-mode! I was supprised as I thought it would be using virtual86 mode for this.
For the other things
(peripheral io) you can read intel manuals, every information you need is there :)
Yes I have some intel and other documents on some hardware i/o eg apic.
AndreaGeddon
06-06-2005, 08:13 AM
* The only logic I can see in the nt5 code skipping the first $data is that it assumes the 'BFRofAttribute' member of the first $data is the $MFT base file record itself. If this is the case, there must always be a $data attribute in the $MFT base file record even though it's got an $attribute_list with $data(s) as well - the first $data in $attribute_list just references this base file record, which is already processed in the code before scanning $attribute_list.
yes it could be an explanation
Or maybe its a bug in the nt5 ntfs bootloader* :rolleyes:
hehe i don't think so, since the boot code works really well :)
In any case I'm writing an ntfs bootloader for xp, so I'll have to use the same method the xp ntfs bootloader uses, without exactly understanding why it does what it does (the first $data skip thingy I was wondering about).
well you could use fat32, it's very easy :) However reading ntfs is not difficult, the hard thing is writing!
So part of the $data attribute is always present as an ordinary attribute in the BFR even though the record may contain an $attribute_list with $data too? As I said above, does the first $data reference the BFR which it is in?
i must check some doucmentation, i've worked on it a lot of time ago. For sure there must be one $data, and if it references the BFR itself then why in nt4 its read and not skipped?
Partly the reason I'm looking into windows boot code is that I can see what bios calls *windows* relies on during the boot phase. I figured this way I can avoid using bios functions that may be unsupported in some other bios. Also as I'm now looking into ntldr disassembly, I see how windows is calling bios functions from protected-mode - it switches back to real-mode! I was supprised as I thought it would be using virtual86 mode for this.
in the boot process it makes jumps among rm/pm, then the kernel is loaded and mapped, and rm is no more used! Normally the execution of DOS/Win16 exes is executed under emulated environments, ntvdm and wow take care of this, i don't remember if virtual86 is used with certain dos executables or is avoided at all. W98 windows used dos boxes implemented with virtual86 tasks if i recall well (long time no see win98!).
However once boot phase is over, the kernel should initialize and you no longer need to access bios routines, a 32bit kernel should not have the need to care about 16bit code, except for retrocompatibility.
Bye!
AndreaGeddon
hehe i don't think so, since the boot code works really well :)
Okay I'll take your word for it. Hmm I was just wondering about the likelihood of an $attribute_list attribute existing in the $MFT file record - the Master File Table would have to be sooo fragmented that the data runs (of the non-resident unnamed $data) describing the $MFT would have to overflow the $MFT file record. I wonder if this has this ever occurred in the real world...
well you could use fat32, it's very easy :) However reading ntfs is not difficult, the hard thing is writing!
I'll stick with ntfs read support for my bootloader, as I have now learnt how the ntfs bootsector locates and loads ntldr. Ok so I won't go near ntfs write support for my first kernel ;)
in the boot process it makes jumps among rm/pm, then the kernel is loaded and mapped, and rm is no more used!
However once boot phase is over, the kernel should initialize and you no longer need to access bios routines, a 32bit kernel should not have the need to care about 16bit code, except for retrocompatibility.
I realise once the bootloader portion is finished, the hal and drivers access the hardware directly, but I was unsure how ntldr accessed the hardware early on in the boot phase just after switching to pmode - of course now I know :D
Thanks for responding to my question AndreaGeddon, hope you don't mind if I have more regarding ntldr... maybe I should just pm you instead since my questions probably aren't constructive for any one else but me :P .
cheers, rwid.
.
redhawk23
02-02-2006, 11:20 AM
Hello rwid,
I see this is a pretty old message, trying to get back to your question whether the $MFT file record can become so fragmented in the real world that its Attribute list becomes external, the answer is yes, we have been facing a lot of problems because of it. And when the MFT's Attribute list becomes external, the system fails to boot. I suspect NTLDR is unable to boot up if the MFT's attribute list becomes external. Can you tell me why this happens? or any data to support this will be great.
Thanks,
Raj
Originally posted by rwid@Jun 7 2005, 01:53 AM
Okay I'll take your word for it. Hmm I was just wondering about the likelihood of an $attribute_list attribute existing in the $MFT file record - the Master File Table would have to be sooo fragmented that the data runs (of the non-resident unnamed $data) describing the $MFT would have to overflow the $MFT file record. I wonder if this has this ever occurred in the real world...
I'll stick with ntfs read support for my bootloader, as I have now learnt how the ntfs bootsector locates and loads ntldr. Ok so I won't go near ntfs write support for my first kernel ;)
I realise once the bootloader portion is finished, the hal and drivers access the hardware directly, but I was unsure how ntldr accessed the hardware early on in the boot phase just after switching to pmode - of course now I know* :D
Thanks for responding to my question AndreaGeddon, hope you don't mind if I have more regarding ntldr... maybe I should just pm you instead since my questions probably aren't constructive for any one else but me* :P .
cheers, rwid.
.
1067
I should note that if the problem lies in NTLDR's handling of the $MFT file record, not in the bootsector's code, then I probably can't help you, as I haven't reversed/studied ntldr's ntfs-handling code.
If it's an NT4 bootsector, you might be having this problem --> Windows NT Does Not Boot with Highly Fragmented MFT (http://support.microsoft.com/kb/q228734/)
0x517A5D
02-03-2006, 11:22 PM
Originally posted by redhawk23+Feb 2 2006, 08:20 AM--><div class='quotetop'>QUOTE(redhawk23 @ Feb 2 2006, 08:20 AM)</div>Hello rwid,
I see this is a pretty old message, trying to get back to your question whether the $MFT file record can become so fragmented in the real world that its Attribute list becomes external, the answer is yes, we have been facing a lot of problems because of it. And when the MFT's Attribute list becomes external, the system fails to boot. I suspect NTLDR is unable to boot up if the MFT's attribute list becomes external. Can you tell me why this happens? or any data to support this will be great.
Thanks,
Raj
1271
[/b]
<!--QuoteBegin-rwid@Feb 2 2006, 08:39 PM
...
My original question in this topic related to the handling of the $data attributes present in an $attribute_list of the $MFT base file record. In the bootsector binary that I disassembled (from an XP formatted partititon), the first $data attribute of the $MFT BFR's $attribute_list is skipped/ignored. These $data attributes would be there to describe the extents of the $MFT file record, and I couldn't understand (and still don't) why it skips the first one. AndreaGeddon informed me that my bootsector code differs from the code found in the NT4/W2K bootsector (he had access to the sources for those windows versions). I thought this may be a bug in XP's bootsector code.
I haven't seen any $MFT BFRs with an $attribute_list, so I cannot verify whether this behaviour is causing a failed boot. I know the code does exactly what I described. One could modify the bootsector code so it doesn't skip the $attribute_list's first $data, and then test it out on a partition with an $attribute_list present in the $MFT BFR, and see if it boots ok.
In post #3, you can see the code that will skip the $attribute_list's first $data attribute. My XP bootsector skips the first one, while the bootsector source code for NT4/W2K indicates the bootsector for those OSs does not skip the first $data.
1272
Hello Raj and rwid!
I once deliberately killed an NT4 install by fragmenting the MFT. I did this by repeatedly filling up the disk with garbage files, then randomly truncating some of them to 0-byte files. Thus, whenever the MFT had to be extended, it couldn't be extended into a contiguous area, so it had to fragment. Eventually the fragmentation got so bad that it had to spill part of the $DATA attribute out to subrecords.
The interesting thing is that not all of the $DATA attribute was moved, which is not the normal behavior seen when spilling a too-large attribute. Instead, the attribute was split. The $DATA for the first 24 record stayed in the main MFT filerecord, and the $DATA for records 24 through n spilled into one of the emergency reserve records 16 through 23. (Maybe the entire first run stayed in the main MFT filerecord, I can't remember.) It was the first (and last) time I'd ever seen a split attribute record. The filesystem was OK (chkdsk griped about a few things but it was entirely usable) but not bootable because the NT4 bootcode doesn't walk the MFT's $ATTRIBUTE_LIST so it couldn't load the data for the entire MFT so it couldn't find NTLDR.
So that answers Raj's question. The older NTFS bootcode just doesn't have the support needed to handle an extremely fragmented MFT. Just an oversight of the bootcode programmers. And you can reproduce this at will with 20 lines of C code.
Raj, what you could do is get a copy of the XP bootcode and install it on the affected systems, being careful to not overwrite the first 84 bytes (which are data about that particular filesystem).
Hmm. Actually, just try booting an XP CD into the recovery console and using the FIXBOOT command. That might well do the trick. Backup beforehand, just in case. You never regret backing up; you will eventually regret not backing up.
I'm kind of assuming you're seeing these problems on NT4 or 2K, which might be wrong. If you're using XP, and the filesystem in question was not upgraded from NT4 or 2K, then I don't know what's going on. Still, in that case, you could try getting a copy of 2003 Server, and using its recovery console to update the bootcode. I do know that 2003 Server has newer bootcode, it may solve your problem.
Rwid, if you think about it, you'll realize that the $DATA for the first few records (special files and emergency reserve) must be contained in the MFT filerecord. If they were spilled, there would be a chicken-and-egg problem. To find the proper subrecord, you would need the run data contained inside that subrecord. You'd be stuck.
(Remember, the only reason you can start to parse an NTFS filesystem is because the bootsector holds the starting cluster of the MFT and the other 15 special files. If you lose that pointer, you're stuck.)
I don't like the algorithm used in this code, but it is technically correct, given the specific circumstances involved. I would prefer that, if an $ATTRIBUTE_LIST existed, that all data be sucked out of it. That would be more orthogonal than this hacky 'skip the first' stuff. But maybe there's another chicken-and-egg problem if you don't use the resident chunk of MFT $DATA. Lessee. If the $ATTRIBUTE_LIST itself gets so big that it needs to spill, what then? Can $ATTRIBUTE_LISTs be nonresident? I don't see why not. (I'm too lazy to parse the $AttrDef file at the moment.)
Ah. If the $ATTRIBUTE_LIST spilled to a sub-record, the routine you call ReadAttribData() would not be able to find it, because the MFT cluster runlist table hasn't yet been created, which is what we're trying to create in the first place! Using this hacky method, the first part of the MFT $DATA stream (which we assume will be resident in the MFT filerecord) will have been parsed, so that the $ATTRIBUTE_LIST, if spilled to a subrecord (further assumed to be accessible via the first part of the $DATA stream; i.e. one of the emergency reserve filerecords), can be found just in time to fill in the rest of the runlist table. (Read it again, it really does make sense.)
If $ATTRIBUTE_LIST was just pushed out to disk as a nonresident attribute, this wouldn't matter. Therefore, this implies that $ATTRIBUTE_LIST must be resident. Bad NTFS design decision there, IMNSHO.
Incidentally, that was a good looking disassembly. What tools did you use?
0x517A5D out.
Originally posted by 0x517A5D@Feb 4 2006, 01:22 PM
Rwid, if you think about it, you'll realize that the $DATA for the first few records (special files and emergency reserve) must be contained in the MFT filerecord.* If they were spilled, there would be a chicken-and-egg problem.* To find the proper subrecord, you would need the run data contained inside that subrecord.* You'd be stuck.
(Remember, the only reason you can start to parse an NTFS filesystem is because the bootsector holds the starting cluster of the MFT and the other 15 special files.* If you lose that pointer, you're stuck.)
I don't like the algorithm used in this code, but it is technically correct, given the specific circumstances involved.* I would prefer that, if an $ATTRIBUTE_LIST existed, that all data be sucked out of it.* That would be more orthogonal than this hacky 'skip the first' stuff.* But maybe there's another chicken-and-egg problem if you don't use the resident chunk of MFT $DATA.* Lessee.* If the $ATTRIBUTE_LIST itself gets so big that it needs to spill, what then?* Can $ATTRIBUTE_LISTs be nonresident?* I don't see why not.* (I'm too lazy to parse the $AttrDef file at the moment.)
Ah.* If the $ATTRIBUTE_LIST spilled to a sub-record, the routine you call ReadAttribData() would not be able to find it, because the MFT cluster runlist table hasn't yet been created, which is what we're trying to create in the first place!* Using this hacky method, the first part of the MFT $DATA stream (which we assume will be resident in the MFT filerecord) will have been parsed, so that the $ATTRIBUTE_LIST, if spilled to a subrecord (further assumed to be accessible via the first part of the $DATA stream; i.e. one of the emergency reserve filerecords), can be found just in time to fill in the rest of the runlist table.* (Read it again, it really does make sense.)
If $ATTRIBUTE_LIST was just pushed out to disk as a nonresident attribute, this wouldn't matter.* Therefore, this implies that $ATTRIBUTE_LIST must be resident.* Bad NTFS design decision there, IMNSHO.
Hi 0x517A5D!
I have understood all along that an unnamed $DATA attribute HAD to be present in $MFT's base file record, and indeed the bootsector makes this assumption when locating additional $MFT record extents. But it seemed redundant to me that the first $DATA in the $ATTRIBUTE_LIST would reference the $MFT's base file record, because the bootsector knew there was an unnamed $DATA in the $MFT bfr anyway, and this first $DATA in the $ATTRIBUTE_LIST seemed unnecessary.
What just occurred to me after reading your reply, is that for all other file records, one could not assume that the first $DATA stream in the $ATTRIBUTE_LIST referenced the bfr of a file record, and the $ATTRIBUTE_LIST must reference all $DATA streams, even if one of them is in the base file record. The bootsector makes the assumption because it knows that being the $MFT, there MUST be an unnamed $DATA in its bfr -- the bootsector would not be able to locate $MFT record extents without it present. And when parsing $DATAs in the $ATTRIBUTE_LIST, it skips the first one, knowing it references the $MFT bfr, which the bootsector has on hand already.
So you have basically confirmed to me that the first $DATA in the $ATTRIBUTE_LIST points to the $MFT base file record, which the bootsector already has located and read. That's all I needed to know! Thanks! :)
You see I thought that an unnamed $DATA was always assumed to be present in a bfr, and that ALL $DATAs in the $ATTRIBUTE_LIST referenced RECORD *EXTENTS*. This is why I thought it would be catastrophic to skip the first one since it would be missing one of $MFT's record extents! But now I now this is not the case...
Incidentally, that was a good looking disassembly.* What tools did you use?
1276
tools: IDA 4.8 + two solid weeks of static analysis + ntfs/linux documentation.
Now I have every last drop of code in the xp ntfs bootsector understood and documented. It was a great learning experience for me, as I knew nothing about ntfs beforehand, and had never reversed ~4kb of 16-bit real-mode machine code. It doesn't sound like much, but when all I have is static analysis, no debugger, and (initially) little understanding of bios calls and ntfs structures, it takes quite some effort. The bootsector's code is very tight -- and a lot can go on in 4kb of code written in 16-bit real-mode assembly, so I learnt :)
One thing, I'm not clear on why you say the $ATTRIBUTE_LIST cannot be non-resident... can you explain again why this is implied? According to the bootsector code, it will handle a non-resident $ATTRIBUTE_LIST and I don't see any limitations in doing so...
rwid.
0x517A5D
02-04-2006, 04:25 PM
Originally posted by rwid@Feb 3 2006, 10:42 PM
1277
I have understood all along that an unnamed $DATA attribute HAD to be present in $MFT's base file record, and indeed the bootsector makes this assumption when locating additional $MFT record extents. But it seemed redundant to me that the first $DATA in the $ATTRIBUTE_LIST would reference the $MFT's base file record, because the bootsector knew there was an unnamed $DATA in the $MFT bfr anyway, and this first $DATA in the $ATTRIBUTE_LIST seemed unnecessary.
Yes, but an $ATTRIBUTE_LIST record lists all attributes except itself, even if those attributes happen to be located in the same filerecord. That's why, if one exists, you just walk it instead of walking the normal list. Except in special situations like this.
tools:* * IDA 4.8 + two solid weeks of static analysis + ntfs/linux documentation.
I should have known it was IDA from the XREF comments.
Now I have every last drop of code in the xp ntfs bootsector understood and documented. It was a great learning experience for me, as I knew nothing about ntfs beforehand, and had never reversed ~4kb of 16-bit real-mode machine code. It doesn't sound like much, but when all I have is static analysis, no debugger, and (initially) little understanding of bios calls and ntfs structures, it takes quite some effort. The bootsector's code is very tight -- and a lot can go on in 4kb of code written in 16-bit real-mode assembly, so I learnt :)
Actually I think the NTFS bootcode could be tightened up quite a bit more. There are a lot of uses of 32-bit pointers where the address _must_ be way under 10000h (because all the buffers fit in that one segment at 0D00h:0). So you could save a byte each time those pointers are loaded into a register or dereferenced.
One thing, I'm not clear on why you say the $ATTRIBUTE_LIST cannot be non-resident... can you explain again why this is implied? According to the bootsector code, it will handle a non-resident $ATTRIBUTE_LIST and I don't see any limitations in doing so...
rwid.
1277
I was thinking that because the bootcode jumps through these hoops to handle a (resident) $ATTRIBUTE_LIST that's been spilled to a subrecord. I just figured that, if $ATTRIBUTE_LIST could be nonresident, that would never happen because the NTFS driver dumps all nonresident-capable attributes out of the filerecord before it resorts to spilling or splitting resident-only attributes.
The linux-ntfs documentation says it can be nonresident, and that's one of the areas where the documentation is guaranteed to be correct, because that info can be parsed out of the $AttrDef file. So I was wrong.
0x517A5D out.
Originally posted by rwid@Feb 4 2006, 01:42 PM
Hi 0x517A5D!
I have understood all along that an unnamed $DATA attribute HAD to be present in $MFT's base file record, and indeed the bootsector makes this assumption when locating additional $MFT record extents. But it seemed redundant to me that the first $DATA in the $ATTRIBUTE_LIST would reference the $MFT's base file record, because the bootsector knew there was an unnamed $DATA in the $MFT bfr anyway, and this first $DATA in the $ATTRIBUTE_LIST seemed unnecessary.
What just occurred to me after reading your reply, is that for all other file records, one could not assume that the first $DATA stream in the $ATTRIBUTE_LIST referenced the bfr of a file record, and the $ATTRIBUTE_LIST must reference all $DATA streams, even if one of them is in the base file record. The bootsector makes the assumption because it knows that being the $MFT, there MUST be an unnamed $DATA in its bfr -- the bootsector would not be able to locate $MFT record extents without it present. And when parsing $DATAs in the $ATTRIBUTE_LIST, it skips the first one, knowing it references the $MFT bfr, which the bootsector has on hand already.
So you have basically confirmed to me that the first $DATA in the $ATTRIBUTE_LIST points to the $MFT base file record, which the bootsector already has located and read. That's all I needed to know! Thanks! :)
You see I thought that an unnamed $DATA was always assumed to be present in a bfr, and that ALL $DATAs in the $ATTRIBUTE_LIST referenced RECORD *EXTENTS*. This is why I thought it would be catastrophic to skip the first one since it would be missing one of $MFT's record extents! But now I now this is not the case...
tools:* * IDA 4.8 + two solid weeks of static analysis + ntfs/linux documentation.
Now I have every last drop of code in the xp ntfs bootsector understood and documented. It was a great learning experience for me, as I knew nothing about ntfs beforehand, and had never reversed ~4kb of 16-bit real-mode machine code. It doesn't sound like much, but when all I have is static analysis, no debugger, and (initially) little understanding of bios calls and ntfs structures, it takes quite some effort. The bootsector's code is very tight -- and a lot can go on in 4kb of code written in 16-bit real-mode assembly, so I learnt :)
One thing, I'm not clear on why you say the $ATTRIBUTE_LIST cannot be non-resident... can you explain again why this is implied? According to the bootsector code, it will handle a non-resident $ATTRIBUTE_LIST and I don't see any limitations in doing so...
rwid.
1277
Hey rwid,
Release that your dissasembly and documentation probably took you a whole bunch of time and effort, but it would really help me with some problems I'm having booting XP Embedded from a Flash Card on this SBC I picked up. Any chance you'd be willing to share it? :D
JHZ
Originally posted by JHZ@Feb 8 2006, 08:54 PM
Hey rwid,
Release that your dissasembly and documentation probably took you a whole bunch of time and effort, but it would really help me with some problems I'm having booting XP Embedded from a Flash Card on this SBC I picked up. Any chance you'd be willing to share it?* :D
JHZ
1279
Hi JHZ,
I'm not sure that the info I have will help you, I have only completed the mbr code and ntfs bootsector code study. If you still want it I guess I could clean up my ida file and give you a commented disassembly listing?
rwid.
vBulletin® v3.6.4, Copyright ©2000-2016, Jelsoft Enterprises Ltd.