ned Productions - Moving an existing Windows installation to a Removable USB drive

Here follows my notes on how to move an existing installation of Microsoft Windows XP SP2 to an external USB hard drive so that it boots from that drive. I hope others find it useful.

Why?!?!? What for?!?!?

In October 2006, my laptop mysteriously started locking up after a few minutes of being turned on. After a great deal of messing around with it, I discovered that the electrical connection for the primary IDE interface on the Southbridge had become fried by a loose screw the system assembler had accidentally left inside the case. As the laptop was just outside its warranty, this left me with a £1000 lemon - everything worked, but if anything were plugged into the primary IDE interface, the machine crashed.

The first thing I bought was a 2.5" PATA IDE to 3.5" PATA IDE converter for about five euro. This allowed me to save off all my data to the main desktop. Thereafter, with a great deal more testing I discovered that the USB interface was undamaged which raised the interesting question of could I stick the 2.5" IDE hard drive into a USB enclosure and continue using the laptop?

A USB2 IDE enclosure is about ten euro and apart from the high power requirements of my high-end 2.5" IDE hard drive (1 amp), it works just fine in any semi-modern computer. A USB1 port supplies such a miniscule current that it won't power anything - but a USB2 port should supply 500mA of 5V. A slightly more expensive USB2 IDE enclosure comes with two USB2 cables sufficient to draw enough power for most 2.5" IDE hard drives, otherwise you'll need an external PSU outputting 5V.

All this said, now I have a working bootable USB drive it actually has become very useful - I can literally plug my laptop's hard drive into any computer I like which really eases data transfer. Seeing as Windows can do this, why Microsoft won't make the small changes required is beyond me - I'm not talking about multi-computer boot ability (this is an entirely different question), but rather that your hard drive can boot on one computer but is accessible by any computer.

WARNING!

Microsoft have said on many occasions that it is impossible to boot Windows from a USB device. This means that what we are about to do is not only nasty, but is very much unsupported. If you apply service patches, add a new peripheral or device to the system or indeed do anything at all you may break everything. You also will be doing low-level registry hacks and config file modifications - if you make a single mistake or slip up ie; type a single thing wrongly, you can totally wreck your system.

DO NOT DO THIS UNLESS YOU ARE VERY, VERY SURE YOU CAN LOSE EVERYTHING ON YOUR HARD DRIVE!!!

Prerequisites

Firstly, you need a BIOS which can boot from a USB device. At the time of writing (February 2007), Award BIOS's tend to NOT support booting from USB HDD's. AMI BIOS's do. Whatever Dell uses for their laptops seems to work.

Secondly, your BIOS really needs to be able to boot from a USB2 rather than a USB1 device. Many BIOS's will boot from a USB device, but even if the USB interface is 2.0 it will only talk to it at 1.1 speeds. This makes the device very, very slow to boot from.

I found that my BIOS wouldn't even see the USB device unless I had "Enable legacy USB" turned on. This may solve the puzzling situation where you have "USB HDD" in the boot list but the BIOS won't see it.

DO NOT EMAIL ME ABOUT WHETHER YOUR COMPUTER OR BIOS CAN OR CANNOT BOOT FROM USB. I DO NOT KNOW SO DON'T BOTHER ASKING.

When the BIOS doesn't support booting from USB devices, it is possible to boot a USB device using an intermediate bootable CD-ROM. Again, don't ask me because I don't know - go search google and find out for yourself.

The other prerequisite is either BartPE or the Ultimate Boot CD for Windows. You'll need this for two reasons - to do offline registry editing and hacking of various system files, and also for a working copy of NTDETECT.COM. To build either BartPE or UBCD, you will need a copy of the Windows XP installation disc - not Windows 2000 or any earlier version. This is because we need specifically the WinXP registry editor though one can make do with regedt32 instead of regedit on older systems.

Howto

What I am about to describe requires understanding what you are doing. No, I really do mean that - each system is unique and different, and you will have to expect that it will require some playing around with the system to get it working perfectly. What I describe is what worked for my system - I probably have done too much, but it will also be too little for some systems.

You will almost certainly need to peruse the very, very long thread at 911cd.net. My thanks to all the posters there for all their tips - mine is a synthesis of various posts and ideas. You may find the following list of links into that thread useful:

Before you begin, remove your 2.5" IDE drive or whatever from your desktop and put it into your USB enclosure. Attach it and make it the bootable device in your BIOS second only to your DVD drive. You should find that it boots at least as far as the Windows XP logo screen - if it doesn't, go search the forums at 911cd.net above.

The Theory

Windows is configured to expect USB devices to be Base devices which means they are important but not necessary for booting. This means that they get loaded after the initial boot sequence of the computer. To be specific, a Windows computer boots the following categories of device driver in this order:

System Reserved
Boot Bus Extender
System Bus Extender
SCSI miniport
port
Primary disk
SCSI class
SCSI CDROM class
FSFilter Infrastructure
FSFilter System
FSFilter Bottom
FSFilter Copy Protection
FSFilter Security Enhancer
FSFilter Open File
FSFilter Physical Quota Management
FSFilter Encryption
FSFilter Compression
FSFilter HSM
FSFilter Cluster File System
FSFilter System Recovery
FSFilter Quota Management
FSFilter Content Screener
FSFilter Continuous Backup
FSFilter Replication
FSFilter Anti-Virus
FSFilter Undelete
FSFilter Activity Monitor
FSFilter Top
filter
boot file system
Base
Pointer Port
Keyboard Port
Pointer Class
Keyboard Class
Video Init
Video
Video Save
file system
Event log
Streams Drivers
NDIS Wrapper
PNP_TDI
NDIS
TDI
NetBIOSGroup
PlugPlay
SpoolerGroup
NetDDEGroup
Parallel arbitrator
extended base
RemoteValidation
PCI Configuration

By default, USB device drivers are loaded at stage "Base". Unfortunately, if Windows can't find its boot device by "boot file system" it issues a BSOD (Blue Screen of Death) 0x7b which means "Boot device not found" just after boot.

Now what we need to do is to firstly create a new boot device group called "USB boot support" right at the start and place the USB device drivers into it. To do this, boot from your BartPE or UBCD disc or even just plug your USB hard drive into another working Windows installation. Once inside, open <whatever>:\WINDOWS\system32\config on your USB hard drive. IF YOU CAN'T FIND YOUR USB HARD DRIVE, YOUR BIOS DOESN'T SUPPORT USB DEVICES!

Make an entire copy of the config directory somewhere else in system32. Call it whatever you like - this is your backup in case anything goes wrong (and also to run comparisons with in case you modify the wrong thing).

Run regedit (regedt32 pre-Windows XP). Click on HKEY_LOCAL_MACHINE. Choose "Load Hive" from the File or Registry menu. Navigate to your USB Windows installation at <whatever>:\WINDOWS\system32\config. Choose the SYSTEM file. Name it 'foobar' in the dialogue box which pops up. The remote hive should now appear under 'foobar', though note I had problems loading Windows XP hives on Windows 2000 machines.

Go to HKLM\foobar\ControlSet001\Services. For each of:

usbehci, set Tag (DWORD)=0x10
usbhub, set Tag (DWORD)=0x11
usbstor, set Tag (DWORD)=0x21
usbuchi, set Tag (DWORD)=0x0f
If there is a usbohci, set Tag (DWORD)=0x20

For each of the above, set Group (REG_SZ) to "USB boot support" instead of "Base". If Group or Tag isn't present, create the necessary key beforehand. Also for each of the above, set StartType to 0 (load immediately) instead of 3 (load on demand).

Go to HKLM\foobar\ControlSet001\Control\GroupOrderList. Create key "USB boot support" as REG_BINARY with the value "05 00 00 00 0f 00 00 00 80 00 00 00 10 00 00 00 11 00 00 00 81 00 00 00".

Go to HKLM\foobar\ControlSet001\Control\ServiceGroupOrder. Insert "USB boot support" after the entry "System Reserved" but before "Boot Bus Extender".

Okay, you have now fixed up the registry for this new boot order. However, if you ever try adding a new USB peripheral you will find it all gets reset. This is because the system INF files are still using the old information, so we need to fix these.

Browse to <whatever>:\WINDOWS\inf on your USB drive. This directory is normally hidden so it will be drawn faint. In this directory you will find the files usb.ini, usbport.ini and usbstor.ini. After each will be usb.pnf, usbport.pnf and usbstor.pnf - these are the precompiled variants. Delete these PNF files as we want them to be rebuilt.

Now load each of usb.ini, usbport.ini and usbstor.ini into Notepad. For each, search for "StartType". Make VERY SURE that the device driver being loaded is one of the drivers we modified in the registry above - if it isn't, don't modify it. You can tell because each section will refer to the driver by name eg; usbuchi. For each driver reference, set "StartType" to 0 instead of 3 and also set LoadOrderGroup to "USB boot support" instead of "Base". If there is no LoadOrderGroup, add one.

Ok, you are doing well - you have moved the USB devices up the boot order. The next thing is to make Windows believe that your USB hard drive always was the boot drive all along. To do this, you need to add the following keys to HKLM\foobar\ControlSet001\Control\CriticalDeviceDatabase:

[HKEY_LOCAL_MACHINE\foobar\ControlSet001\Control\CriticalDeviceDatabase\usb#class_08]
"ClassGUID"="{36FC9E60-C465-11CF-8056-444553540000}"
"Service"="USBSTOR"

[HKEY_LOCAL_MACHINE\foobar\ControlSet001\Control\CriticalDeviceDatabase\usb#root_hub]
"Service"="usbhub"
"ClassGUID"="{36FC9E60-C465-11CF-8056-444553540000}"

[HKEY_LOCAL_MACHINE\foobar\ControlSet001\Control\CriticalDeviceDatabase\usb#root_hub20]
"Service"="usbhub"
"ClassGUID"="{36FC9E60-C465-11CF-8056-444553540000}"

What these do is to tell Windows how to map critical boot devices to their respective drivers as it's too early in the boot cycle for Windows to otherwise know (arguably). The usb#class_08 should map all USB mass storage device controllers, but there are some which don't - for this, you need to add a vid_xxxx&Pid_xxxx entry (see 911cd.net forums above).

Lastly, you need to may need to map your actual hard drive. You can try skipping this section, if you can't boot then return to this point but before you do, insert your USB hard drive into a working Windows XP computer and open the Device Manager, pull up the properties page for the USB hard drive and choose the Details tab. You should see the Device Instance Id as follows: USBSTOR\DISK&VEN_aaaaaaaa&PROD_bbbbbbbbbbbbbbbb&REV_cccc\xxxxxxxxxxxx. Insert that like this:

[HKEY_LOCAL_MACHINE\foobar\ControlSet001\Control\CriticalDeviceDatabase\USBSTOR#Diskaaaaaaaabbbbbbbbbbbbbbbbcccc]
"ClassGUID"="{4D36E967-E325-11CE-BFC1-08002BE10318}"
"Service"="disk"

If the bbbbbbbbbbbbbbbb isn't sixteen characters long, pad it out with underscores ('_'). You may find the first Hardware Instance Id matches what you need perfectly.

Most of the guides on 911cd.net advise you to delete the MountedDevices key or to replace the CriticalDeviceDatabase key with a generic copy entirely. Unless you want your USB Windows installation to be fully bootable by all machines, don't do this. If you do do this, your drive letter assignments will be all reset along with other device settings, so generally speaking don't bother.

Most of the guides on 911cd.net also advise you to replace the SP2 USB driver files with their SP1 counterparts. I had no problems with the SP2 variants, so you might as well leave them as-is.

To unload your USB based registry, you select it in regedit and choose "Unload hive". If you get an error, close regedit, restart it and try again. MAKE SURE YOU UNLOAD THE HIVE - FAILURE TO DO THIS WILL PROBABLY CORRUPT YOUR REGISTRY!

In fact, if you'd like to be really paranoid about saving it, export the entire "foobar" tree from regedit, choosing hive file format in the Save As box. Save it as something like "system2", unload the hive as above and then replace system with system2.

You have one last thing to do. SP2 introduced a new NTDETECT.COM which to be frank, is simply buggy. You will probably find that it causes intermittent BSOD's 0x7b eg; you can boot under safe mode or with boot logging enabled but normal boots are 50/50 successful. The solution is easy - in the BartPE or UBCD CD-ROM in the i386 directory there is an older NTDETECT.COM. Simply copy this one over the one in the root directory of your windows installation.

Now in theory, you should just be able to reboot and it all just works. You may find however that your machine stutters during logon (if it appears to hang after entering the administrator password, try hitting Ctrl-Alt-Del to wake it up) - this is probably due to Windows now running from a removable device for which it by default will disable the write cache - also, Windows won't put a swap file onto a removable device and rather more importantly, some programs refuse to install to an instant-removable device eg; Windows Update.

How removable Windows treats your device depends on what Windows thinks it is. For example, hot-plug SATA drives are removable fixed discs but not as removable as a USB Flash stick which is more like a floppy disc. If you find things refusing to work, you can try installing the Hitachi filter driver below to fake a fixed disk status.

Before anything else though, you should enable write caching - open the property sheet for the USB hard drive in the Device Manager and choose "Optimise for performance" from the Policies tab. With this enabled, you see slightly less than IDE interface transfer rates but you still see a few extra milliseconds added onto seek times due to the USB interface overhead. Either way, performance is barely different from a native IDE interface. Not enabling this is pointless as pulling the USB cable when you booted from it is fatal to your system whichever way.

As Windows simply won't use a removable device for a page file, you should disable using a swap file completely using My Computer properties=>Advanced=>Performance. Clear the values for each drive with a value set completely - don't worry, your machine will run just fine without a swap file if you have enough RAM.

Installing the Hitachi filter driver (optional)

Download this file which is a microdriver by Hitachi. Expand the contents and open cfadisk.inf in Notepad. There are several lines under [cfadisk_device] starting with %Microdrive_devdesc%. Delete all, replacing with:

%Microdrive_devdesc% = cfadisk_install,USBSTOR\Diskaaaaaaaabbbbbbbbbbbbbbbbcccc

... where Diskaaaaaaaabbbbbbbbbbbbbbbbcccc comes from the part above. Save the file. Open Device Manager and find your USB hard drive. Right click and choose "Update driver". Choose "Install from a specific location". Now choose "Don't search. I will choose the driver to install" followed by "Have disk". Choose the directory where you expanded the contents of the Hitachi microdriver and install. After a reboot, you should find that the system is much happier with letting put swap files etc. on your usb hard drive.

I hope you find this page useful!

ned Productions – Moving an existing Windows installation to a Removable USB drive

Why?!?!? What for?!?!?

WARNING!

Prerequisites

Howto

Installing the Hitachi filter driver (optional)