11/02/2015

Windows P2V, Post TCP/IP Reconfigurations

After building a VM instance for a Physical to Virtual Conversion. The first boot can appear hung on the startup step Identifying Network. Here's how to avoid that.


Typically a P2V will be created from a raw disk image capture or from the backup files of a live system. On first boot the Windows Plug-n-Play service will inventory the detected hardware and enable it with the drivers currently on the virtual machines hard disk. Any "missing" hardware will remain configured, but its device driver will not be started.

If the virtual machine environment "emulates" a hardware device for which the "on disk" image contains a compatible device driver, the new Virtual Machine will inventory the new hardware and automatically install the compatible device driver and proceed to "Enable" it.

For network interfaces this can be particularly problematic.

The new interface will not have been assigned a static TCP/IP address or default gateway, nor a DNS source. It will first attempt DHCP and if that fails will proceed to self configure itself with an Automatically Provisioned IP Address - APIPA. The Network Location Awareness - NLA features available since Windows Vista will then engage and proceed to "Test" the network in order to match it up to a "known" Network firewall profile {Domain, Private or Public}. And apply that default set of firewall rules to regulate allowed or blocked inbound and outbound TCP/IP traffic.

Then yet another new feature called Network Status Connectivity Indicator - NSCI will attempt to use the default gateway to contact a Microsoft Beacon site to prove or disprove the Network can be used to connect to the Internet.

DHCP
APIPA
NLA - with multiple profiling "tests"
NSCI

All this takes time and introduces lengthy delays when starting a new network interface on an unknown network, and even longer if the virtual network interface has been deliberately isolated from any other network.

It should also be mentioned with Windows Vista the TCP/IP stack was further "tuned" to discover the maxmium MTU transmission unit for a given connection and ramped up depending on the default selected agorithm and would also reset or ramp down if a connection failed to establish. This was called "autotuning" and can be changed from a dynamic to static behavior from from the netsh  command prompt.

Additionally TCP/IP stacks can be offloaded onto dedicated hardware for certain chipsets, and jumbo packet support can influence both device driver, virtual machine and host network transfer rates.

Virtual technologies supporting shared physical hardware with virtual machines like [sr-iov] and hewlett packard "virtual connection" or systray tool for managing "binding" and "compositing" bonded network interfaces can help or conflict within new virtual machines. Interrupt Moderation or Throttling virtual machine interrupts for handling network interfaces is also another potential problem issue.

There are also alternative device drivers which can be introduced [after] first boot, which paravirtualize or "enlighten the device driver" that it is actually running in a virtual machine and can better cooperate with the Host to optimze network interface behavior. This is in contrast to "Full" virtualization in which all physical hardware is virtualize, or "Hardware assisted" virtualization in which the physical hardware participates in supporting virtualization independent of the guest operating system device driver being aware that it is being virtualized.

Many of the service features only really make sense on a mobile platform like a laptop, or on a client system on a fully configured host network. However they still exist on the Windows Server platforms and in general are difficult to resolve.

For one thing even if all of the timeouts are allowed to expire. Attempting to reconfigure the TCP/IP address of the new network interface with a previously used TCP/IP address, even from a network it is no longer connected will produce a warning that the TCP/IP address is currently "assigned" to a missing piece of hardware. Removing it from that piece of hardware is less than reliable even when following the instructions provided in the dialog box.. and then a complete reboot and expiration of all the timeouts will be required before any mistakes or missteps can be discovered. This can take upwards of 30 minutes or more!

The symptom of this long tale of first boot is that "Identifying Network" in the system tray appears to hang, and any attempt to open the [Network  and Sharing Center] will produce a blank or non-responsive window, until all of the network interface self configuration steps have completed.

The way to resolve this problem is to:

A. Disable or "Disconnect the Cable" to the new network interface that will be created by the Host environment for the virtual machine before the new virtual machine is started. Then the network interface will not attempt DHCP, APIPA, NLA or NSCI and immediately open in the desktop environment for the logged in user (and) the [Network and Sharing Center] will be immediately available and responsive.

B. Boot first into a simplified environment in which services that may depend upon network connectivity are automatically disabled, or severely restricted. So that the Plug-n-Play service can "discover" the new network interface hardware and install and activate a device driver for it. Since it will be "unplugged" from the virtual machines point of view, it will not proceed to begin DHCP, APIPA, NLA or NCSI. After initial discovery and driver installation the Windows operating system will typically be required to restart to finish implementing the changes. If possible this is also a good time to disable any services that depend upon network connectivity until the new interface can be statically configured, since each of those services will then proceed to attempt to use network services and compound the start up problem by adding their timeouts to a Normal Startup.

C. On the next boot, into a reduced functionality environment. Use the ncsa.pl control applet, control netconnections or [Network and Sharing Center] wizard panel to access the new network interface and proceed to configure a static IPv4 address, gateway and DNS source. It is also recommended to configure a static IPv6 address, gateway and DNS source since many services prioritize IPv6 over IPv4 and must timeout in that layer before traveling back to IPv4 to begin opening up tcp and winsock services.

While much of this can be done by attaching and booting the Windows RE recovery environment from the original installation environment.

It can be made "far" easier by using a custom "Microsoft Desktop Optimization Pack - MDOP" feature called "DaRT - Diagnostics and Recovery Toolset" Software Assurance and Volume License customers have access to this.
 
MDOP
|
SRS Options ->DaRT




The MDOP comes as an installable CD/DVD image iso with an autorun installer which can be used to install the [DaRT Recovery Image "Wizard"]. Running this Wizard helps create a user customized DaRT.iso > bootable CD/DVD iso or USB image which can then be used to start the virtual machine.

Booting the DaRT.iso image the system asks if the drive letters of the existing disk image should be mapped in a familar C:\ pattern, then lands at a "System Recovery Options" page, the option at the bottom of the list of system recovery tools > starts the DaRT toolset window.

The two most powerful tools are [Computer Management] and [Registry Editor].


[Computer Management] refers to the "offline' virtual machine image sitting on the virtual hard disk to which this bootable iso has been attached. Any actions in the Computer Management tool affect the contents on the actual offline virtual machine disk image.

Under this tool is access to the currently enabled device drivers and their startup type at boot time, which can be disabled, so as not to start.


This can be useful for drivers which are installed with applications to start at boot time and could cause additional problems. Disabling them here makes sure they will not start, and generally makes uninstalling them and their application package easier since no startup timeouts have to be endured and no shutdown procedure must be run to disable the driver after startup from within the operating system.

Also under this tool is access to the currently enabled services and their startup type at boot time, which can be disabled, so as not to start.


For similar reasons and for a speedier boot while finalizing the initial network configuration you may choose to disable various services.

[Registry Editor] refers to the "offline" virtual machine image registry on the virtual hard disk to which this bootable iso has been attached. Any actions in the RE tool affects the contents on the actual virtual machine.


Generally disabling the NSCI service from the registry is good.

Less agressive Domain network probing in the NLAsvc service can also be configured, but neutralizing the NSCI is usually sufficient.

Another feature of the DaRT "Wizard" is the ability to copy a folder of scripts and tools "into" the boot image that can be accessed from within this recovery environment, which can better automate and assist in finalizing the configuration of the virtual machine. One possible use is copy additional files onto from the recovery environment virtual disk image to the virtual machine hard disk and even set a script for the virtual machine to run on "first boot".

P2V – HP Proliant Support Pack Cleaner

Hewlett Packard ProLiant systems often have monitoring and alerting software and services which must be removed, a popular batch file is widely available that takes care of disabling and removing the services. It is only made faster and more effective by pre-booting into DaRT and disabling the associated services so they do not have a chance to hang the virtual machine. When the windows unistaller tool is used they are quickly remove.

P2V - GhostBuster Device Remover - commandline, task scheduler, UAC options

The "GhostBuster" driver script is also a widely used script for finding and automatically removing enabled hardware drivers for which no hardware is currently detected.


And the TCP/IP and Winsock stacks can be reset, or specific Automatic Tuning features disabled or further customized for the environment. Internet Options refering to unreachable SSL revocation lists and services, or windows update servers can be shut off or adjusted to use proxy services or a local cache.