Imaging VMware ESX Guest using ImageX
VMware ESX has the ability to clone templates, and this is a great feature when using iSCSI or FC datastores. When leveraging NFS datastores however, you lose thin provisioning on those NFS datastores. One way to get around this is to use a third party imaging software.
Microsoft provides a free imaging package titled imageX. You can read about on technet from here. A quick synopsis is that, unlike other imaging software, ImageX focuses on the files instead of disk blocks. This allows ImageX to leverage a Single Instance Store (SIS). Image X will compress the first image at around 33 to 50% of the on disk size, and will store the image in a file with a WIM extension. The WIM file holds the SIS and also indexes of which blocks of data are associated with the image. The benefit of the SIS and imageX can be found when appending a second image to the WIM. imageX will create a second index in the WIM file. It will then start imaging the machine, it will find a file and compare that to the SIS. If the file is found, it will add a pointer to the new index and move on. If the file is not found, imageX will add the file into the SIS and then add a pointer in the new index and move on.
To use imagex, you will need to download the Windows Automated Installation Kit (WAIK). After installing the WAIK, you can follow the instructions found here, on svrops.com, to create a WinPE boot CD. Before you create the CD using he OSCDIMG, you will need to inject the Network and SCSI drivers required for ESX. To do this, you will first need to get the correct drivers. You can either scour the internet looking for them, or you can just select install VMware tools from a guest VM. On the Guest VM, open my computers, Open the CDrom, \program files\VMware\VMware Tools\Drivers\ and copy the Vmxnet and SCSI folders. Now, on the machine that you installed the WAIK on, run the following commands
peimg /inf=DRIVE:<location you copied the network driver>vmxnet.inf /image=DRIVE:<mount location of wimfile>
peimg /inf=DRIVE:<location you copied the network driver>vmscsi.inf /image=DRIVE:<mount location of wimfile>
you can also use the same commands to inject other drivers, if you are going to leverage the boot CD across other hardware also. After you’ve injected all of the drivers that you plan on, go ahead and complete the instructions found on svrops.com. Now, after the OSCDIMG command, you will have a bootable WINPE iso. One point, make sure that you when you run the unmount command, ensure that you use the /commit command. If not, all of your changes will be lost, and you get to do it again.
Image Capture
So now that you have a bootable winpe iso, with imagex and the correct drivers, what do you do. First, build a gold image(s) of Windows XP, vista, 2003 and/or 2008. Patch everything with the latest service packs, security patches, etc… Also, its best to build the initial gold image with multi processors. That way you can use the same image for single/multi processors with out needing to change the HAL. next sysprep the gold image. You can find instructions for running sysprep here. After running the sysprep, start the VM and mount the CDrom as the iso created.
After booting, you will need to mount a network share, use a command similar to: Net use m: \\<servername>\share.
Create a new VM manually. It doesn’t need to be identical, but ensure that the hard disks are large enough to hold the uncompressed data from the gold image. Next, boot to the ImageX WinPE iso created earlier. After booting up, you will need to run diskpart, you can find websites online that detail everything about diskpart, but to create a basic C drive, you will need to run the following commands.
Diskpart.exe
Select disk 0
Create partition primary size=<size of disk in Mbytes>
Select partition 1
Format fs=NTFS label=”Sys” Quick
Active
Now, for each additional disk, select disk <disk> and run all of the same commands as above except, change the label to a description of the drive. Also, run the active command on the sys drive. After you have configured all of the drives type exit to get back to the command prompt. From the command line, run the net use command again. Next, change to X:\program files\imagex\. Type imagex /apply m:\<wimfile>.wim <index number> c: /verify. After the image is applied, you can rerun the imagex /apply command change the index number and the drive. After complete, unmount CDrom and reboot the VM. You should now be greeted by the Windows mini-setup.
While not as fast as VMware builtin clone from template, it does allow you to continue to leverage the thin provisioning inherent in NFS datastores. In our environments, it takes roughly 20 minutes to build a Windows 2003 VM, versus about 10 minutes to build the same VM from template.
Leveraging Solaris 10 ZFS Functionality for ESXi
With the decision of VMware to release the ESXi version of the industry leading hypervisor for free, one might wonder where else can I save money and still get enterprise level performance. One area is in the storage. The Solaris ZFS file system is a great starting point for small businesses looking to move into enterprise level virtualization while still hitting a relatively small price point. Sun recently released three new products in the 7000 series of products: the 7110, 7210 and the 7410. While the 7210 and 7410 would make a techie drool. For the small business, the 7110 offers most of the benefits of the two larger versions, at a very affordable price. The 7110 starts at around $10,000 list and would rise to $16,000 with the three year platinum warranty and SCSI controller. Now, you could build a SUN server with similar performance characteristics, but you would lose one of the best features of the new Sun Storage; Storage Analytics. This feature is an enhancement of Dtrace that allows the administrator to drill down into the performance of the storage, to isolate performance issues.
So what do you have to do to get VMware ESXi and Solaris ZFS doing their thing together? Build your Solaris 10 Server. There are many good resources for available on the internet for configuring Solaris 10 and ZFS. Here are the instructions from SUN. Some basic recommendations: keep the system drives and the data drives separate. Use a mirrored ZFS pool for the system drive. Use RaidZ2 for the data drives, with a minimum of 5 drives: 3+2 and no more than 11 drives: 9+2. When you go above the 11 drive mark, you’ll just add an additional raid set to the ZFS pool. Leave a drive or two available for a spare(s) in the pool.
After you have your ZFS pool created.
You first need to first create the file system. All you really need to decide is: do you want a quota (Upper limit), reservation (minimum limit) and a name. you would enter the following command
zfs create <ZFSPool>/<fileSystemName>
You can get creative with the organization, if you know that you are going to have multiple NFS filesystems, you might enter zfs create pool/VM_NFS/FS01 or pool/VM_NFS/root. Ok, done, you’ve created a ZFS file system. Now you might want to set a quota and/or reservation.
zfs set quota=250G <ZFSPool>/<fileSystemName>
There you now have set a limit of that pool of 250 GB. Also, at this point, that is a thin provisioned size. Very little space will be utilized until data is written. The next step is to set a reservation, if so desired.
zfs set reservation=100G <ZFSPool>/<fileSystemName>
You now have a file system that has taken up 100 GB of the pool. While no data has actually be written to disk, the pool will show that the available size has been reduced by 100 GB. Now, the last step before we can start configuring the ESXi datastore is to mount the filesystem as a NFS share.
zfs set sharenfs=root=<IP address of the ESXi server> <ZFSPool>/<fileSystemName>
The key in the above command is the sharenfs=root=. That command will give the ESXi server root access to the NFS share. Without that, you will be able to mount the share, but you will not be able to create or open VMs hosted on the share. Ok that’s it, on to the ESXi server.
Build the ESXi Server
First download the ESXi server software. You can always download the free version, and then apply the purchased license purchased at a later date. Install ESXi per the instructions found here. Now, open your browser and go to the IP address that you assigned to the ESXi server from a Windows workstation. In the left hand corner of the page, you will see a link that states “Download VMware Infrastructure Client.” Go ahead and install the client. When it is complete, click on the shortcut and use the username Root and the password you created during the ESXi install. The next step is to install either the free license or a purchased license for the ESXi install. Left click the server name and select the configuration tab. Next select the “Licenses Feature” option. Select the top “Edit” to the right of “License Source.” Select the “Use License” option and either type or paste the license provided by VMware.
Next, you need to enable NTP time. You can either point it towards and internal time server if available, or towards public name servers. For my home network or other small businesses, I point towards the following: pool.ntp.org, 0.pool.ntp.org, 1.pool.ntp.org and 2.pool.ntp.org. You enter the NTP servers by selecting the Configuration tab, “Time Configuration” option and properties in the upper left hand corner. Ensure the “NTP Client Enabled” Checkbox is checked. Next, select the Options button, and select the “NTP Servers” option. Delete the default server. Click Add, type the first of the NTP server and repeat for each the remaining 3 NTP servers. Check the “Restart NTP service to apply changes.” Select OK twice and wait until the change is applied.
Next we need to configure the networking portion. If you are a small enough, you can get by with two network ports and push both client traffic and NFS traffic across them. You might notice latency if your traffic begins to exceed, if you experience this, you will need to add/configure additional NICs. Ensure that any traffic that involves clients and/or NFS traffic is redundant. Also, Vmotion and the NFS datastore traffic should be placed on GB interfaces on GB switch ports.
We are now ready to mount the NFS datastore. First select the configuration tab and select the Storage option. In the right hand corner, select “Add Storage.” Select the “Network File System” option and select Next. Next enter the IP address or Fully Qualified Domain Name (FQDN) of the Solaris NFS server. For the folder enter /<ZFSPool>/<fileSystemName>. Ensure the “Mount read only” is unchecked and then enter a easily recognizable name for the datastore. Select OK and you should have a datastore created that shows a size equal to the Quota configured earlier. Go ahead and create a VM, you done.
Other features of ZFS that is useful to ESXi
One cool feature of Solaris ZFS is the ability to make writable clones of file systems. Imagine building 1,2 5 VMs. Sysprep them, clone the file system and re-present the new cloned filesystem to the ESXi server. Or you could build a test environment of a web and SQL server that could be stood up in seconds. The clone process leverages the snapshot functionality.
zfs snapshot <ZFSPool>/<fileSystemName>@<snapshotName>
zfs clone <ZFSPool>/<filesystemName>@<snapshotName> <ZFSPool>/<fileSystemName1>
zfs set sharenfs=root=<IP address of the ESXi server> <ZFSPool>/<fileSystemName1>
zfs set quota=250G <ZFSPool>/<fileSystemName1>
zfs set reservation=100G <ZFSPool>/<fileSystemName1>
Now, you have a second file System that contains an exact copy of file system as it existed at the time of the snapshot. The snapshot name can be nearly anything you want, if you are creating a snapshot that will be use for cloning test environments, you might name the snapshot : webServer_SQL_Test_Gold. Also, if these are windows VMs, run sysprep against the servers and shut them down fully before creating the snapshot and the clone. Another thing to be aware of is that you are limited to 8 NFS datastores by default in ESX, to extend past that you will need to modify the NFS.maxvolumes under the Configuration tabà advanced setting. You can run up to 32 NFS datastores per ESX host/cluster.
You can also present iSCSI drives to your ESX clients from the ZFS pool.
On the Solaris Wiki site, you will find a great write-up about how to present ISCSI ZFS luns to various initiators.
If you have questions, concerns, gripes, etc… leave a comment and let me know.
VMware ESX 4 Storage vMotion
One of the features being updated in the next release of VMware ESX is storage vMotion. Storage vMotion was released in ESX 3.5 and while great for administrators that leverage either iSCSI or VMFS, it was of limited use to administrators leveraging NFS datastores.
One benefit of using NFS datastores is the default creation of thin VMDKs. Administrators can create larger VMDKSs on VM creation, and not waste space on the storage unit. I consider this a set it and forget it option. We only need to monitor the datastore and increase the size of the NFS mount when required. While thin provisioning is the default configuration option when using NFS datastores, the thin provisioning was not kept when moving the VMDKs from one storage unit to another. This is a limitation of the underlying ESX hypervisor, as we could not even move the VMDKs through the service console without losing the thin provisioning.
Enter ESX 4. According to VMware demo at http://download3.vmware.com/vdcos/demos/Storage_VMotion_800x600.html, we now have the option to live migrate from thin to thin. It also appears that we can now move VMDKs from thick to thin, thin to thick, even RDM to either thick or thin or any of the above to RDM. One of the selling points of storage vMotion in 3.5 was the ability to move the VM live between storage. You could build your server on tier 2 storage, when ready to go to production; you could move the VM to tier 1 storage with no downtime to the VM. Now, NFS datastore administrators can fully leverage this technology.
What are some benefits of the enhanced storage vMotion?
- Build VMs on tier 2 storage and move to tier 1 storage when ready for production.
- Remove our lock-in to FC, iSCSI or NFS. Build using one technology now and migrate in the future based on performance requirements.
- Migrate to new storage when replacing aged storage heads.
While on the topic of storage, one feature that I would like to see implemented by VMware is the ability to leverage VCB for NFS datastores. While many might question why you would want to use VCB for backing up NFS, when you can use the the same storage backup solution as the NFS host, there are good reasons. Backing up VMs becomes more difficult as the numbers increase. By leveraging VCB, the stress of backups is moved from the virtual infrastructure to a separate windows server. This could provide options such as backing up outside of the normal backup nightly backup window, or adding additional incremental or differential backups.
VMware ESXi guest unknown (invalid)
First, A little background of my home network; I have a two system network consisting of Dell workstation. The first runs ESXi 3.5 Update 2 and the second runs Open Solaris 2008.05. I utilize the ZFS for both the root and the data drives. I use an NFS share for my ESX datastore. Well today, while working from home, we experienced a brownout and down came both servers and my Vista workstation. I waited to boot the ESXi server until the Solaris box successfully booted. After booting, I started the ESXi server and was greeted by my two VMs stating unknown (invalid).
I tried browsing the datastore and was able to see my VMs. So I thought, OK, I’ll try to remove one of the VMs from inventory and re-import. ESX imported it and lo and behold……. Unknown (invalid). Ok, during this time, I checked my music folder on the workstation, which is an ISCSI mount on the same server. It looked fine. So I rebooted the ESX server to see if that helped. After the reboot, still no dice. I then logged into the Solaris server and decided to reboot it and then the ESXi server again. This time, we had a winner. My VMs were there with the correct names, and I was able to start them up. The VMware lock files were in place and weren’t cleared until after the storage reboot. The next time, I might try to un-mounting the NFS share and remounting.
Live and learn.
About mordtech
Experience:
I’m an IT professional with over ten years of Microsoft Server support. I’ve also supported Checkpoint and VMware. I’m currently employed by a large IT outsourcing corporation in the US.
I’ve implemented Windows Active Directory, Checkpoint VPN/Firewalls, VMware Infrastructure, Sharepoint/Project Server amongst others.
Personal:
I’m married and a father of two boys, which is by far the most trying job of my life: the boys, not the marriage. We reside in the midwest and aren’t looking forward to the fast approaching winter.
sitemeters
