Imaging VMware ESX Guest using ImageX
VMware ESX has the ability to clone templates, and this is a great feature when using iSCSI or FC datastores. When leveraging NFS datastores however, you lose thin provisioning on those NFS datastores. One way to get around this is to use a third party imaging software.
Microsoft provides a free imaging package titled imageX. You can read about on technet from here. A quick synopsis is that, unlike other imaging software, ImageX focuses on the files instead of disk blocks. This allows ImageX to leverage a Single Instance Store (SIS). Image X will compress the first image at around 33 to 50% of the on disk size, and will store the image in a file with a WIM extension. The WIM file holds the SIS and also indexes of which blocks of data are associated with the image. The benefit of the SIS and imageX can be found when appending a second image to the WIM. imageX will create a second index in the WIM file. It will then start imaging the machine, it will find a file and compare that to the SIS. If the file is found, it will add a pointer to the new index and move on. If the file is not found, imageX will add the file into the SIS and then add a pointer in the new index and move on.
To use imagex, you will need to download the Windows Automated Installation Kit (WAIK). After installing the WAIK, you can follow the instructions found here, on svrops.com, to create a WinPE boot CD. Before you create the CD using he OSCDIMG, you will need to inject the Network and SCSI drivers required for ESX. To do this, you will first need to get the correct drivers. You can either scour the internet looking for them, or you can just select install VMware tools from a guest VM. On the Guest VM, open my computers, Open the CDrom, \program files\VMware\VMware Tools\Drivers\ and copy the Vmxnet and SCSI folders. Now, on the machine that you installed the WAIK on, run the following commands
peimg /inf=DRIVE:<location you copied the network driver>vmxnet.inf /image=DRIVE:<mount location of wimfile>
peimg /inf=DRIVE:<location you copied the network driver>vmscsi.inf /image=DRIVE:<mount location of wimfile>
you can also use the same commands to inject other drivers, if you are going to leverage the boot CD across other hardware also. After you’ve injected all of the drivers that you plan on, go ahead and complete the instructions found on svrops.com. Now, after the OSCDIMG command, you will have a bootable WINPE iso. One point, make sure that you when you run the unmount command, ensure that you use the /commit command. If not, all of your changes will be lost, and you get to do it again.
Image Capture
So now that you have a bootable winpe iso, with imagex and the correct drivers, what do you do. First, build a gold image(s) of Windows XP, vista, 2003 and/or 2008. Patch everything with the latest service packs, security patches, etc… Also, its best to build the initial gold image with multi processors. That way you can use the same image for single/multi processors with out needing to change the HAL. next sysprep the gold image. You can find instructions for running sysprep here. After running the sysprep, start the VM and mount the CDrom as the iso created.
After booting, you will need to mount a network share, use a command similar to: Net use m: \\<servername>\share.
Create a new VM manually. It doesn’t need to be identical, but ensure that the hard disks are large enough to hold the uncompressed data from the gold image. Next, boot to the ImageX WinPE iso created earlier. After booting up, you will need to run diskpart, you can find websites online that detail everything about diskpart, but to create a basic C drive, you will need to run the following commands.
Diskpart.exe
Select disk 0
Create partition primary size=<size of disk in Mbytes>
Select partition 1
Format fs=NTFS label=”Sys” Quick
Active
Now, for each additional disk, select disk <disk> and run all of the same commands as above except, change the label to a description of the drive. Also, run the active command on the sys drive. After you have configured all of the drives type exit to get back to the command prompt. From the command line, run the net use command again. Next, change to X:\program files\imagex\. Type imagex /apply m:\<wimfile>.wim <index number> c: /verify. After the image is applied, you can rerun the imagex /apply command change the index number and the drive. After complete, unmount CDrom and reboot the VM. You should now be greeted by the Windows mini-setup.
While not as fast as VMware builtin clone from template, it does allow you to continue to leverage the thin provisioning inherent in NFS datastores. In our environments, it takes roughly 20 minutes to build a Windows 2003 VM, versus about 10 minutes to build the same VM from template.
Leveraging Solaris 10 ZFS Functionality for ESXi
With the decision of VMware to release the ESXi version of the industry leading hypervisor for free, one might wonder where else can I save money and still get enterprise level performance. One area is in the storage. The Solaris ZFS file system is a great starting point for small businesses looking to move into enterprise level virtualization while still hitting a relatively small price point. Sun recently released three new products in the 7000 series of products: the 7110, 7210 and the 7410. While the 7210 and 7410 would make a techie drool. For the small business, the 7110 offers most of the benefits of the two larger versions, at a very affordable price. The 7110 starts at around $10,000 list and would rise to $16,000 with the three year platinum warranty and SCSI controller. Now, you could build a SUN server with similar performance characteristics, but you would lose one of the best features of the new Sun Storage; Storage Analytics. This feature is an enhancement of Dtrace that allows the administrator to drill down into the performance of the storage, to isolate performance issues.
So what do you have to do to get VMware ESXi and Solaris ZFS doing their thing together? Build your Solaris 10 Server. There are many good resources for available on the internet for configuring Solaris 10 and ZFS. Here are the instructions from SUN. Some basic recommendations: keep the system drives and the data drives separate. Use a mirrored ZFS pool for the system drive. Use RaidZ2 for the data drives, with a minimum of 5 drives: 3+2 and no more than 11 drives: 9+2. When you go above the 11 drive mark, you’ll just add an additional raid set to the ZFS pool. Leave a drive or two available for a spare(s) in the pool.
After you have your ZFS pool created.
You first need to first create the file system. All you really need to decide is: do you want a quota (Upper limit), reservation (minimum limit) and a name. you would enter the following command
zfs create <ZFSPool>/<fileSystemName>
You can get creative with the organization, if you know that you are going to have multiple NFS filesystems, you might enter zfs create pool/VM_NFS/FS01 or pool/VM_NFS/root. Ok, done, you’ve created a ZFS file system. Now you might want to set a quota and/or reservation.
zfs set quota=250G <ZFSPool>/<fileSystemName>
There you now have set a limit of that pool of 250 GB. Also, at this point, that is a thin provisioned size. Very little space will be utilized until data is written. The next step is to set a reservation, if so desired.
zfs set reservation=100G <ZFSPool>/<fileSystemName>
You now have a file system that has taken up 100 GB of the pool. While no data has actually be written to disk, the pool will show that the available size has been reduced by 100 GB. Now, the last step before we can start configuring the ESXi datastore is to mount the filesystem as a NFS share.
zfs set sharenfs=root=<IP address of the ESXi server> <ZFSPool>/<fileSystemName>
The key in the above command is the sharenfs=root=. That command will give the ESXi server root access to the NFS share. Without that, you will be able to mount the share, but you will not be able to create or open VMs hosted on the share. Ok that’s it, on to the ESXi server.
Build the ESXi Server
First download the ESXi server software. You can always download the free version, and then apply the purchased license purchased at a later date. Install ESXi per the instructions found here. Now, open your browser and go to the IP address that you assigned to the ESXi server from a Windows workstation. In the left hand corner of the page, you will see a link that states “Download VMware Infrastructure Client.” Go ahead and install the client. When it is complete, click on the shortcut and use the username Root and the password you created during the ESXi install. The next step is to install either the free license or a purchased license for the ESXi install. Left click the server name and select the configuration tab. Next select the “Licenses Feature” option. Select the top “Edit” to the right of “License Source.” Select the “Use License” option and either type or paste the license provided by VMware.
Next, you need to enable NTP time. You can either point it towards and internal time server if available, or towards public name servers. For my home network or other small businesses, I point towards the following: pool.ntp.org, 0.pool.ntp.org, 1.pool.ntp.org and 2.pool.ntp.org. You enter the NTP servers by selecting the Configuration tab, “Time Configuration” option and properties in the upper left hand corner. Ensure the “NTP Client Enabled” Checkbox is checked. Next, select the Options button, and select the “NTP Servers” option. Delete the default server. Click Add, type the first of the NTP server and repeat for each the remaining 3 NTP servers. Check the “Restart NTP service to apply changes.” Select OK twice and wait until the change is applied.
Next we need to configure the networking portion. If you are a small enough, you can get by with two network ports and push both client traffic and NFS traffic across them. You might notice latency if your traffic begins to exceed, if you experience this, you will need to add/configure additional NICs. Ensure that any traffic that involves clients and/or NFS traffic is redundant. Also, Vmotion and the NFS datastore traffic should be placed on GB interfaces on GB switch ports.
We are now ready to mount the NFS datastore. First select the configuration tab and select the Storage option. In the right hand corner, select “Add Storage.” Select the “Network File System” option and select Next. Next enter the IP address or Fully Qualified Domain Name (FQDN) of the Solaris NFS server. For the folder enter /<ZFSPool>/<fileSystemName>. Ensure the “Mount read only” is unchecked and then enter a easily recognizable name for the datastore. Select OK and you should have a datastore created that shows a size equal to the Quota configured earlier. Go ahead and create a VM, you done.
Other features of ZFS that is useful to ESXi
One cool feature of Solaris ZFS is the ability to make writable clones of file systems. Imagine building 1,2 5 VMs. Sysprep them, clone the file system and re-present the new cloned filesystem to the ESXi server. Or you could build a test environment of a web and SQL server that could be stood up in seconds. The clone process leverages the snapshot functionality.
zfs snapshot <ZFSPool>/<fileSystemName>@<snapshotName>
zfs clone <ZFSPool>/<filesystemName>@<snapshotName> <ZFSPool>/<fileSystemName1>
zfs set sharenfs=root=<IP address of the ESXi server> <ZFSPool>/<fileSystemName1>
zfs set quota=250G <ZFSPool>/<fileSystemName1>
zfs set reservation=100G <ZFSPool>/<fileSystemName1>
Now, you have a second file System that contains an exact copy of file system as it existed at the time of the snapshot. The snapshot name can be nearly anything you want, if you are creating a snapshot that will be use for cloning test environments, you might name the snapshot : webServer_SQL_Test_Gold. Also, if these are windows VMs, run sysprep against the servers and shut them down fully before creating the snapshot and the clone. Another thing to be aware of is that you are limited to 8 NFS datastores by default in ESX, to extend past that you will need to modify the NFS.maxvolumes under the Configuration tabà advanced setting. You can run up to 32 NFS datastores per ESX host/cluster.
You can also present iSCSI drives to your ESX clients from the ZFS pool.
On the Solaris Wiki site, you will find a great write-up about how to present ISCSI ZFS luns to various initiators.
If you have questions, concerns, gripes, etc… leave a comment and let me know.
Microsoft Licensing 3 – Clusters
In this post, I’ll discuss licensing when working in a clustered Microsoft environment. I’ll pick some of the more common Microsoft Apps and detail what is required to properly license them. Windows 2003 and 2008 support eight node clusters. In a two node cluster, you can technically have them configured in an active/active configuration. However, this is not considered best practice by Microsoft. They recommend running in an Active/Passive configuration. Three, four and five node configurations must have one passive node; the other nodes can be active. In the cluster, you can only have up to four active nodes, so nodes five through eight must be passive. All nodes must be licensed with either Windows Server Enterprise or Windows Server Datacenter. In most cases, Windows Server Enterprise makes the most financial sense. Enterprise has an MSRP of $4,000 per server up to eight physical processors versus $3,000 per physical processor using the Datacenter SKU. If there are more than 8 physical processors, you must use Datacenter.
Hyper-V cluster
The Datacenter SKU makes the most sense as the basis of your Hyper-V cluster in most Hyper-V environments. With the free unlimited guest OS licensing on each server, the breakeven point is 8 guest OS across the 2 node cluster. You receive the right to run 4 instances of Windows OS with each license of Enterprise. In a cluster, during a failover situation, more than 4 VMs might be running on the single node. Therefore, you would need to buy an additional 4 licenses of Standard. That would put Enterprise and datacenter, both at $12,000 for a 2 node cluster. Above 8 guests on the cluster, or when you are running copies of Enterprise on the guest VMs. If you have four physical processors, you would need to run roughly 20 guest VMs to break even.
This is a good point to add a quick discussion about virtualized environment hardware. Should you buy bigger 4 or 8 processor machines with multi-core or go wide in your cluster with dual processor boxes. When I’m designing clusters for a virtualized environment whether VMware Infrastructure or Hyper-V, I go wide first, and then scale up. My reasoning for this is simple, in both the VMware licensing model and Microsoft’s Datacenter licensing model; it is per socket costs. If you have 4 cluster nodes of dual processors or 2 cluster nodes of 4 processors. Both VMware and Microsoft will charge the same cost. Normally though the hardware costs to purchase two quad processor servers would be higher than four dual processor servers. You also gain the ability to have a higher utilization rate when going wide. In a two node cluster, you can only run at 50% capacity. But with a 4 node cluster, all four nodes can run at 75%. Another issue is that when running the larger hardware, a single physical server failure will shutdown basically 50% of your environment until they restart on the other node. When going wide, only about 25% of your environment will go down.
SQL Server
With SQL server, you would more than likely use the Windows 2003 Enterprise license. Unless you’re servers have more than eight processors, and if they do, you probably need this blog entry to explain licensing. Starting in SQL 2005, SQL enterprise is no longer a requirement for a SQL cluster. The Enterprise license now gives you additional features such as data warehousing. Microsoft is generous with SQL licensing in the cluster. You do not need to purchase a license for the passive node. License the active node with either per device or processor licenses and go. Again, in the same scenario as with Hyper-V, the licensing is the same whether you have two eight processor servers versus eight dual processors servers. Again, if you need more processing power on your database, you probably don’t need this blog. I should be talking to youJ.
Exchange
In a clustered environment of Exchange, you must run Exchange Server Enterprise per the Exchange Server 2007: Platforms, Editions and Versions web page. Also, you need one copy of Exchange for each node in the cluster. You do not get the benefit of not licensing passive nodes like you would on the SQL cluster. Exchange Enterprise licenses have an MSRP of $4,000 per server. As previously stated, and supported by Dell and IBM tests, Exchange does not scale well above 2 processors. So again take the cluster wide.
There are whitepapers by Dell and VMware that Exchange actually scales better in the Virtual environment than in physical. On a quad processor quad core IBM server, VMware was able to scale to 16,000 mailboxes. This was done with eight dual vCPU VMs each hosting 2,000 users. A blog discussing this can be found here. Dell wrote a similar paper on a dual quad core server.
My suggestions here; skip Microsoft clustering; get a couple of dual quad core processor servers and two licenses of VMware ESX 3.5. Load the servers up with as much ram as they will take and buy Exchange 2007 standard at $700 per VM. Build two Exchange servers using Standby Continuous Replication (SCR) between the two. Configure a Rule in the VMware cluster to put the two servers on different Physical ESX hosts. The VMware HA will protect you from a physical hardware failure; SCR will minimize the impact of an OS or application failure on the primary Exchange server. Of course you will still need additional servers for the other functionality in Exchange 2007: Edge transport, Client Access, Hub transport and Unified messaging. But with the cost savings of not buying additional servers, you can build standalone VMs to provide each piece of the Exchange environment.
SharePoint Server
With SharePoint Server, the best play would be to run a network load balance cluster for the SharePoint front ends and place databases on SQL cluster above. This will be a significantly cheaper solution as it will not required cluster able hardware and would only required Windows Server Standard instead of Windows Server Enterprise. It would also provide as high if not better uptime as a clustered front end.
SQL Server Pricing: http://www.microsoft.com/sqlserver/2005/en/us/pricing.aspx
Exchange Server licensing: http://www.microsoft.com/exchange/howtobuy/default.mspx
Exchange licensing comparison: http://technet.microsoft.com/en-us/library/bb232170.aspx
About mordtech
Experience:
I’m an IT professional with over ten years of Microsoft Server support. I’ve also supported Checkpoint and VMware. I’m currently employed by a large IT outsourcing corporation in the US.
I’ve implemented Windows Active Directory, Checkpoint VPN/Firewalls, VMware Infrastructure, Sharepoint/Project Server amongst others.
Personal:
I’m married and a father of two boys, which is by far the most trying job of my life: the boys, not the marriage. We reside in the midwest and aren’t looking forward to the fast approaching winter.
sitemeters
