Author Archive
I tend to recommend using ESXi versus ESX for several reasons. However, this week I was reminded of the shortfalls ESXi has yet to mitigate. First, in response to a complaint regarding slow storage performance I responded by gathering metrics using various tools available in vSphere (i.e. performance graphs, esxtop, etc.). I was quickly reminded vscsiStats functionality, a indispensable storage troubleshooting tool, is not available in ESXi. Scott Drummonds over at Pivot Point (blog) has provided vscsiStats binaries out-of-band that can be installed within an ESXi server. The problem is that applying these binaries to ESXi is not supported by VMware nor will VMware release security related patches for these unsupported binaries. There is no “supported” workaround for running vscsiStats in ESXi.
The second issue was in regards to troubleshooting a vMotion related problem with a virtual machine (well, what appeared at the time to be a vMotion issue). Basically, the virtual machine would not vMotion regardless of what was tried. Even after confirming no virtual devices were causing the problem the only solution was to power off the virtual machine and then perform the migration. I attempted to review the virtual machine vmware.log file after the virtual machine was powered back on. Unfortunately, the only way to read a vmware.log file is to view it directly from the console of the ESXi host that is running the virtual machine. Because SSH is not supported in ESXi (yes, it can be enabled) I was not able to read the vmware.log file remotely. There is no “supported” workaround to remotely view the vmware.log file when using ESXi.
These two issues alone can be deal-breakers for some.
UPDATE 1: VMware made huge steps towards closing the supportability and functionality gap with the ESXi 4.1 release. The two issues identified above have been mitigated as ESXi 4.1 allows supported command-line access locally and remotely via SSH. Additionally, I am happy to report the vscsiStats tool is now available and officially supported in ESXi 4.1 at /usr/lib/vmware/bin/vscsiStats. Great job VMware!!!
Are Virtual Machines free since I can run multiple independent instances of an Operating System in isolation on a single physical server? I can only pay for a physical server once; if I already paid for it, why can’t I consider each Virtual Machine a no-cost server installation?
The uncontrolled installation of perceived free virtual server installations is called virtual machine sprawl—by many accounts is a new epidemic in the datacenter with the advent and rapid adoption of Virtualization. If Virtual Machines are free, how do I pay for additional capacity requirements when I use all existing capacity? This article describes the costs associated with Virtual Machines and two strategies for calculating cost per Virtual Machine. This information can be used to recover hardware and software costs associated with virtualization and can also be used in reports and analyses of projected project costs.
In the mainframe days, Technology departments developed a method to recover technology costs by charging customers for services provided. This method of cost recovery is called the “chargeback”. Chargeback models are not perfect. In fact, Chargeback is often a point of contention within organizations depending on which side of the aisle you are on. For example, IT managers are typically fond of the chargeback model. It’s a way to prevent customers from demanding excessive services without first considering the cost impact. Business managers, however, see chargeback from the other side. Often the cost passed back to the customer includes a margin of profit which is looked at in a negative way because the business manager feels he is being asked to pay for “more” than the actual usage cost. Executive management typically falls in the middle. They like chargeback because independent accounting and performance metrics can be tracked back to individual business units and used as a scorecard. However, Executives struggle with chargeback because it’s “one organization”; the money used for IT capital expenditures is often allocated using an Enterprise strategy (i.e. “Business Unit A” needs a new database server and the money will come from a general IT expenditures account).
Assuming you still want to pursue a chargeback model you will need to figure out how to accurately account for resource usage in the virtual environment that covers actual costs, is perceived by business customers as fair, and also allows enough (profit) to purchase additional capacity and replacement hardware and software in future years. There are two models of Chargeback that work fairly well to recover costs associated with providing Virtual Infrastructure services. I will call these “Simple” and “Complex”.
In the Simple model, the cost per Virtual Machine is calculated as the Virtual Infrastructure is built-out. By totaling the costs associated with physical server hardware purchases, virtualization software licensing, Operating System licensing, storage costs, and any other costs associated with standing up a new virtual machine (i.e. maintenance and personnel costs) and dividing by the number of Virtual Machines that are expected to be supported on the Virtual Infrastructure over its lifetime, you can determine an estimated per Virtual Machine cost. This model is simple and easy to calculate but is largely unfair to customers because it doesn’t account for differences between actual resource usage. For example, Business Unit A uses a single-CPU Virtual Machine with 1GB of RAM and Business Unit B uses a four-CPU Virtual Machine with 8GB or RAM—though, both Business Units are paying the same.
In the Complex model, the cost per Virtual Machine is calculated using actual resource usage information of a Virtual Machine. This model is especially complex in a virtual environment because of the many different ways a Virtual Machine can be configured and deployed. Further, because Virtual Machines are often shuffling around physical servers as capacity requirement change, it’s impossible to use a simple ledger-style Chargeback method. To implement a fair and balanced Chargeback model using the Complex model, the use of Chargeback tools (VMware Chargeback and VKernel Chargeback) purpose-built for virtualization becomes a requirement. The Complex model requires the cost per Virtual Machine to be calculated using the Simple model described above as a first step. Actual CPU, Memory and Storage information is then collected as an additional cost and added to the basic per Virtual Machine cost. By using the complex model a fair and accurate per Virtual Machine cost can be associated to an individual Virtual Machine.
Virtual Machines are not “free”. Each Virtual Machine has hardware costs, software costs, infrastructure costs, personnel costs and other hidden costs (HVAC, Electricity, Datacenter footprint) that are all factors to consider when creating a Virtual Machine.
This is Part 1 in a series of posts related to the definition and configuration of VMware Distributed Power Management.
The consolidation of physical servers into Virtual Machines provides significant hardware cost savings and also reduces power consumption throughout the datacenter. VMware Distributed Power Management (DPM) is yet another way to further increase return on investment by continuously analyzing the Virtual Infrastructure and consolidating Virtual Machines onto fewer hosts during times of low utilization. Hosts that are identified by DPM as underutilized are powered off. As capacity requirements increase, DPM powers on additional hosts to handle the load using either Wake-on-LAN (WOL), iLO, or through IPMI calls to a hardware Baseboard Management Controller (BMC).
How do you get started with DPM? First, you need the appropriate licensing that enables Distributed Resource Scheduler (DRS). This means you will need to license VMware vSphere Enterprise or VMware vSphere Enterprise Plus versions of ESX(i). Second, you will need to configure a VMware DRS cluster within vCenter made up of two or more ESX(i) 4.x hosts. Lastly, you will need to enable and configure the DPM feature on the DRS cluster.
The most difficult process to tackle when setting up DPM is selecting and configuring the “wake-up” solution. There are three options, WOL, iLO and IPMI. WOL has been around for many years and works fairly well; though, it is broadcast based. The physical NICs in the ESX(i) host must support WOL and WOL usually needs to be enabled in the BIOS. WOL will not be able to wake-up ESX(i) hosts if the vCenter server is not on the same subnet as the vMotion NICs unless you enable IP directed Broadcasts. WOL works by sending a UDP “magic packet” to all hosts on a subnet with a payload that describes which host should wake-up. All the hosts on the subnet analyze the packet and only the host the command was meant for powers on.
iLO and IPMI can be described together because they are very similar. iLO (Integrated Lights-out) and BMC IPMI (Intelligent Platform Management Interface) are out-of-band devices that allow an administrator or process to interact directly with a server at a hardware level. Because communication with these devices uses unicast they can be located on any subnet within your routed network. iLO devices typically contain more features than most basic IPMI Baseboard Management Controllers (ex. Console Redirection, Virtual Disks, etc.). Most iLO and IPMI solutions are configured in a similar way. At boot, you press a defined key combination to enter the configuration menu. You will need to configure the IP address, subnet, and gateway. Additionally, you may need to specify a VLAN if you are using trunking on your iLO or IPMI network interface. You will also need to define a username and password combination for access to the iLO or IPMI device. vCenter will require this information during the DPM configuration. Also, while you are configuring the device you will want to take note of the MAC address.
An often unknown but important piece of information is that most BMC IPMI devices share a servers internal integrated NICs. Typically, these integrated NICs are also being used by your ESX(i) host for Service Console, vMotion or Virtual Machine traffic. This is a perfectly acceptable configuration except in the following circumstance: On Dell hardware for instance; the BMC (the device which provides the IPMI interface) can only run at 100Mb when the server is powered off. It is common for the integrated NICs to operate at a speed of 1000Mb. It’s also very common for network administrators to force a speed and duplex setting on switch ports for servers at 1000Mb. What happens is that while ESX(i) is up the NICs will run at their designated 1000Mb speed. When the server is powered down the integrated NICs transition to a low power mode of 100Mb. When the switch port is configured for 1000Mb the BMC NICs disconnect and you lose the ability to remotely administer the BMC (i.e. a speed/duplex mismatch occurs). The solution is to enable AUTO/AUTO on the switch ports used by the BMC NICs. The server will negotiate 100Mb when powered off and will negotiate 1000Mb when ESX(i) is up.
In Part 2 of this DPM series we will configure the cluster for DRS and DPM.
Now that you know what NAA identifiers are you may be wondering how to use NAAs in a meaningful way. You will likely utilize the NAA numbers when troubleshooting storage issues. For example, when a VM is performing poorly you will want to find out specific storage statistics related to the troublesome VM in question. Using the NAA identifier directly when troubleshooting is very difficult since the identifier is quite long. To make matters worse, each LUN NAA is very similar when multiple LUNs are presented from the same storage array. Another example is when troubleshooting storage issues related to SCSI reservations or sense code type problems; the VMware vmkernel and messages logs report storage issues by NAA in most cases.
Let’s assume you have a Virtual Machine named “ServerA” and it resides on a Datastore named “RAID_5_15K”. On Monday when you arrive to work you are greeted with customer complaints that ServerA is performing poorly. You start by checking the usual suspects; CPU and Memory. Both CPU and Memory look to be operating within normal ranges. So you think to yourself the performance issue may be related to storage. In vCenter you begin looking through the graphs related to storage and you quickly realize the graphs are showing LUN information by NAA identifier. At this point you are not sure which NAA is related to ServerA because all you know is that ServerA resides on Datastore “RAID_5_15K”.
Fortunately, VMware provides a couple different ways to map the NAA identifiers to Datastores—none of which are available through the built-in graphing tools. The most difficult way to identify the NAA to Datastore mapping is to use the vSphere client and by browsing to the ESX host Configuration tab and selecting Storage. In the list of Datastores you will see a column with the friendly user-defined Datastore name as well as the NAA identifier for each Datastore. A quicker way to find the NAA to Datastore mapping is to run the ESX(i) command “esxcfg-scsidevs -m” at the ESX Service Console or remotely using the vMA or Remote CLI and grepping (UNIX: grep) the Datastore in question. The esxcfg-scsidevs -m command provides a text based reference of all Datastores and their associated NAA identifiers. You can copy this information to a document and keep it around for quick reference.
Now that you have the NAA to Datastore map you can quickly identify the NAA that belongs to the Datastore “RAID_5_15K”. Using either the built-in graphing functionality in the vSphere Client GUI or the more robust real-time performance tool ESXTOP you can effectively identify storage issues that may be negatively impacting ServerA.
With the introduction of ESX(i) 4 and the included support of the SCSI-3 protocol comes a new way of displaying LUN information in the virtual environment called Network Naming Address Authority (NAA) identifiers. Each Fiber Channel or iSCSI LUN presented to an ESX(i) host has a unique SCSI NAA identifier. The NAA for each LUN is reported by the storage device to the ESX(i) host through an INQUERY which returns a VPD (Vital Product Data) response that includes the NAA identifier. Although the NAA is unique for each LUN, the NAA should remain consistent across all hosts for any particular LUN. The NAA identifier is generated by combining the word naa. with the 128-bit WWN of the LUN (ex. naa.600601607a401900d41ffaa5c17add11). In the preceding example, you can see the LUN is first identified by the word NAA; it is then followed by the WWN of the LUN. Because the WWN contains the IEEE assigned vendor OUI 006001 you can use a tool to find the LUN storage vendor (in this case “CLARIION”).
The length of the NAA presents a problem for the ESXTOP tool. ESXTOP is designed to show shorter device names (as seen in ESX 3.5 and earlier) when viewing “disk device” information. When you view disk device information through ESXTOP on an ESX(i) 4 host the NAA identifier is truncated which prevents the ability to identify a specific LUN (see truncated example below). ESXTOP contains an “L” option which allows users to specify the length of the Device field. By pressing the “L” key while in the “disk device” view and setting the value to 40, the NAA of each LUN become visible (see expanded example below).
| Truncated |
Expanded |
 |
 |
I opened a VMware Service Request yesterday because I ran into a problem that I couldn’t figure out. In one of our environments, we have a 15 host DRS/HA cluster we are migrating from ESX 4 to ESXi 4. The process involves removing the existing ESX host from vCenter before rebuilding the host with ESXi. On two of the 15 hosts, the “Remove” option was greyed out. I first suspected a permissions issue was preventing the hosts from being removed. Though, it didn’t make sense that other hosts in the same cluster were able to be removed and the permissions where identical on the two hosts in question.
I tried disconnecting the hosts from vCenter and then remove–but no success there. I tried re-connecting the hosts to vCenter–still no success. I tried moving the hosts out of the cluster and placing it under the root vCenter object (i.e. the vCenter name)–again no success. I didn’t want to edit the backend database directly before I contacted VMware, so I opened a Service Request. After two days, VMware dug up an existing case that had recently been opened by a customer experiencing the same issue. The only way to cleanly remove the hosts is to create a new temporary cluster, place the hosts in question in this new cluster, and then delete the cluster. This process not only deletes the cluster but it also removes all the hosts underneath the cluster object.
VMware does not yet know what is causing this to occur or how widespread this issue is. They suspect it has something to do with Update Manager and some recent updates. If you run into this condition, it is rare, hopefully this article will help you resolve the problem.
Are some of your ESX / ESXi hosts taking a long time to boot? On some of our ESXi 4.0 Update 1 hosts we are seeing boot times over 10 minutes. Watching the boot process we see the issue is related to storage; the ESXi DCUI appears to hang at “Loading module multiextent”.
Here is why:
During the boot process an ESX(i) host rescans its accessible LUNs. If you are using MSCS clusters with Raw Device Mappings (RDMs) you will likely experience a lengthy delay during the scanning process at boot time. The issue is caused by a timeout condition during the rescan operation on RDM LUNs. In my experience you will see the slow boot issue on all hosts that are zoned to see MSCS RDM LUN(s). For example, if you have a DRS/HA cluster with 15 hosts and are using MSCS with RDMs; within that cluster you will see slow boot times on all the cluster hosts. This occurs because all hosts in a DRS/HA cluster should have access to all common datastores.
You can mitigate this issue (not resolve it) by implementing a parameter change recommended in VMware internal KB1016106. The Scsi.UWConflictRetries parameter for ESX(i) 4 Update 1 hosts has a default value of 1000. This increases the time spent enumerating LUN and VMFS volumes.
Follow the steps below:
The Scsi.UWConflictRetries parameter for ESX and ESXi 4 Update 1 hosts have a default value of 1000.
To resolve this issue and speed up the boot process, modify this value to 80.
Click on Host -> Configuration -> Advanced Settings
In the Advanced Settings -> Select SCSI
Now, Change the Scsi.UWConflictRetries value to 80(Default is 1000).
As an alternative, you might consider creating a DRS/HA cluster dedicated to MSCS Virtual Machines and mask the RDM LUNs from all your other ESX(i) hosts not participating in the dedicated MSCS DRS/HA cluster.
It’s been a longstanding debate…which version of VMware’s baremetal hypervisor should I choose–ESX or ESXi? Which one is better? Why would VMware offer two seemingly identical versions of their core hypervisor product? There are answers to all these questions and I will address them in this article. By the time you are finished with this article you should have a good understanding of each product and will have the necessary knowledge to select the best option for your particular requirements.
Lets begin by understanding VMware’s reasons for offering both ESX and ESXi. VMware ESX and ESXi are very similar in many ways but one. ESX contains a Linux operating system VMware calls the “Service Console” that is used as a management application to interact directly with the hypervisor. ESXi does not contain a Service Console; therefore, management tasks are performed through other means–mainly using remote tools via an exposed ESXi API. According to VMware, the ESX version is legacy and will be phased out in the future. ESXi is the so-called “next generation” hypervisor version. VMware is keeping ESX around because many companies have created third-party applications/agents that must be run from inside the Service Console and also because customers are not totally sold on running enterprise workloads without access to the Service Console in time of an emergency. These two issues alone have contributed to preventing VMware from decommissioning the ESX version.
So what’s the difference between ESX and ESXi? As previously mentioned, the main difference between ESX and ESXi is the lack of the Service Console in the ESXi version. Because ESXi natively lacks many of the features provided by the Service Console, many see ESXi as a lesser version of ESX used in smaller environments. However, in reality, ESX and ESXi are identical from a core functionality standpoint. Below are some pros and cons of the ESXi version:
Pros
- Smaller installation footprint with reduced installation complexity. The manual installation of the “installable” version of ESXi takes about five minutes from baremetal to hypervisor. The “embedded” version of ESXi is ready out of the box with no installation media required. ESX on the other hand takes considerable time creating the Service Console partitions during installation.
- ESXi is more secure than ESX. Why? By eliminating the Service Console, ESXi requires fewer patches and also reduces the number of open ports as compared to ESX.
- The embedded version of ESXi requires less energy because the hypervisor runs from an internal USB memory module which eliminates the need to have expensive energy consuming physical hard disks. It’s also worth mentioning in larger environments a substantial cost-savings is obtained with ESXi embedded as expensive hard disks and RAID controllers can be eliminated from server purchases.
- Some have mentioned boot-time is reduced when using the ESXi version because the services that are started with the Service Console are not present in ESXi. This is hit-or-miss in my experience. In some environments where MSCS clusters are being used with RDMs the boot time is substantial regardless of VMware hypervisor version used.
- Many hardware vendors have created VIB files that contain OEM CIM providers that allow the ability to interact with vendor specific hardware sensors remotely. For example, one of the major concerns for Dell customers using ESXi used to be that the Dell OpenManage Server Administrator tool could not be installed inside ESXi (note: Dell did release a Dell specific version of ESXi 3.5 with the Server Administrator embedded). With ESXi 4 Dell has released a vSphere VIB file that allows administrators to remotely monitor Dell hardware instrumentation using the familiar Dell Server Administrator web interface. The benefit of this model is that the Dell OpenManage Tomcat web server is no longer required to be running on the ESXi server which further reduces the attack surface. Since the OEM CIM providers are packaged as a VIB file future updates should be as easy as installing a typical ESXi security patch.
Cons
- Administrators don’t want to give up the Service Console. The Service Console is like a giant safety net an administrator can always count on. It’s there when something goes wrong with the hypervisor unexpectedly. In a worse case scenario and administrator can logon to the Service Console at the server console and interact with the hypervisor and troubleshoot and correct most issues. Conversely, ESXi exposes an API so that remote tools such as the vMA, vSphere PowerCLI, and the vCLIcan perform many of the management functions that are provided by the ESX Service Console. Unfortunately, if the API services are unavailable due to an issue with the hypervisor how do you interact with ESXi? That’s were the ESXi Direct Console User Interface (DCUI) is necessary. By logging onto the DCUI in “unsupported” mode, administrators have access to some of the core esxcfg-* troubleshooting commands.
- Because of the lack of a Service Console troubleshooting ESXi is sometimes more difficult. For example, the vmkfstools command is not natively available in ESXi–it must be run from a remote console. Further, ESXi has limited native logging capabilities as compared to ESX. It is necessary to setup a syslog server or use the vilogger daemon in the vMA when using ESXi for long-term log archiving.
- Some consider the embedded version of ESXi to contain a single point of failure (SPOF). Because ESXi embedded is installed on some form of memory such as USB–it stands to reason that if the USB memory were to fail the ESXi hypervisor instance would also fail. At this time there is no ESXi embedded redundancy that might normally be provided hardware RAID. The embedded version can however be configured to boot from SAN which mitigates the SPOF argument. In the event of an unlikely USB failure VMware HA would potentially be able to start the failed Virtual Machines on alternate hardware (assuming you are licensed and using an HA cluster).
- Some third-party software cannot be used with ESXi because of dependencies on the Service Console. Most examples would be related to backup software where a single backup agent is installed inside the Service Console and is responsible for coordinating the backups of Virtual Machines residing on the host. Another example is where storage administrators require the use of agents on hosts connected to enterprise storage (i.e. Navisphere Naviagent, etc.).
Based upon the information in this article how do I choose between ESX and ESXi? If you do not have any requirements to install third-party software that uses the ESX Service Console (i.e. backup agents, storage agents, etc.) I would strongly recommend you install the ESXi version. The reduced attack surface, reduced security patch requirements, and reduced complexity of ESXi far outweigh issues related to the inability to directly administer an ESXi hypervisor. In the unlikely event an ESXi server experiences an issue that cannot be resolved using the vSphere Client (the GUI management tool) there are many tools that can be used that provide Service Console-like features (i.e. vMA, PowerCLI, vCLI, DCUI). Further, remember that VMware will be eliminating the Service Console in the future and it would make good business sense to start building processes and procedures to support ESXi now before you are forced too.
