Posts Tagged ‘vCenter’
After upgrading to vCenter 5 running on Windows Server 2008 R2 you may begin seeing an ADWS event with ID 1209 logged in the Active Directory Web Services event log within the Windows Event Viewer. This event will be logged once per minute with the following text:
“Active Directory Web Services encountered an error while reading the settings for the specified Active Directory Lightweight Directory Services instance. Active Directory Web Services will retry this operation periodically. In the mean time, this instance will be ignored. Instance name: ADAM_VMwareVCMSDS”
This event by itself is not something that should cause you to believe your vCenter installation is not working properly. Essentially what is occurring is that the VMwareVCMSDS ADAM instance does not have a valid SSL Port set within the registry. You can confirm this by browsing to the following registry location on your vCenter server:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\ADAM_VMwareVCMSDS\Parameters]
You should see that the Port SSL registry entry is either missing or does not contain a value. You can simply add the REG_DWORD value while you are already in the registry with the value of 636. Or you can use the below Microsoft approved method to add the port value.
Add ADAM_VMwareVCMSDS SSL Port Value
- Stop the vCenter ADAM instance within the Microsoft Services Control Panel. The service name is VMwareVCMSDS. Optionally, you simply run net stop vmwarevcmsds from an elevated command prompt.
- At a command prompt, type dsdbutil.
- Within the dsdbutil, type activate instance VMwareVCMSDS.
- Type SSL port 636.
- Type Quit.
- Type net start vmwarevcmsds.
Once the service starts, I highly suggest you reboot the vCenter server to allow all the vCenter services as well as the Microsoft Active Directory Web Services service to restart. If you don’t reboot the server you may encounter a fairly nondescript error message when attempting to logon using the vSphere Client until you reboot.
VMware has issued a KB for this particular issue:
Active Directory Web Services fails to read the settings for the specified Active Directory Lightweight Directory Services instance
Note: vCenter 5 introduced a new “Inventory” service that communicates over its own web services port and uses its own SSL certificate. vCenter 5 specific instructions will be noted below.
|
Step 1
|
To proceed with the below process you will need to install the latest version of OpenSSL on Windows/Linux or optionally you can leverage the OpenSSL install on a VMware Management Assistant (vMA) appliance.
Tip: If you install OpenSSL on Windows you will need to set the environment variable OPENSSL_CONF to the directory where the default openssl.cfg file is located (this is typically c:\OpenSSL-Win32\bin\openssl.cfg). In the default scenario, at the command-prompt type set OPENSSL_CONF=c:\openssl-win32\bin\openssl.cfg
You can confirm the environment variable is correct by simply typing set at the command prompt and looking for the OPENSSL_CONF line.
|
|
Step 2
|
The very first item you will need to create is the replacement certificate Private Key.
At a command-prompt type openssl genrsa 2048 > rui.key
This will create a file within your current working directory called “rui.key”—this is your private key.
|
|
Step 3
|
Using the private key you will need to create a Certificate Signing Request (CSR) used by your Certificate administrator (or alternately used by a public/commercial Certificate Authority) to issue the Public Key.
At the command-prompt type openssl req –new –key rui.key > rui.csr
You will be prompted for the following information:
- Country Name (2 letter code): US
- State or Province Name (full name): California
- Locality Name (eg City): San Francisco
- Organization Name (eg Company): DuckWorks
- Organizational Unit Name (eg Section): Information Technology
- Common Name (this is your fully qualified server name): vCenter.duckworks.com
- Email Address: <don’t enter one>
- A challenge password: <don’t enter one>
- An optional company name: <don’t enter one>
This will create a file within your current working directory called rui.key—this is your Certificate Signing Request (CSR).
|
|
Step 4
|
Using notepad or any file editor, open the file rui.csr you created in step 3 above. Copy the text starting with (including) —–BEGIN CERTIFICATE REQUEST—– and ending (including) —–END CERTIFICATE REQUEST—–.
|
|
Step 5
|
You are now going to create the replacement certificate Public Key using a Microsoft Certificate Authority.
- Browse to your Microsoft Certificate Authority website (usually https://<servername>/certsrv/). Note: Your Certificate Authority may not use https but may be accessible using http://<servername>/certsrv/.
- Select Request a Certificate.
- Select Advanced Certificate Request.
- Select Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file.
- Copy the contents gathered in Step 4 into the textbox (you can also browse to your CSR file as well).
- Select the Web Server Certificate Template.
- Select Submit.
- Select Base 64 encoded option.
- Select Download certificate
- Important: When saving the certificate make sure you rename the certificate to rui.crt (note the .CRT file extension—don’t leave .CER as the file extension). When saving a base64 type certificate, .cer and .crt are interchangeable.
You can open the rui.crt file within Windows and it should look similar to the following:
|
|
Step 6
|
Create a PFX (pkcs12) file containing the public and private key pairs.
- Copy the two files rui.key and rui.crt into a folder (it’s easier to keep the files together).
- Using a command-prompt, type openssl pkcs12 –export –in rui.crt –inkey rui.key –name rui –passout pass:testpassword –out rui.pfx (note: using “testpassword” is significant because it’s used as the keystore password in the Tomcat server.xml file—you can use a different password to secure the PFX file but you will need to update the server.xml file to match the password you used.)
- Copy all three files (rui.crt, rui.key, rui.pfx) to C:\ProgramData\VMware\Vmware VirtualCenter\SSL\ (WIN2008) (recommend archiving the existing SSL certificates).
- From the command-prompt, type net stop vpxd (This will stop your core vCenter service so make sure you have a maintenance window).
- From the command-prompt, change your directory (CD) to the installation path of vCenter.
- From the command-prompt, type vpxd –p (you will be prompted for the database password used on your ODBC connection—the password will be re-encrypted using the new certificate).
- From the command-prompt, type net start vpxd
|
|
Step 7
|
This step is for vCenter 5 only. Use the following steps to replace the certificate used by the vCenter 5 Inventory service. This process is simple because you can use the certificate generated using the steps above for the inventory service.
- Copy rui.key, rui.crt, and rui.pfx to your vCenter Inventory Service installation path (ex. C:\Program Files\VMware\Infrastructure\Inventory Service\SSL\)
- Restart the vCenter Inventory Service within the Windows Service Control panel (services.msc).
|
After following the above steps your vCenter server will now be using the new certificate for all web services. Additionally, you will no longer be presented with a certificate warning popup when using the vSphere Client if the certificate authority that issued the replacement certificate is trusted by your computer (in this specific case any domain joined computers will automatically trust all certificates issued by your internal enterprise Microsoft Certificate Authority).
I find it a bit interesting, but not surprising, that VMware is soon going to be releasing a native iPad application providing similar functionality that is found in the vSphere client for Windows. I wonder why VMware is utilizing programming resources on the iPad when they have yet to come to market with a supported Linux version of vCenter? For many years virtualization administrators in Linux/Unix shops have been telling VMware that vCenter and its associated database server and vSphere clients are the only Windows boxes they support and would very much appreciate a fully functional vCenter and client for Linux. Back in 2008, VMware was saying they would be releasing versions of vCenter running on Linux along with familiar Windows versions. Still waiting… (a limited CTP version does exist here).
I also wonder if VMware has given up on the vCenter Mobile Access (vCMA) product since it is still in a community technology preview version, there have been no major enhancements, and its been a long time since it was introduced. The vCMA had the “cool” factor when it was released–I remember showing people how I could vMotion a VM from one ESX host to another from my Blackberry. That cool factor faded away to the point where I haven’t used the vCMA in over a year–it’s just too kludgy to get anything done. Is the iPad vSphere application the new vCMA–the new vendor specific application that will introduce the iPad into corporate virtualization environments? Will it take over the functionality of the vCMA?
Don’t get me wrong, I see tremendous possibilities for the iPad within the corporate environment. The VMware vSphere iPad application could be very useful to large organizations that have lots of ESX hosts. Imagine an administrator being able to evacuate and place an ESX host in maintenance mode while troubleshooting a hardware issue within the datacenter (or from Hawaii on business). I can even see the VMware vSphere iPad application allowing virtualization administrators to manage a significant portion of their daily workload away from the office.
In conclusion, it’s great that VMware is working on new innovative ways to enhance access to vCenter from various devices; however, if I had my way I would rather VMware spend more time doing the following (in this order):
- Enhance the vCenter product for Windows. When I say “enhance” I mean work on the fit-and-finish of the product. All too often I am presented with ambiguous error messages or stumble on a failed process, yet the event reporting within vCenter can’t seem to tell me what’s wrong.
- Work on the overall performance of the vCenter UI. There are reports all over the Internet of the horrific performance within the vCenter user interface. I see it everyday. Viewing inventory takes 10 seconds to load up once the vCenter interface is visible (this doesn’t count the time to logon and load the plugins). Granted, performance is linked to hardware specifics and one must build an appropriate server environment to support vCenter; though, I am talking about poor performance on vCenter servers running with new multiple Xeon quad-cores with 8GB+ physical memory with a large dedicated physical DB server back-end.
- Enable performance monitoring across all hosts from a single UI window. Since a DRS cluster is essentially a pool of CPU and memory resources–why are we still required to troubleshoot performance by analyzing single ESX servers (think esxtop)?
- Enhance command-line troubleshooting tools. For example, an esxtop command that has a global view of clusters and storage. Yes it’s great to see the read/write MBps to a specific VMFS LUN but I want to see the total across all hosts not just the localized view of a single ESX host.
- Stop developing new features that are only added to the growing list of VMware products including vCloud Director, vCloud Request Manager, Orchestrator, CapacityIQ, Site Recovery Manager, Lab Manager, and Configuration Manager. Put some of the features in vCenter for continued value-add. For example, why haven’t we seen simple Virtual Machine replication in vCenter?
- Finish and release a fully functional Linux vCenter server with associated Linux vSphere client.
- Create better quality upgrade and patch bundles. Why do customers cross their fingers hoping everything is going to work as expected after upgrading vCenter or an ESX host? How many times have I seen an upgrade break vCenter (for example, certificates, web services, health monitoring)? Answer, many times.
- (last) Develop a mobile vSphere client.
One of the great new features of VMware vCenter 4.1 is Distributed Resource Scheduler (DRS) Groups. DRS Groups provides functionality that allows separation and placement of virtual machines onto specific ESX/ESXi hosts within a DRS cluster. Using DRS Groups, limiting the available hosts to a virtual machine or group of virtual machines is simple. Why might you want to use DRS Groups? I can think of many great scenarios where I could use DRS Groups; though, I will discuss one specific example regarding vCenter Server placement.
It is probably safe to assume most VMware administrators have implemented vCenter Server as a virtual machine within a DRS/HA cluster. A virtual vCenter Server running within a DRS/HA cluster provides many great high-availability and manageability benefits; however, there is a specific challenge that has not been solvable until vCenter Server 4.1. In the event vCenter Server become unavailable, an administrator would need to connect directly to an ESX/ESXi host using the vSphere Client where vCenter is located to manage the server there (i.e. open a console connection, restart the vCenter server, power the vCenter server up, etc.). But because vCenter Server is running inside a DRS cluster it is sometimes very time consuming locating the specific ESX/ESXi host where vCenter is running if you have many ESX/ESXi hosts within the DRS/HA cluster. For example, if there are 12 ESX/ESXi hosts running within a DRS cluster; the vCenter server could be running on any one of the 12—could you imagine using the vSphere Client and connecting to up to 12 hosts before locating the vCenter Server? Could you afford wasting 20 minutes during an emergency trying to locate vCenter Server?
Using DRS Groups vCenter Server can be limited to run on a limited number of ESX/ESXi hosts within a DRS cluster. For example, using DRS Groups an administrator can designate three of the 12 hosts where vCenter Server can run. In the event vCenter Server becomes unavailable it would be much easier to locate vCenter if you know it is primarily running on any of three hosts instead of 12 hosts.
The following VMware KB article provides a starting point for you to further investigate DRS Groups: http://kb.vmware.com/kb/1022842 .
This is Part 1 in a series of posts related to the definition and configuration of VMware Distributed Power Management.
The consolidation of physical servers into Virtual Machines provides significant hardware cost savings and also reduces power consumption throughout the datacenter. VMware Distributed Power Management (DPM) is yet another way to further increase return on investment by continuously analyzing the Virtual Infrastructure and consolidating Virtual Machines onto fewer hosts during times of low utilization. Hosts that are identified by DPM as underutilized are powered off. As capacity requirements increase, DPM powers on additional hosts to handle the load using either Wake-on-LAN (WOL), iLO, or through IPMI calls to a hardware Baseboard Management Controller (BMC).
How do you get started with DPM? First, you need the appropriate licensing that enables Distributed Resource Scheduler (DRS). This means you will need to license VMware vSphere Enterprise or VMware vSphere Enterprise Plus versions of ESX(i). Second, you will need to configure a VMware DRS cluster within vCenter made up of two or more ESX(i) 4.x hosts. Lastly, you will need to enable and configure the DPM feature on the DRS cluster.
The most difficult process to tackle when setting up DPM is selecting and configuring the “wake-up” solution. There are three options, WOL, iLO and IPMI. WOL has been around for many years and works fairly well; though, it is broadcast based. The physical NICs in the ESX(i) host must support WOL and WOL usually needs to be enabled in the BIOS. WOL will not be able to wake-up ESX(i) hosts if the vCenter server is not on the same subnet as the vMotion NICs unless you enable IP directed Broadcasts. WOL works by sending a UDP “magic packet” to all hosts on a subnet with a payload that describes which host should wake-up. All the hosts on the subnet analyze the packet and only the host the command was meant for powers on.
iLO and IPMI can be described together because they are very similar. iLO (Integrated Lights-out) and BMC IPMI (Intelligent Platform Management Interface) are out-of-band devices that allow an administrator or process to interact directly with a server at a hardware level. Because communication with these devices uses unicast they can be located on any subnet within your routed network. iLO devices typically contain more features than most basic IPMI Baseboard Management Controllers (ex. Console Redirection, Virtual Disks, etc.). Most iLO and IPMI solutions are configured in a similar way. At boot, you press a defined key combination to enter the configuration menu. You will need to configure the IP address, subnet, and gateway. Additionally, you may need to specify a VLAN if you are using trunking on your iLO or IPMI network interface. You will also need to define a username and password combination for access to the iLO or IPMI device. vCenter will require this information during the DPM configuration. Also, while you are configuring the device you will want to take note of the MAC address.
An often unknown but important piece of information is that most BMC IPMI devices share a servers internal integrated NICs. Typically, these integrated NICs are also being used by your ESX(i) host for Service Console, vMotion or Virtual Machine traffic. This is a perfectly acceptable configuration except in the following circumstance: On Dell hardware for instance; the BMC (the device which provides the IPMI interface) can only run at 100Mb when the server is powered off. It is common for the integrated NICs to operate at a speed of 1000Mb. It’s also very common for network administrators to force a speed and duplex setting on switch ports for servers at 1000Mb. What happens is that while ESX(i) is up the NICs will run at their designated 1000Mb speed. When the server is powered down the integrated NICs transition to a low power mode of 100Mb. When the switch port is configured for 1000Mb the BMC NICs disconnect and you lose the ability to remotely administer the BMC (i.e. a speed/duplex mismatch occurs). The solution is to enable AUTO/AUTO on the switch ports used by the BMC NICs. The server will negotiate 100Mb when powered off and will negotiate 1000Mb when ESX(i) is up.
In Part 2 of this DPM series we will configure the cluster for DRS and DPM.
I opened a VMware Service Request yesterday because I ran into a problem that I couldn’t figure out. In one of our environments, we have a 15 host DRS/HA cluster we are migrating from ESX 4 to ESXi 4. The process involves removing the existing ESX host from vCenter before rebuilding the host with ESXi. On two of the 15 hosts, the “Remove” option was greyed out. I first suspected a permissions issue was preventing the hosts from being removed. Though, it didn’t make sense that other hosts in the same cluster were able to be removed and the permissions where identical on the two hosts in question.
I tried disconnecting the hosts from vCenter and then remove–but no success there. I tried re-connecting the hosts to vCenter–still no success. I tried moving the hosts out of the cluster and placing it under the root vCenter object (i.e. the vCenter name)–again no success. I didn’t want to edit the backend database directly before I contacted VMware, so I opened a Service Request. After two days, VMware dug up an existing case that had recently been opened by a customer experiencing the same issue. The only way to cleanly remove the hosts is to create a new temporary cluster, place the hosts in question in this new cluster, and then delete the cluster. This process not only deletes the cluster but it also removes all the hosts underneath the cluster object.
VMware does not yet know what is causing this to occur or how widespread this issue is. They suspect it has something to do with Update Manager and some recent updates. If you run into this condition, it is rare, hopefully this article will help you resolve the problem.
