Note: vCenter 5 introduced a new “Inventory” service that communicates over its own web services port and uses its own SSL certificate. vCenter 5 specific instructions will be noted below.
|
Step 1
|
To proceed with the below process you will need to install the latest version of OpenSSL on Windows/Linux or optionally you can leverage the OpenSSL install on a VMware Management Assistant (vMA) appliance.
Tip: If you install OpenSSL on Windows you will need to set the environment variable OPENSSL_CONF to the directory where the default openssl.cfg file is located (this is typically c:\OpenSSL-Win32\bin\openssl.cfg). In the default scenario, at the command-prompt type set OPENSSL_CONF=c:\openssl-win32\bin\openssl.cfg
You can confirm the environment variable is correct by simply typing set at the command prompt and looking for the OPENSSL_CONF line.
|
|
Step 2
|
The very first item you will need to create is the replacement certificate Private Key.
At a command-prompt type openssl genrsa 2048 > rui.key
This will create a file within your current working directory called “rui.key”—this is your private key.
|
|
Step 3
|
Using the private key you will need to create a Certificate Signing Request (CSR) used by your Certificate administrator (or alternately used by a public/commercial Certificate Authority) to issue the Public Key.
At the command-prompt type openssl req –new –key rui.key > rui.csr
You will be prompted for the following information:
- Country Name (2 letter code): US
- State or Province Name (full name): California
- Locality Name (eg City): San Francisco
- Organization Name (eg Company): DuckWorks
- Organizational Unit Name (eg Section): Information Technology
- Common Name (this is your fully qualified server name): vCenter.duckworks.com
- Email Address: <don’t enter one>
- A challenge password: <don’t enter one>
- An optional company name: <don’t enter one>
This will create a file within your current working directory called rui.key—this is your Certificate Signing Request (CSR).
|
|
Step 4
|
Using notepad or any file editor, open the file rui.csr you created in step 3 above. Copy the text starting with (including) —–BEGIN CERTIFICATE REQUEST—– and ending (including) —–END CERTIFICATE REQUEST—–.
|
|
Step 5
|
You are now going to create the replacement certificate Public Key using a Microsoft Certificate Authority.
- Browse to your Microsoft Certificate Authority website (usually https://<servername>/certsrv/). Note: Your Certificate Authority may not use https but may be accessible using http://<servername>/certsrv/.
- Select Request a Certificate.
- Select Advanced Certificate Request.
- Select Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file.
- Copy the contents gathered in Step 4 into the textbox (you can also browse to your CSR file as well).
- Select the Web Server Certificate Template.
- Select Submit.
- Select Base 64 encoded option.
- Select Download certificate
- Important: When saving the certificate make sure you rename the certificate to rui.crt (note the .CRT file extension—don’t leave .CER as the file extension). When saving a base64 type certificate, .cer and .crt are interchangeable.
You can open the rui.crt file within Windows and it should look similar to the following:
|
|
Step 6
|
Create a PFX (pkcs12) file containing the public and private key pairs.
- Copy the two files rui.key and rui.crt into a folder (it’s easier to keep the files together).
- Using a command-prompt, type openssl pkcs12 –export –in rui.crt –inkey rui.key –name rui –passout pass:testpassword –out rui.pfx (note: using “testpassword” is significant because it’s used as the keystore password in the Tomcat server.xml file—you can use a different password to secure the PFX file but you will need to update the server.xml file to match the password you used.)
- Copy all three files (rui.crt, rui.key, rui.pfx) to C:\ProgramData\VMware\Vmware VirtualCenter\SSL\ (WIN2008) (recommend archiving the existing SSL certificates).
- From the command-prompt, type net stop vpxd (This will stop your core vCenter service so make sure you have a maintenance window).
- From the command-prompt, change your directory (CD) to the installation path of vCenter.
- From the command-prompt, type vpxd –p (you will be prompted for the database password used on your ODBC connection—the password will be re-encrypted using the new certificate).
- From the command-prompt, type net start vpxd
|
|
Step 7
|
This step is for vCenter 5 only. Use the following steps to replace the certificate used by the vCenter 5 Inventory service. This process is simple because you can use the certificate generated using the steps above for the inventory service.
- Copy rui.key, rui.crt, and rui.pfx to your vCenter Inventory Service installation path (ex. C:\Program Files\VMware\Infrastructure\Inventory Service\SSL\)
- Restart the vCenter Inventory Service within the Windows Service Control panel (services.msc).
|
After following the above steps your vCenter server will now be using the new certificate for all web services. Additionally, you will no longer be presented with a certificate warning popup when using the vSphere Client if the certificate authority that issued the replacement certificate is trusted by your computer (in this specific case any domain joined computers will automatically trust all certificates issued by your internal enterprise Microsoft Certificate Authority).
Stumbled upon an issue with the latest release of vSphere ESXi 4.1 Update 1 (fully patched) where following VMware KB 1032666 to modify ESXi default password hashing from MD5 to something stronger such as SHA-256 or SHA-512. Some federal government agencies cannot use MD5 for password hashing since it considered cracked (see wikipedia MD5). Tried two different “approved” ways to edit the system-auth PAM file. One, use [#chmod 644 system-auth] to set permissions on the file so that it is user editable (or just use :wq! after editing). Two, use [#chmod +t system-auth] before editing. Unfortunately, after a reboot the system-auth file returns back to its pre-edited version.
I opened an SR with VMware and they in turn opened a PR and shortly thereafter confirmed the issue is not by design and in-fact is a bug. VMware estimates that this issue will be resolved in the Update 2 release of vSphere ESXi 4.1. Haven’t had a chance to see if the issue is present in vSphere ESXi 5.0.
Why is it that Microsoft makes creating and issuing certificates from a Microsoft Certificate Authority (CA) so easy yet makes managing the certificate lifecycle (revocation, expiration, renewal) so difficult? In less than 30 minutes, somebody with a basic knowledge of Windows can successfully create and begin issuing digital certificates from an internal Microsoft CA. The problems start once an administrator realizes that issued certificates are very difficult to track and manage–using a largely manual process. For example, assume a certificate is issued to provide SSL over HTTPS for a web server. The certificate template is likely going to specify the certificate is good for at least two (2) years. When you least expect it two years pass and the certificate unexpectedly expires and causes a service outage. Now extrapolate a bit, consider an environment where you have hundreds of webservers and thousands of issued certificates. All of the sudden a “manual” process is no longer viable.
Microsoft surely must have provided a way to directly manage their CA implementation through script…right? Unfortunately they didn’t. Microsoft provides access to their CA through a command-line tool “certutil” and also minimally through COM. Management through COM is possible but likely a complicated endeavor for the non-programmer. Ideally, Microsoft would have provided an easily accessible API that’s consumable by VBScript or Powershell. However, with recent changes to the certutil command-line tool in Windows Server 2008, Windows Vista and Windows 7, and some quick Powershell magic a decent certificate management tool is possible.
The following Powershell script exports the full list of issued certificates from a Microsoft CA using certutil and saves the certificate information as a CSV file. The CSV file is then imported and each certificate is checked to see if it falls within the certificate expiration window (definable in script). An e-mail is generated that contains basic certificate information for certificates that fall within the expiration window. This script can be scheduled to run using Microsoft Task Scheduler. Remember, this will only work if run on a Windows Server 2008, Windows Vista or Windows 7 computer due to the “CSV” option only being available in later versions of certutil.
# variables
[string]$caServerName = "<your ca server name\ca name>"
[string]$caCertExportPath = "c:\Temp\certlist.csv"
[string]$smtpSender = "no-reply@your-domain.com"
[string]$smtpRecipient = "user@your-domain.com"
[string]$smtpServer = "<fqdn of SMTP server>"
[int]$daysUntilExpiry = 30
$expiringCerts = @()
function Send-EmailCertNotice ([string]$_certificateList) {
$MailMessage = @{
To = $smtpRecipient
From = $smtpSender
Subject = @"
The following issued digital certificates will expire soon.
"@
Body = @"
The following digital certificates issued by <your_company_name> will expire in the next ($daysUntilExpiry) days.
Please request a certificate replacement/renewal from <e-mail@your-domain.com> if the following certificates are still needed.
$_certificateList
NOTE: This notification is being sent by an automated certificate management process and
cannot receive reply e-mail. If you have any questions please contact <e-mail@your-domain.com>.
"@
Smtpserver = $smtpServer
BodyAsHtml = $false
ErrorAction = "SilentlyContinue"
}
Send-MailMessage @MailMessage
}
# export certs to CSV file
certutil -view -config $caServerName csv > $caCertExportPath
# load cert CSV into an array
$issuedCerts = Import-Csv $caCertExportPath
if ($issuedCerts.Length -gt 0) {
foreach ($cert in $issuedCerts) {
try {
$certExpires = [datetime]$cert."Certificate Expiration Date"
$cert."Certificate Expiration Date" = $certExpires
}
catch [Exception] {
}
if ($certExpires -gt $(Get-Date) -and $certExpires -lt $(Get-Date).AddDays($daysUntilExpiry)) {
# filter out EFS type certs.
if ($cert."Certificate Template" -ne "EFS") {
$expiringCerts += $cert
}
}
}
$bodyVal = $expiringCerts | Select-Object @{n="Certificate ID"; e="Request ID"},
"Certificate Template", "Certificate Expiration Date", "Issued Common Name",
"Serial Number" | Sort-Object "Certificate Expiration Date" | Out-String
Send-EmailCertNotice $bodyVal
}
This document is the public draft release of the vSphere 4.1 Security Hardening Guide. This guide is an incremental update to the vSphere 4.0 Security Hardening Guide based on new and changed features of vSphere. Please provide your feedback in the comments section. This draft will remain posted for comments until approximately the end of February 2011.
Link to document: vSphere 4.1 Security Hardening Guide (draft)
I think I have some form of “web browser OCD”… Despite having a choice between lots of options I can’t seem to find a single web browser that works for all my needs. Over the last year I have jumped between Internet Explorer (IE), Firefox, Chrome, Safari and some other less known browsers, typically using each for about a week before I give up due to some minor issue. I have found that I keep going back to using IE even though I tend to enjoy browsing the web with Firefox and Chrome much more. Why? Mostly due to compatibility issues. Lets face it, the majority (57.08%) browsers used on the Internet are IE, as a result, web designers continue building webpages for the masses who are likely using IE. I especially have problems with banking websites; some of which unfortunately use ActiveX controls for extended functionality like TWAIN scanning.
Of all the browser options, I tend to enjoy Google’s Chrome browser the most. Though, I keep giving up on Chrome because I use a lot of bookmarks and Chrome has historically handled bookmarks poorly. For example, within the native Chrome UI one must set the homepage to “Use the new tab page” in order to see a tree-view style bookmarks menu. Alternately, a Chrome user could access his bookmarks using the Bookmark Manager, though, the Bookmark Manager by-design fills the entire web UI and clicking on any one bookmark opens the webpage in a new tab instead of in the same tab the Bookmark Manager is running in–not a sleek implementation in my opinion.
Funny thing, even though I already extend Chrome with extensions I never considered looking for a Chrome extension that provided a tree-view style bookmark menu until a couple days ago. I stumbled across an extension called Neat Bookmarks which provides a great bookmark UI experience similar to IE’s bookmark tree-view and is accessible as a single toolbar button. If you are looking for a more traditional bookmark menu in Chrome I recommend you give Neat Bookmarks a try. I hope now that I have found a solution to my bookmark woes in Chrome I can start focusing on other issues other than browser brand.
Mattias Sundling over at Quest has put together a brief whitepaper titled “Maximizing VM Performance”. It is not intended to be a deep-dive but it covers the main VM performance considerations in plain terms.
Check it out here: Maximizing VM Performance
I find it a bit interesting, but not surprising, that VMware is soon going to be releasing a native iPad application providing similar functionality that is found in the vSphere client for Windows. I wonder why VMware is utilizing programming resources on the iPad when they have yet to come to market with a supported Linux version of vCenter? For many years virtualization administrators in Linux/Unix shops have been telling VMware that vCenter and its associated database server and vSphere clients are the only Windows boxes they support and would very much appreciate a fully functional vCenter and client for Linux. Back in 2008, VMware was saying they would be releasing versions of vCenter running on Linux along with familiar Windows versions. Still waiting… (a limited CTP version does exist here).
I also wonder if VMware has given up on the vCenter Mobile Access (vCMA) product since it is still in a community technology preview version, there have been no major enhancements, and its been a long time since it was introduced. The vCMA had the “cool” factor when it was released–I remember showing people how I could vMotion a VM from one ESX host to another from my Blackberry. That cool factor faded away to the point where I haven’t used the vCMA in over a year–it’s just too kludgy to get anything done. Is the iPad vSphere application the new vCMA–the new vendor specific application that will introduce the iPad into corporate virtualization environments? Will it take over the functionality of the vCMA?
Don’t get me wrong, I see tremendous possibilities for the iPad within the corporate environment. The VMware vSphere iPad application could be very useful to large organizations that have lots of ESX hosts. Imagine an administrator being able to evacuate and place an ESX host in maintenance mode while troubleshooting a hardware issue within the datacenter (or from Hawaii on business). I can even see the VMware vSphere iPad application allowing virtualization administrators to manage a significant portion of their daily workload away from the office.
In conclusion, it’s great that VMware is working on new innovative ways to enhance access to vCenter from various devices; however, if I had my way I would rather VMware spend more time doing the following (in this order):
- Enhance the vCenter product for Windows. When I say “enhance” I mean work on the fit-and-finish of the product. All too often I am presented with ambiguous error messages or stumble on a failed process, yet the event reporting within vCenter can’t seem to tell me what’s wrong.
- Work on the overall performance of the vCenter UI. There are reports all over the Internet of the horrific performance within the vCenter user interface. I see it everyday. Viewing inventory takes 10 seconds to load up once the vCenter interface is visible (this doesn’t count the time to logon and load the plugins). Granted, performance is linked to hardware specifics and one must build an appropriate server environment to support vCenter; though, I am talking about poor performance on vCenter servers running with new multiple Xeon quad-cores with 8GB+ physical memory with a large dedicated physical DB server back-end.
- Enable performance monitoring across all hosts from a single UI window. Since a DRS cluster is essentially a pool of CPU and memory resources–why are we still required to troubleshoot performance by analyzing single ESX servers (think esxtop)?
- Enhance command-line troubleshooting tools. For example, an esxtop command that has a global view of clusters and storage. Yes it’s great to see the read/write MBps to a specific VMFS LUN but I want to see the total across all hosts not just the localized view of a single ESX host.
- Stop developing new features that are only added to the growing list of VMware products including vCloud Director, vCloud Request Manager, Orchestrator, CapacityIQ, Site Recovery Manager, Lab Manager, and Configuration Manager. Put some of the features in vCenter for continued value-add. For example, why haven’t we seen simple Virtual Machine replication in vCenter?
- Finish and release a fully functional Linux vCenter server with associated Linux vSphere client.
- Create better quality upgrade and patch bundles. Why do customers cross their fingers hoping everything is going to work as expected after upgrading vCenter or an ESX host? How many times have I seen an upgrade break vCenter (for example, certificates, web services, health monitoring)? Answer, many times.
- (last) Develop a mobile vSphere client.
One of the great new features of VMware vCenter 4.1 is Distributed Resource Scheduler (DRS) Groups. DRS Groups provides functionality that allows separation and placement of virtual machines onto specific ESX/ESXi hosts within a DRS cluster. Using DRS Groups, limiting the available hosts to a virtual machine or group of virtual machines is simple. Why might you want to use DRS Groups? I can think of many great scenarios where I could use DRS Groups; though, I will discuss one specific example regarding vCenter Server placement.
It is probably safe to assume most VMware administrators have implemented vCenter Server as a virtual machine within a DRS/HA cluster. A virtual vCenter Server running within a DRS/HA cluster provides many great high-availability and manageability benefits; however, there is a specific challenge that has not been solvable until vCenter Server 4.1. In the event vCenter Server become unavailable, an administrator would need to connect directly to an ESX/ESXi host using the vSphere Client where vCenter is located to manage the server there (i.e. open a console connection, restart the vCenter server, power the vCenter server up, etc.). But because vCenter Server is running inside a DRS cluster it is sometimes very time consuming locating the specific ESX/ESXi host where vCenter is running if you have many ESX/ESXi hosts within the DRS/HA cluster. For example, if there are 12 ESX/ESXi hosts running within a DRS cluster; the vCenter server could be running on any one of the 12—could you imagine using the vSphere Client and connecting to up to 12 hosts before locating the vCenter Server? Could you afford wasting 20 minutes during an emergency trying to locate vCenter Server?
Using DRS Groups vCenter Server can be limited to run on a limited number of ESX/ESXi hosts within a DRS cluster. For example, using DRS Groups an administrator can designate three of the 12 hosts where vCenter Server can run. In the event vCenter Server becomes unavailable it would be much easier to locate vCenter if you know it is primarily running on any of three hosts instead of 12 hosts.
The following VMware KB article provides a starting point for you to further investigate DRS Groups: http://kb.vmware.com/kb/1022842 .
I tend to recommend using ESXi versus ESX for several reasons. However, this week I was reminded of the shortfalls ESXi has yet to mitigate. First, in response to a complaint regarding slow storage performance I responded by gathering metrics using various tools available in vSphere (i.e. performance graphs, esxtop, etc.). I was quickly reminded vscsiStats functionality, a indispensable storage troubleshooting tool, is not available in ESXi. Scott Drummonds over at Pivot Point (blog) has provided vscsiStats binaries out-of-band that can be installed within an ESXi server. The problem is that applying these binaries to ESXi is not supported by VMware nor will VMware release security related patches for these unsupported binaries. There is no “supported” workaround for running vscsiStats in ESXi.
The second issue was in regards to troubleshooting a vMotion related problem with a virtual machine (well, what appeared at the time to be a vMotion issue). Basically, the virtual machine would not vMotion regardless of what was tried. Even after confirming no virtual devices were causing the problem the only solution was to power off the virtual machine and then perform the migration. I attempted to review the virtual machine vmware.log file after the virtual machine was powered back on. Unfortunately, the only way to read a vmware.log file is to view it directly from the console of the ESXi host that is running the virtual machine. Because SSH is not supported in ESXi (yes, it can be enabled) I was not able to read the vmware.log file remotely. There is no “supported” workaround to remotely view the vmware.log file when using ESXi.
These two issues alone can be deal-breakers for some.
UPDATE 1: VMware made huge steps towards closing the supportability and functionality gap with the ESXi 4.1 release. The two issues identified above have been mitigated as ESXi 4.1 allows supported command-line access locally and remotely via SSH. Additionally, I am happy to report the vscsiStats tool is now available and officially supported in ESXi 4.1 at /usr/lib/vmware/bin/vscsiStats. Great job VMware!!!
Are Virtual Machines free since I can run multiple independent instances of an Operating System in isolation on a single physical server? I can only pay for a physical server once; if I already paid for it, why can’t I consider each Virtual Machine a no-cost server installation?
The uncontrolled installation of perceived free virtual server installations is called virtual machine sprawl—by many accounts is a new epidemic in the datacenter with the advent and rapid adoption of Virtualization. If Virtual Machines are free, how do I pay for additional capacity requirements when I use all existing capacity? This article describes the costs associated with Virtual Machines and two strategies for calculating cost per Virtual Machine. This information can be used to recover hardware and software costs associated with virtualization and can also be used in reports and analyses of projected project costs.
In the mainframe days, Technology departments developed a method to recover technology costs by charging customers for services provided. This method of cost recovery is called the “chargeback”. Chargeback models are not perfect. In fact, Chargeback is often a point of contention within organizations depending on which side of the aisle you are on. For example, IT managers are typically fond of the chargeback model. It’s a way to prevent customers from demanding excessive services without first considering the cost impact. Business managers, however, see chargeback from the other side. Often the cost passed back to the customer includes a margin of profit which is looked at in a negative way because the business manager feels he is being asked to pay for “more” than the actual usage cost. Executive management typically falls in the middle. They like chargeback because independent accounting and performance metrics can be tracked back to individual business units and used as a scorecard. However, Executives struggle with chargeback because it’s “one organization”; the money used for IT capital expenditures is often allocated using an Enterprise strategy (i.e. “Business Unit A” needs a new database server and the money will come from a general IT expenditures account).
Assuming you still want to pursue a chargeback model you will need to figure out how to accurately account for resource usage in the virtual environment that covers actual costs, is perceived by business customers as fair, and also allows enough (profit) to purchase additional capacity and replacement hardware and software in future years. There are two models of Chargeback that work fairly well to recover costs associated with providing Virtual Infrastructure services. I will call these “Simple” and “Complex”.
In the Simple model, the cost per Virtual Machine is calculated as the Virtual Infrastructure is built-out. By totaling the costs associated with physical server hardware purchases, virtualization software licensing, Operating System licensing, storage costs, and any other costs associated with standing up a new virtual machine (i.e. maintenance and personnel costs) and dividing by the number of Virtual Machines that are expected to be supported on the Virtual Infrastructure over its lifetime, you can determine an estimated per Virtual Machine cost. This model is simple and easy to calculate but is largely unfair to customers because it doesn’t account for differences between actual resource usage. For example, Business Unit A uses a single-CPU Virtual Machine with 1GB of RAM and Business Unit B uses a four-CPU Virtual Machine with 8GB or RAM—though, both Business Units are paying the same.
In the Complex model, the cost per Virtual Machine is calculated using actual resource usage information of a Virtual Machine. This model is especially complex in a virtual environment because of the many different ways a Virtual Machine can be configured and deployed. Further, because Virtual Machines are often shuffling around physical servers as capacity requirement change, it’s impossible to use a simple ledger-style Chargeback method. To implement a fair and balanced Chargeback model using the Complex model, the use of Chargeback tools (VMware Chargeback and VKernel Chargeback) purpose-built for virtualization becomes a requirement. The Complex model requires the cost per Virtual Machine to be calculated using the Simple model described above as a first step. Actual CPU, Memory and Storage information is then collected as an additional cost and added to the basic per Virtual Machine cost. By using the complex model a fair and accurate per Virtual Machine cost can be associated to an individual Virtual Machine.
Virtual Machines are not “free”. Each Virtual Machine has hardware costs, software costs, infrastructure costs, personnel costs and other hidden costs (HVAC, Electricity, Datacenter footprint) that are all factors to consider when creating a Virtual Machine.
