Sunday, February 27, 2011

Setting 2008R2 NTP from the commandline

There I was, happily configuring a new Windows 2008R2 server, when I wanted to set the NTP hosts.

net time /setsntp:"ntpserver1,ntpserver2"

Error! It appears the command has changed since Windows 2008. It turns out to be the following now:

W32tm /config /syncfromflags:manual /manualpeerlist:ntpserver1,ntpserver2

Oh well, gotta remember that one.

Saturday, February 26, 2011

Doing work on remote Windows servers not in a domain

Having to manage multiple servers from multiple customers that are not in a domain, you sometimes need to get creative when you need to script batchfiles.

Luckily, Microsoft has something that worked with Windows NT4 and still works with Windows 2008R2, provided there are no firewalls blocking you.

From the command prompt of your administrative pc, type:

net use \\hostnametomanage\ipc$ /user:adminuser password

If you get a "Command completed successfully" back, you are in luck. With psexec, available from the excellent sysinternals pstools suite you can now do everything on the remote server, as if it were done locally.

With that one command and psexec you can use batchscripting to do almost anything you want.

Friday, February 25, 2011

Using Perfmon for SQL Performance Tuning

A while ago I found this older but still fantastic blog post about SQL performance tuning:

Excellent video in there, well worth watching. Yes, the video is from 2006, but even now, it is just as relevant.

A little after I saw the blog post, I got the chance to put this in practice too; A customer suspected his SQL cluster wasn't performing too well. I added the counters from the video to the perfmon counter log, and kept the perfmon running.

Then our customer came back after a half a year; he found the SQL performance had degraded even more. We had a nice list of performance statistics now that went deeper than just CPU and memory usage. Now, I got the number of User Connections, and saw that it went up 4 times.

The perfmon counters Brent Ozar was talking about:

  • These are listed OBJECT first, then COUNTER
  • Memory – Available MBytes
  • Paging File – % Usage
  • Physical Disk – Avg. Disk sec/Read
  • Physical Disk – Avg. Disk sec/Write
  • Physical Disk – Disk Reads/sec
  • Physical Disk – Disk Writes/sec
  • Processor – % Processor Time
  • SQLServer: Buffer Manager – Buffer cache hit ratio
  • SQLServer: Buffer Manager – Page life expectancy
  • SQLServer: General Statistics – User Connections
  • SQLServer: Memory Manager – Memory Grants Pending
  • SQLServer: SQL Statistics – Batch Requests/sec
  • SQLServer: SQL Statistics – Compilations/sec
  • SQLServer: SQL Statistics – Recompilations/sec
  • System – Processor Queue Length

To find out what they all mean, see the video:

Thursday, February 24, 2011

You shouldn't virtualize *everything*

VMWare says you can virtualize everything, but that doesn't mean you should. One of those things you actually shouldn't (dare I say can't) virtualize, is NTP. If someone asks you to virtualize the NTP server, here are two nice links to give you why that's a bad idea:

9.2.2. Xen, VMware, and Other Virtual Machine Implementations

NTP was not designed to run inside of a virtual machine. It requires a high resolution system clock, with response times to clock interrupts that are serviced with a high level of accuracy. No known virtual machine is capable of meeting these requirements.

Run NTP on the base OS of the machine, and then have your various guest OSes take advantage of the good clock that is created on the system. Even that may not be enough, as there may be additional tools or kernel options that you need to enable so that virtual machine clients can adequately synchronize their virtual clocks to the physical system clock.

Page  18:
Using NTP in Linux and Other Guests
The Network Time Protocol is usable in a virtual machine with proper configuration of the NTP daemon.
The following points are important:

Do not configure the virtual machine to synchronize to its own (virtual) hardware clock, not even as a fallback with a high stratum number. Some sample ntpd.conf files contain a section specifying the local clock as a potential time server, often marked with the comment “undisciplined local clock.” Delete any such server specification from your ntpd.conf file

Tuesday, February 22, 2011

Continued: Storage vMotion via PowerCLI

I spoke before about Storage vMotion via PowerCLI, and today I got to put it into practice. It works, *but* there's a small thing: It doesn't do thin provisioning on the fly, as far as I can tell. So if you had a thick VM, and you wanted to make it thin, normally via the interface you get an option to do thin provisioning during the Storage vMotion. The Move-VM statement has no such feature yet (I think, at least I couldn't find it). Some searching around told me that the awesome LucD posted a function on poshcode to get the thin provisioning into Move-VM (sort of). I haven't tested it out yet, but it seems to be what I needed.

Oh and for my own reference: If I want to move all vm's from "cluster1" to the New-Storage datastore:

Get-Cluster cluster1 | Get-VM | Move-VM -Datastore(Get-Datastore "New-Storage")

Monday, February 21, 2011

Citrix XenApp on VMWare

A few years ago, there was a lot of discussion regarding Citrix XenApp/Terminal Server running on VMWare. Our personal experience was that it wasn't that good, and with anything above a few (4-5) concurrent users performed terrible when doing day to day tasks. We even tried the tweaks that were going around (stuff like only use 1 vcpu on your Citrix server, and various other tweaks going around). Our general feeling was to not do it anymore, and left it at that.

However, time goes on, and technology moves forward. But the articles remained online, and no one really talked about the advancements that have been made. Then, on VMWorld 2009 there was a session about running XenApp using vSphere. If you create a (free) account on, you can watch it for free

Basically, a few best practices are given:
  1. Use the newest CPU's in your hosts (Nehalem architecture or higher). So if you are still using older hosts (like HP G5 series servers) then think again. The newer CPU's alone would give about 30% better performance (according to the video, ymmv)
  2. Use vSphere, mainly because it supports MMU virtualization which gives a good performance boost for Xenapp (but if you are still on ESX3.5, you *really* should be thinking of upgrading anyway, instead of thinking of virtualizing your Xenapp servers)
  3. The usercount on a VM will never be the same as on a physical server. The idea is to be running multiple smaller VM's and thereby getting more users in total per physical box.
  4. Don't use p2v'd systems. It's much better to start with a clean OS, but if you must, remove old hardware, hardware management agents, and unused OS features (wallpaper, menu animations, systemtray animating things such as network indicators and system clock. Of course there are other tweaks you always need to do for any terminal server environment.The sweetspot of the Xenapp VM's is often  2vCPU's and 4GB RAM. More vCPU's will usually give less performance.
  5. Old blogposts are no longer valid, so tweaks like "disable page sharing and memory ballooning" are no longer necessary.
  6. Use realistic tests. It's no good if your environment performs well in synthetic tests if the applications that the users will use are not performing up to par.
The video also has someone from eBay giving a talk about their environment, and they came to 6 to 8 users per VM, each with 2vCPU's and 4GB RAM, running 10 VM's on one DL380G6. Like I said, the video is free to watch, and highly informative.

Monday, February 14, 2011

Removing old hardware in Windows VM's

If you P2V (physical to virtual) a system, or even when you create a VM on your VMWare workstation and move that to an ESX host (V2V, virtual to virtual), old hardware is "left behind" in Windows. The goal of a VM is to make it use as little resources as possible, so it is best you clean that up. Secondly, if the hardware in question is an old network interface, the IP address on that interface still exists, so Windows starts moaning there's a duplicate IP address.

There's a relatively simple way of fixing that: First, open up a command prompt and type

set devmgr_show_nonpresent_devices=1

Next, type: devmgmt.msc

The device manager will start up. In there, go to the menu "View" and click on "Show hidden devices". Now all sorts of previously hidden hardware will show up. Right mouseclick on the old hardware you want to remove (the icons that are of a lighter color than the existing hardware).

Now in Windows 2008/2008R2, if you do the same with a command prompt, you will get nothing. Fix that issue by running the command prompt as an administrator.

Friday, February 11, 2011

More than one RDP session per user on Windows 2008

Just saw an awesome post on a Dutch site that I noticed before, but never really took the time to investigate further:

It appears Windows 2008 (and noticed the same thing with 2008 R2) by default only accepts one RDP session per user. So when administrator logs in, he could potentially kick out another person who uses the administrator account. 

So Rudolf Kleijwegt posted the following command:

reg add "HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server" /v fSingleSessionPerUser /t REG_DWORD /d 0x0 /f

AWESOME tip, and I'll definitely add this to our buildserver.

Tuesday, February 8, 2011

vCenter: Don't forget the SQL Server Agent!

So this is one I learned the hard way: If you use SQL server for your vCenter installation, don't forget to start the SQL Server agent automatically!

vCenter installs rollup jobs which it needs to process the statistics to make it need less granular so it takes up less space.

These are called rollup jobs, and are controlled by the statistics in the vCenter server settings. I hear you ask: So what happens if I don't start SQL Server agent automatically?

Well.. Nothing, at first. But after a week, the statistics don't get rolled up, so the first table in the SQL database that is used for the statistics (VPX_HIST_STAT1) keeps growing and growing, and the data won't be processed into the next table, called VPX_HIST_STAT2, nor the next, etc. There are 4 tables. The last 3 tables will have no data, so when you look at the day, week, month or year statistics, you'll have no statistics. Starting up the SQL Server Agent quickly will give SQL time to roll up the data, but wait too long and you have too much data to process. There is a way to delete the data, described in this article, but you will lose all statistics.

So save yourself some trouble: If you are installing SQL Server for vCenter, don't forget the SQL Server Agent

TCPDump of NFS traffic that is using jumbo frames

I had an issue a while ago where I needed to troubleshoot poor NFS performance, so VMWare support asked me to do a tcpdump of the traffic. I was using jumbo frames and the performance was poor after about 5 minutes of not using the NFS.

On the vswitch that has the NFS VMKernel (you ARE using a dedicated vswitch for NFS, right?) create a new service console.

Use an unused IP address from the range that you use on your storage LAN. If you are using a different vlan, don't forget to set that too. You will be using the vswif that is created with this previous action to do the tcpdump. Go to the properties of the vSwitch and set promiscuous mode to "Accept".

Now to do a tcpdump of your traffic.

tcpdump -i vswif2 -w ./dump_1 -s 256 (-s 256 is a buffer of 256MB I believe)

You will get a lovely SMALL file.. Not what you want. This is because vswif2 (the newly created vswif) is not using jumbo frames. To get vswif2 on jumbo frames type the following in the command prompt:

ip link set dev vswif2 mtu 9000
service network restart vswif2

Now do the tcpdump again, and watch your harddrive fill up with data. Stop tcpdump with CTRL-C (and don't wait too long, because it fills up quickly).

Friday, February 4, 2011

VMWare data traffic with jumbo frames

Set the jumbo frames on vSwitch1 (which I am using for NFS). Log on to the console of the ESX host, and type:

esxcfg-vswitch -m 9000 vSwitch1

# remove the old vmkernel for storage which is on MTU 1500

esxcfg-vmknic -d "VMkernel - Storage"

# add the vmkernel back for storage again with MTU  9000 (vmkernel has ip

esxcfg-vmknic -a -i -n -m 9000 "VMkernel - Storage"

Now the vSwitch1 and NFS vmkernel are using MTU 9000, and if your physical switch and SAN are configured for jumbo frames too, you should have a speed increase.

Thursday, February 3, 2011

Pushing Host Profiles via PowerCLI.

Nice.. I had to push a host profile to a whole number of hosts, but they were all running a lot of VM's. Anyone who's seen vSphere knows that via the interface, you need to set a server in maintenance mode, apply the host profile, then take it out of maintenance mode. This is fine for one or two machines, but by the time you get to the 5th or 6th server, you get a bit bored, especially when you know you've got 20 to do.. So in comes Powershell (I should say PowerCLI) again, to save me some time.. I thought.

First things first: Figure out how to get a host profile:

$MyHostProfile = Get-VMHostProfile -Name "coolprofile"

Fail. Fail? Yes, fail. Somehow I run into a wall, with PowerCLI telling me that my profile can't be found. Some googling tells me that there's a bug in PowerCLI, but there's a workaround:

Get-VMHostProfile -Entity *

Woohoo! It shows me a hostprofile that was already pushed to the client before. So my statement would now become:

$MyHostProfile = Get-VMHostProfile -Entity * 

Note that this workaround would only work with one host profile. I'll figure out how to get a specific hostprofile implemented some day, but I've got one, so I got lucky (this time).

Now to apply a host profile to an esxhost:

Apply-VMHostProfile -Entity $esxhost -Profile $MyHostProfile -Confirm:$false

Damn, that was easy. Especially since the first part was so difficult..OK, now now to get a host in maintenance mode: 

Set-VMHost -VMHost $esxhost -State maintenance 

Great, that works! Now how do I get it out? 

Set-VMHost -VMHost $esxhost -State connected 

Cool, works too! Now I have to put the whole thing together:

$hosts = "esx1","esx2","esx5","esx6"
$MyHostProfile = Get-VMHostProfile -Entity *
foreach ($esxhost in $hosts) {
Set-VMHost -VMHost $esxhost -State maintenance
Apply-VMHostProfile -Entity $esxhost -Profile $MyHostProfile -Confirm:$false
Set-VMHost -VMHost $esxhost -State connected

As you can see, I used an array of hosts in this case (the first line of code. Didn't want all hosts in my case) but I could have changed that to all hosts by just doing a get-vmhost.

Running this now puts each host in maintenance mode, applies the host profile, and takes it out again.. Now imagine that for 20, 30 or even 100 hosts...... Yes, I *like* PowerCLI...

Storage VMotion via PowerCLI

I keep running into occasions where I need to storage vmotion a lot of VM's to different locations. I looked at a solution some time ago, but that was an old version of PowerCLI. Turns out nowadays it's really easy:

get-vm "MyVM"| move-vm -datastore(Get-Datastore "New-Storage")

This simply moves the MyVM virtual machine to the New-Storage datastore.. Remove the "MyVM", and suddenly ALL vm's move to the New-Storage datastore: SWEET. Gotta love PowerCLI, and especially since I've got a job coming up where I need to be moving a whole bunch of vm's to new storage....

Migrating/upgrading a 32bit vCenter 4.0 installation to a 64bit 4.1 installation

With the coming of vSphere 4.1, vCenter now has a prerequisite of needing a 64bit Windows operating system. Many customers that upgraded from ESX3.5 to vSphere were using a 32bit Windows installation, but also many vSphere installations were built using 32bit Windows. vCenter 4.0 even needed 32bit DSN's so it appeared it was easier to just install vCenter using a 32bit installation. To make things worse, many vCenter implementations have SQL Server installed on the same server as vCenter itself. This article will be about the upgrade process of upgrading windows 2003 32bit with SQL 2005 32 bit to a 64bit environment, and migrating the data.

During testing, this upgrade was quite difficult because of little snags, so it is very wise to do a dry run of this using a P2V’d system and testing the upgrade thoroughly before you do this on a live system. You wouldn't want to reinstall your OS and find out your migration data is useless.

-          Windows 2003/8/8R2 x64 Standard
-          SQL Server 2005/8 x64 Standard (2008R2 is NOT supported yet)
-          VMWare vSphere 4.1 vCenter installation media
-          External source, such as an external harddisk or fileshare.

Basic steps
  1. Make sure all data is safely backed up, and you have all the information off of your 32bit installation
  2. Stop virtual center service, update manager service
  3. Run the backup.bat from VMWare's datamigration tool
  4. Copy the datamigration complete with data to an external source
  5. Stop SQL Server service and SQL Server agent, copy the ESX and Update Manager database to the external source
  6. Wipe server and install server with 64 bit OS, using the same name and IP as the previous one
  7. Install Microsoft SQL Server
  8. Copy vCenter and Updatemanager database to the newly installed server
  9. Set permissions on the databases, and on msdb
  10. Set compatibility mode of the db to SQL 2005 if your SQL Server came from SQL 2000.
  11. Create the ODBC links
  12. Copy the datamigration directory back to the server
  13. Make sure you have the vCenter installation media on dvd or on the server in a directory as well
  14. Run install.bat from the datamigration directory
  15. Answer the installation questions from the installer
Detailed Steps

So you want to migrate a 32bit system to a 64bit system. This means you will be doing a complete re-installation of the operating system. I will assume only the database (in my tests SQL2005) is installed and standard users have been created, and no extra software has been installed.

Once the first VMWare services are stopped, management through vCenter will be impossible, but the VM’s will keep on running. Services like HA and DRS will be unavailable then, but VM’s should not notice anything of this. 

Collect information and backup the system
Note the following:
-    IP address(es)/subnetmask(s)
-    Hostname
-    Workgroup (if the machine is part of a domain, take note of that as well)
-    Routes
-    Host file (c:\windows\system32\drivers\etc)
-    Usernames (and passwords, if you have them)
-    Possible created groups

For SQL look at the following things:
-    Which databases are configured
-    Where are the databases stored
-    Which Service Pack is the database currently
-    Any extra settings made in SQL, such as maintenance and backup plans.
-    The ODBC DSN’s used to connect the database to the Virtual Center.

Of course, if you have installed extra software, make sure you have copied any settings and data from that application. Finally, make sure you have a working backup, or pull one of the disks of the raid 1 set. This will ensure you can easily go back to the 32bit environment.

Stop vCenter services
In computer management, stop the following services
-    VMware VirtualCenter Server
-    VMware VirtualCenter Management Webservices (will be stopped along with the first service)
-    VMware Update Manager Service

Fortunately, VMWare has made a datamigration tool to help with the migration from 32bit to 64 bit. On the installation media, there is a directory called datamigration. In that directory is a zipfile. Extract that zipfile to your harddisk, or to the external harddisk, as this extracted directory will also hold the data from the migration. I have tried to run the tool from a network share, but it wouldn't work properly, so my advice is to not do that.

Open a command prompt, change to the datamigration directory and type “backup.bat”. The script will prompt you if the data needs to be backed up. Type Y and press enter to continue.
New files and directories will be created i
n the datamigration directory. The “data” directory will hold the configuration data from vCenter, Update Manager and vCenter Orchestrator (directories will be created vc, vum and vco). There will also be a "log" directory to show the results of the migration.

When the backup is done, check the vc and vum directories for data (and if vco has been used, check if there is data there too). The vc directory should have a vc_ssl with keys in them, and also a vc_data file, which is kind of small (in my test environment of 4 servers, it was 24kb).  The vum will have data there too, as well as update files downloaded from the VMWare site. This will be quite big (several GB).

Copy the data
Copy the entire datamigration directory to the external source (if you haven’t extracted it to the external source beforehand).

Stop SQL services and copy the databases
Stop the following services for SQL:
-    SQL Server
-    SQL Server Agent
You can now copy the MDF and LDF files belonging to vCenter and the Update Manager from the server to the external source. To speed up the copy, you could shrink the database before you copy the files. My test environment database shrank from 2GB to 200MB.

“Nuke and repave”
Reinstall the server with the 64bit operating system, using the same name and same IP information.  There will be issues with vCenter if other IP information is used. Patch the system as you would normally patch a system. Create routes, possible hosts file entries, users and groups. Re-install a SQL 64bit edition, and install the same service pack as the 32bit edition. You can go from SQL 2005 to SQL 2008 without needing extra configuration. I like to configure SQL so that it installs the databases to another disk from the SQL installation.

Copy databases back and configure SQL
Copy the MDF and LDF to the newly installed server, in the directory where the other databases are stored.  In the SQL Server Management Studio, attach the databases.

Make sure vCenter user has been assigned “db_owner” for both the vCenter and Update Manager database, as well as the MSDB database. This is necessary to be able to upgrade the database during the vCenter installation.

Another important thing is that the compatibility level be set to SQL 2005 (90). If the SQL instance you came from used to be SQL 2000 (e.g. when you have an old vCenter that has been upgraded from VI3) it is likely that the compatibility level is set to SQL 2000. Not setting this properly will make the upgrade fail!  

Configure other settings as needed, such as re-implement maintenance plans, and set the recovery model to Simple again. Do the same for the Update manager database if you use that.

Recreate ODBC connections
To recreate the database connections you will need DSN’s. Go to the Data Sources (ODBC) tool in Control Panel -> Administrative Tools and add a system DSN for the vCenter database and Update Manager. If you plan to migrate update manager as well, you need to use a 32 bit DSN for that. Run c:\windows\syswow64\odbcad32.exe to get the 32 bit version, and add a system DSN there.

With me so far? Good, then the fun part can start: Getting vCenter back up and running.

Installation/Upgrade of vCenter server 4.1
Copy the datamigration directory back to the server, and for speed sake, copy/extract the vCenter installation directory to the local harddrive. In any case you need to have this available when the datamigration installer is started.
Open a command prompt and change to the datamigration directory. If you are using Windows 2008, open a command prompt by "run as administrator". Type “install” to start. The installer script will check to see if the vCenter data is available, and tell you if all data is available or not. If it is available, it will use these settings during installation. It will ask for the directory of the installation media (e.g. D:\vim410).
After this it immediately checks for the Upgrade Manager upgrade data and ask for the installation media again. This is the same directory as before.

As a sidenote: I have had three or four attempts of testing this in a test environment, and a the first times the installer said there was no upgrade data available. I needed to run the backup script on the source again. That is why it is important to check for the existence of files during the backup and to test this upgrade before you try this on a live environment.

After the question for the Upgrade Manager has started, the installer for vCenter 4.1 will start.  Select the vCenter DSN and answer “yes” to the question if the database should be upgraded. At one moment, the question will come if the host agents can be upgraded automatically. I answer no.

In vCenter the ESX hosts will be in a disconnected state, but the VM information will be seen, and you should be able to see the performance data of the VM’s. I select no, because you can still fall back to the old installation until the complete installation finishes (remember that disk you took from the raid 1 set, or the backup of the previous installation? The ESX hosts can then still be managed by vCenter 4).

After the installation, the installation for Upgrade Manager will immediately start. Follow the instructions as they appear on screen.
Install the updated Virtual Infrastructure Client from the vpx directory in the installation media (vi-client.exe) and then connect to the vCenter. The hosts are in a disconnected state, but you can simply rightclick the hosts, and click "Connect". The agent will update, and HA will be re-enabled.

This should be all you need to complete a successful migration.

Note to self: Make some screenshots next time...