Waiting for a VM to come back

In my previous post, I wrote about upgrading vCPU's, and that I want to do several other things as well. If you upgrade a vCPU on a VM, it could very well be that the Windows guest wants a second reboot. I've found that simply rebooting and waiting for a while is inefficient. So I usually check if the VMware tools are back online.

I've made that into a function, to exactly that:

function WaitforVM($VM)

{

    while (((get-vm $VM ).ExtensionData.Guest.ToolsRunningStatus ) -ne "guestToolsRunning" )

        {

            Write-Host "Waiting for machine to come back....." -ForegroundColor Yellow

            Sleep 5

        }

}

Now all you have to do, is to start the VM after the CPU addition, wait, then restart and wait again:

Start-VM -VM $VM -Confirm:$False

WaitforVM -VM $VM

Restart-VMGuest -VM $VM -Confirm:$False

WaitforVM -VM $VM

Nice, right? Obviously this function can be used for other things, like being used in scripts when you create a new VM from template and boot it up to see if that VM is live already.

Automatically upgrade vCPU's

I am lazy. Well, not lazy, I just don't like working at ungodly hours. Here's the story: I got this request to upgrade 2 customer VM's from 3 vCPU's to 4 vcpu's. No biggy, I thought. But he wanted it done after 1 AM.. Wait, that's not during office hours!

OK, I could fix that with my trusty Powershell toolkit, and a scheduled task. But then I talked to the customer who went like: "Sure that's fine with me to script it, but make sure the sessions are drained from the webserver".. Hmm, F5 loadbalancers. I've not worked too much with those before, really. I get the general ideas within loadbalancing, but let's try and script against it.

So I had a number of things I wanted to happen:

• Log in to the F5 loadbalancer, preferably not with a script filled with credentials hardcoded into it. 
• Wait an X amount of time for connections to drop from the specific pool
• Shut down the machine, and wait for it to be properly turned off.
• Set the number of CPU's and memory. This had an extra challenge, namely that the person who built the VM put in 1 vCPU with 3 cores, instead of 3 vCPU's.
• Send an email with the results, so I can see what happened from my phone.

I'll spend a few posts on those different things.

Firstly the vCPU upgrade, which is the basis of this entire story:

Normally, you can turn off a VM, upgrade vCPU's, then start a VM again, through this simple set of commands:

$VM=Get-VM -Name 'WEB01'

Stop-VMGuest –VM $VM –Confirm:$False

do {

        $status = (get-VM $VM).PowerState

    }until($status -eq "PoweredOff")

$VM | Set-VM -NumpCPU 4 –Confirm:$False | Start-VM

However, since someone used 1 vCPU with multiple cores instead of multiple vCPU sockets, things work a little differently. If you would try this command, the VM would be shut down, but no upgrade would happen, and start back up with the same amount as before.

Fortunately there's another trick for that:

$VMSpec = New-Object -Type VMware.Vim.VirtualMachineConfigSpec -Property @{"NumCoresPerSocket" = 4;"numCPUs" = 4}

$VM.ExtensionData.ReconfigVM_Task($VMSpec)

(That first line shouldn't be cut off like that, should be 1 line, but alas: blog template gets in the way)

Now the VM gets upgraded to 4 vCPU's, although you would think it'd be 4x4 cpu's looking at the syntax.

So now the final complete code would be:

Add-PSsnapin VMware.VimAutomation.Core

Connect-VIServer myVcenter -ErrorAction Stop

$VM=Get-VM -Name 'WEB01'

Stop-VMGuest –VM $VM –Confirm:$False

do {

        $status = (get-VM $VM).PowerState

    }until($status -eq "PoweredOff")

$VMSpec = New-Object -Type VMware.Vim.VirtualMachineConfigSpec -Property @{"NumCoresPerSocket" = 4;"numCPUs" = 4}

Start-VM $VM

Yeay, success! Now on to the next bit...

Selecting and removing old snapshots

Me: "Say Customer X, I see you still have some snapshots here and there. Can I remove them? It's not wise to keep them for a long time."
Customer X: "Oh yeah, I forgot about those. Well: those updates we did worked out fine, so anything from before 11-07-2016 can be deleted"

Get-VM | Get-Snapshot | Where {$_.Created -lt (Get-Date 11-07-2016) | Remove-Snapshot -confirm:$false

Me: "OK, done!"

Another possibility: "Can you give me a list of snapshots that are older than a week?"

Get-VM | Get-Snapshot | Where {$_.Created -lt ((Get-Date).AddDays(-7))} | Select VM, Name

Did I say how much I love Powershell/PowerCLI? ;-)

Surpress SSH and shell warnings on ESXi

Small oneliner to surpress the warning you get when you enable SSH:



Like so:

Get-VMHost | Set-VmHostAdvancedConfiguration -Name UserVars.SuppressShellWarning -Value 1

Rename interfaces based on IP range with Invoke-VMScript

Invoke-VMScript is a lifesaver if you need to do things inside a VM, and you don't want to spend hours logging on to each machine.

Here I needed to rename multiple network interface labels based on IP range:

$renamescript = @'

$adapter1 = Get-NetAdapter | Get-NetIPAddress |where IPaddress -Like "10.168.73.*"| Select interfacealias

$adapter2 = Get-NetAdapter | Get-NetIPAddress |where IPaddress -Like "192.168.72.*"| Select interfacealias

'@

Rename-NetAdapter -Name $adapter1.interfacealias -NewName "Frontend Network"

Rename-NetAdapter -Name $adapter2.interfacealias -NewName "Backend Network"

For each machine you can then do:

Invoke-VMScript -VM $hostname -GuestUser administrator -GuestPassword P4ssw0rd -ScriptText $renamescript

Combine this with the CSV file post I made before, and you can start your script, get a cup of whatever, and have Powershell do the hard work for you.

Getting a list of vm's with OS'es

I wanted to do a Get-VM|select name, GuestOSFullname, but that doesn't work. So some googling gave me this little gem on the vmware forum:

New-VIProperty -Name GuestFullName -ObjectType VirtualMachine -ValueFromExtensionProperty 'Guest.GuestFullName' -Force

Now I can do a Get-VM | Select Name, GuestFullName and I get an overview of which machines have which OS.

I'm sure there are other ways of getting this too, but this was what I was looking for.

P.S., if I do Get-VM | Select Name,Guest , I do get the OS, but it is prepended with the hostname.

Add PowerCLI modules to Powershell console

I've recently become a fan of the Powershell ISE, especially from V3 on. To load VMware PowerCLI module and to have VMware cmdlets available in powershell console:

Add-PSSnapin VMware.VimAutomation.Core

The same can be done with the update manager:

Add-PSSnapin VMware.VumAutomation

HP Firmware versions with PowerCLI

For documentation purposes  I needed BIOS and ILO versions of the vSphere environment.

I got the script from http://vnugglets.com/2011/11/get-hp-firmware-info-for-hp-vmware.html and combined some updates in the comments in the script so that it also shows which NetXen firmware and NetXen drivers are on the system. I adapted the script to give the info for all machines in the cluster, not just a single cluster.

Foreach ($strHostsClusterName in Get-Cluster){

Get-View -ViewType HostSystem -Property Name, Runtime.HealthSystemRuntime.SystemHealthInfo.NumericSensorInfo -SearchRoot (Get-View -ViewType ClusterComputeResource -Property Name -Filter @{"Name" = "^$([RegEx]::escape($strHostsClusterName))$"}).MoRef | %{

    $arrNumericSensorInfo = @($_.Runtime.HealthSystemRuntime.SystemHealthInfo.NumericSensorInfo)

    # HostNumericSensorInfo for BIOS, iLO, array controller

    $nsiBIOS = $arrNumericSensorInfo | ? {$_.Name -like "*System BIOS*"}

    $nsiArrayCtrlr = $arrNumericSensorInfo | ? {$_.Name -like "HP Smart Array Controller*"}

    $nsiILO = $arrNumericSensorInfo | ? {$_.Name -like "Hewlett-Packard BMC Firmware*"}

    $nsiNXdev = $arrNumericSensorInfo | ? {$_.Name -like "nx_nic device*"}

    $nsiNXdrv = $arrNumericSensorInfo | ? {$_.Name -like "nx_nic driver*"}

    # assume all at same level and take first or set as n/a

    if ( $nsiNXdev.Count -gt 0 ) {

       $nsiNXdevice = $nsiNXdev[0].Name

    } else {

       $nsiNXdevice = "n/a"

    }

    if ( $nsiNXdrv.Count -gt 0 ) {

       $nsiNXdriver = $nsiNXdrv[0].Name

    } else {

    $nsiNXdriver = "n/a"

    }

New-Object PSObject -Property @{

VMHost = $_.Name

"SystemBIOS" = $nsiBIOS.name

"HPSmartArray" = $nsiArrayCtrlr.Name

"iLOFirmware" = $nsiILO.Name

"nx_nic device" = $nsiNXdevice

"nx_nic driver" = $nsiNXdriver

    } ## end new-object

} ## end Foreach-Object

}

If I feel like it, I'll someday add LSI Logic cards and such as well. For now, it got me the info I needed, without logging on to 20 servers ;-)

Run the script from PowerCLI, by first connecting to the vCenter (Connect-Viserver <vcenter>) and then ./fwscript.ps1 | Export-CSV d:\fwlist.csv

CDP info from vSphere platform

For documentation purposes, I wanted to get an overview of all host connections to which switches they are connected, and which ports exactly. As luck would have it, VMware already has made some scripts to do this.

I adapted it to suit my needs:

$Hosts = Get-VMHost | sort -property Name

foreach ($vhost in $Hosts){

$vmh = Get-VMHost -Name $vhost

If ($vmh.State -ne "Connected") {

  Write-Output "Host $($vmh) state is not connected, skipping."

  }

Else {

  Get-View $vmh.ID | `

  % { $esxname = $_.Name; Get-View $_.ConfigManager.NetworkSystem} | `

  % { foreach ($physnic in $_.NetworkInfo.Pnic) {

    $pnicInfo = $_.QueryNetworkHint($physnic.Device)

    foreach( $hint in $pnicInfo ){

      # Write-Host $esxname $physnic.Device

      if ( $hint.ConnectedSwitchPort ) {

        $hint.ConnectedSwitchPort | select @{n="VMHost";e={$esxname}},@{n="VMNic";e={$physnic.Device}},DevId,PortId

        }

      }

    }

  }

}

}

To run this, simply copy this script to a file, open up powershell, connect to vcenter, and run it with:

.\VMHostCDPInfo.ps1 | Format-Table -AutoSize |Out-File cdpinfo.txt

It ignores all nics where it cannot retrieve any CDP info from, but other than that, you get a nice list, which you can put with your documentation which looks like:

VMHost          VMNic   DevId    PortId
------          -----   -----    ------
server01        vmnic0  switch01 GigabitEthernet1/25
server01        vmnic10 switch02 GigabitEthernet2/0/31

Unable to remove ESXi host from vCenter

I needed to remove an ESXi host from vCenter that I wanted to re-use, but I was unable to. After disconnecting the server, the "Remove" button was greyed out:

 Some searching gave the tip to remove it via PowerCLI (Remove-VMHost), but that did not work either:

It turned out the ESXi host could not be removed because it was part of a cluster.

Solution:

Move the ESXi host entry (which is disconnected) to the top of your vCenter tree, out of any cluster. Then re-run the Remove-VMHost command (most likely the Remove function will be visible in the GUI as well, but I just pressed up and enter in my PowerCLI screen, and it started to remove)

Getting an overview of patches for your ESX hosts

A customer asked for an overview of which patches exactly were needed for his ESX hosts. He wanted to review the patches before they were applied with Update Manager. Unfortunately, you can't get the list from the gui in a nice way, so some PowerCLI goodness was needed.

First off, you need to have the Update Manager cmdlets installed to be able to use the Get-Compliance cmdlet in the code.

The following script will go over each host, and output the severity, patch ID, release date, additional info (a link to the kb article) and a short update to what the patch is for.

It will look something like this:

HostSecurity,ESX410-201204402-SG,"4/26/2012 10:00:00 AM","For more information, see http://kb.vmware.com/kb/2014988.","Updates libxml2"

Now for the code itself:

ForEach ($HostToCheck in Get-VMHost){

$Details = Get-Compliance $HostToCheck -Detailed| Select -ExpandProperty NotCompliantPatches| Select @{N="Hostname";E={$HostToCheck}}, Severity, IdByVendor, ReleaseDate, Description, Name

$ComplianceResult += $Details

}

$ComplianceResult | Export-CSV -Path c:\NeededPatches.CSV -NoType

Speeding up PowerCLI

As you may have seen, I like PowerCLI. Only thing is, it's not the fastest to start up, and running the first cmdlet takes some time. After that, it goes fine.

The first tip from the VMware PowerCLI Blog is to go to:

Windows > Control Panel > Internet Options > Advanced > Security > Check for publisher's certificate revocation and uncheck that box (also helps the SQL Server Management Studio start up faster if it is not connected to the internet).

Unfortunately, this did not help me so much with running the first cmdlet. So I was mighty glad that I came across this article from vNugglets to speed up PowerCLI 5. This led to another VMware PowerCLI Blog article with all the steps for earlier versions.

Basically, the issue is that due to the fact that the .Net framework compiles the underlying code on first use, so you can precompile things to speed up first run:

Open up an administrative command prompt, and run the following commands:

C:\Windows\Microsoft.NET\Framework\v2.0.50727\ngen.exe install "VimService41.XmlSerializers, Version=4.1.0.0, Culture=neutral, PublicKeyToken=10980b081e887e9f"

C:\Windows\Microsoft.NET\Framework\v2.0.50727\ngen.exe install "VimService40.XmlSerializers, Version=4.0.0.0, Culture=neutral, PublicKeyToken=10980b081e887e9f"

C:\Windows\Microsoft.NET\Framework\v2.0.50727\ngen.exe install "VimService25.XmlSerializers, Version=2.5.0.0, Culture=neutral, PublicKeyToken=10980b081e887e9f"

If you are running on 64-bit OS, you need to run the following as well:

C:\Windows\Microsoft.NET\Framework64\v2.0.50727\ngen.exe install "VimService41.XmlSerializers, Version=4.1.0.0, Culture=neutral, PublicKeyToken=10980b081e887e9f"

C:\Windows\Microsoft.NET\Framework64\v2.0.50727\ngen.exe install "VimService40.XmlSerializers, Version=4.0.0.0, Culture=neutral, PublicKeyToken=10980b081e887e9f"

C:\Windows\Microsoft.NET\Framework64\v2.0.50727\ngen.exe install "VimService25.XmlSerializers, Version=2.5.0.0, Culture=neutral, PublicKeyToken=10980b081e887e9f"

This needs to be done once, and works for all users and applications.

I did a test on one setup, using the test  on the PowerCLI blog which is:

Measure-Command { Get-VMHost } | fl TotalSeconds

This test went from 24.4 seconds to 4.6 seconds! More than 5 times as fast!! Again, thanks go to vNugglets!

How much space are the VM's using?

Quick script to give me an overview of which VM has how much diskspace in use, and also how much is free.

Get-VM -name MYVM-* | Where { $_.PowerState -eq "PoweredOn" } |

Get-VMGuest |

Select VmName -ExpandProperty Disks | Select VmName, Path, Capacity, @{ N="PercFree"; E={ [math]::Round( ( 100 * ( $_.FreeSpace / $_.Capacity ) ),0 ) } }

| Out-File c:\temp\myvmspace.txt

Note that this is of only the VM's that start with "MYVM-". Change this according to whatever you want ( or remove the -name to just get all vm's, sizes and free space).

List the portgroups for the VM's

As part of a makeshift DR solution, one of my requirements is to have a list of vm's that shows which networkcard is in which portgroup. Powershell to the rescue!

Get-VM | Get-NetworkAdapter | Select-Object @{N="VM";E={$_.Parent.Name}},@{N="NIC";E={$_.Name}},@{N="Network";E={$_.NetworkName}}| Export-Csv vms_in_networks.csv

Hey presto, you have a list that shows which nic goes where.

Powershell selection stuff..

Something to remember for myself about selecting:


get-vm | gm as an example, shows all methods and properties you can do with get-vm.

get-vm | where {$_.PowerState -eq "PoweredOn" -and $_.Name -ne "PietjePuk" }

Shows all VM's that are powered on but not that one VM that is called "PietjePuk".
$_.Powerstate is one of the properties of the get-vm statement.

This is how you can select VM's based on different properties.

So get-vm | where {$_.MemoryMB -eq "4096"} shows you all vm's that have 4GB of memory.

-eq = equals
-ne = does not equal
-and = a secondary variable to select.

Updating VMWare Tools on all powered on Windows VM's

Quick PowerCLI line to update the VMWare Tools all Windows VM's that are powered on without rebooting them:

get-vm | where {$_.PowerState -eq "PoweredOn" -and $_.Guest.OSFullName -match "Microsoft Windows*" } | Update-Tools -NoReboot

Oh how I love PowerCLI...

Continued: Storage vMotion via PowerCLI

I spoke before about Storage vMotion via PowerCLI, and today I got to put it into practice. It works, *but* there's a small thing: It doesn't do thin provisioning on the fly, as far as I can tell. So if you had a thick VM, and you wanted to make it thin, normally via the interface you get an option to do thin provisioning during the Storage vMotion. The Move-VM statement has no such feature yet (I think, at least I couldn't find it). Some searching around told me that the awesome LucD posted a function on poshcode to get the thin provisioning into Move-VM (sort of). I haven't tested it out yet, but it seems to be what I needed.


Oh and for my own reference: If I want to move all vm's from "cluster1" to the New-Storage datastore:

Get-Cluster cluster1 | Get-VM | Move-VM -Datastore(Get-Datastore "New-Storage")

Pushing Host Profiles via PowerCLI.

Nice.. I had to push a host profile to a whole number of hosts, but they were all running a lot of VM's. Anyone who's seen vSphere knows that via the interface, you need to set a server in maintenance mode, apply the host profile, then take it out of maintenance mode. This is fine for one or two machines, but by the time you get to the 5th or 6th server, you get a bit bored, especially when you know you've got 20 to do.. So in comes Powershell (I should say PowerCLI) again, to save me some time.. I thought.

First things first: Figure out how to get a host profile:

$MyHostProfile = Get-VMHostProfile -Name "coolprofile"

Fail. Fail? Yes, fail. Somehow I run into a wall, with PowerCLI telling me that my profile can't be found. Some googling tells me that there's a bug in PowerCLI, but there's a workaround:

Get-VMHostProfile -Entity *

Woohoo! It shows me a hostprofile that was already pushed to the client before. So my statement would now become:

$MyHostProfile = Get-VMHostProfile -Entity * 

Note that this workaround would only work with one host profile. I'll figure out how to get a specific hostprofile implemented some day, but I've got one, so I got lucky (this time).

Now to apply a host profile to an esxhost:

Apply-VMHostProfile -Entity $esxhost -Profile $MyHostProfile -Confirm:$false

Damn, that was easy. Especially since the first part was so difficult..OK, now now to get a host in maintenance mode: 

Set-VMHost -VMHost $esxhost -State maintenance 

Great, that works! Now how do I get it out? 

Set-VMHost -VMHost $esxhost -State connected 


Cool, works too! Now I have to put the whole thing together:

$hosts = "esx1","esx2","esx5","esx6"
$MyHostProfile = Get-VMHostProfile -Entity *
foreach ($esxhost in $hosts) {
Set-VMHost -VMHost $esxhost -State maintenance
Apply-VMHostProfile -Entity $esxhost -Profile $MyHostProfile -Confirm:$false
Set-VMHost -VMHost $esxhost -State connected
}


As you can see, I used an array of hosts in this case (the first line of code. Didn't want all hosts in my case) but I could have changed that to all hosts by just doing a get-vmhost.

Running this now puts each host in maintenance mode, applies the host profile, and takes it out again.. Now imagine that for 20, 30 or even 100 hosts...... Yes, I *like* PowerCLI...

Storage VMotion via PowerCLI

I keep running into occasions where I need to storage vmotion a lot of VM's to different locations. I looked at a solution some time ago, but that was an old version of PowerCLI. Turns out nowadays it's really easy:

get-vm "MyVM"| move-vm -datastore(Get-Datastore "New-Storage")

This simply moves the MyVM virtual machine to the New-Storage datastore.. Remove the "MyVM", and suddenly ALL vm's move to the New-Storage datastore: SWEET. Gotta love PowerCLI, and especially since I've got a job coming up where I need to be moving a whole bunch of vm's to new storage....