Update Manager 4.1 failed to remediate

I came across another nice puzzle after updating one of our customers' vCenter and Update Manager to 4.1U3 (yeah, some customers hold off on updating forever, I know..): When trying to update some ESX hosts attached to the vCenter, I got the following failure:




The host returns esxupdate error codes: 7. Check the Update Manager log files and esxupdate log files for more details

The logfiles contained the following info:

Remediation did not succeed for XXXXesx01: SingleHostRemediate: esxupdate error, version: 1.30, operation: 7: ('http://XXXX:9084/vci/hostupdates/hostupdate/vmw/vibs/cross_oem-vmware-esx-drivers-net-vxge_400.2.0.28.21239-1OEM.vib', '/var/cache/esxupdate/3375545638666279871', '[Errno 14] HTTP Error 404: Not Found') . error 4/11/2013 3:18:13 PM 

This error turned out to be because of a known error. Apparently pre-Update Manager 4.1U2, the webserver serving the patches was case insensitive, and from U2 on, it was made case-sensitive. However, certain patches that were previously downloaded before, were stored case-insensitive, but they were supposed to be case sensitive.

The KB article is here: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2011656

Because redownloading at this moment was not an option, we had to rename the files to be with the proper case sensitivity. The files are fortunately not that many:

From:

• bind-libs-9.3.6-4.p1.el5_5.3.i386.vib 
• bind-libs-9.3.6-4.p1.el5_5.3.x86_64.vib 
• bind-utils-9.3.6-4.p1.el5_5.3.x86_64.vib 
• bind-libs-9.3.6-4.p1.el5_5.3.i386.vib 
• cross_oem-vmware-esx-drivers-net-vxge_400.2.0.28.21239-1oem.vib 
• cross_oem-vmware-esx-drivers-scsi-3w-9xxx_400.2.26.08.036vm40-1oem.vib 
• vmware-esx_swmgmt_provider-4x.1.0.1-1.4.348481.vib

To (note the bold typeface):

• bind-libs-9.3.6-4.P1.el5_5.3.i386.vib 
• bind-libs-9.3.6-4.P1.el5_5.3.x86_64.vib 
• bind-utils-9.3.6-4.P1.el5_5.3.x86_64.vib 
• bind-libs-9.3.6-4.P1.el5_5.3.i386.vib 
• cross_oem-vmware-esx-drivers-net-vxge_400.2.0.28.21239-1OEM.vib 
• cross_oem-vmware-esx-drivers-scsi-3w-9xxx_400.2.26.08.036vm40-1OEM.vib 
• vmware-esx_swMgmt_provider-4x.1.0.1-1.4.348481.vib 

CDP info from vSphere platform

For documentation purposes, I wanted to get an overview of all host connections to which switches they are connected, and which ports exactly. As luck would have it, VMware already has made some scripts to do this.

I adapted it to suit my needs:

$Hosts = Get-VMHost | sort -property Name

foreach ($vhost in $Hosts){

$vmh = Get-VMHost -Name $vhost

If ($vmh.State -ne "Connected") {

  Write-Output "Host $($vmh) state is not connected, skipping."

  }

Else {

  Get-View $vmh.ID | `

  % { $esxname = $_.Name; Get-View $_.ConfigManager.NetworkSystem} | `

  % { foreach ($physnic in $_.NetworkInfo.Pnic) {

    $pnicInfo = $_.QueryNetworkHint($physnic.Device)

    foreach( $hint in $pnicInfo ){

      # Write-Host $esxname $physnic.Device

      if ( $hint.ConnectedSwitchPort ) {

        $hint.ConnectedSwitchPort | select @{n="VMHost";e={$esxname}},@{n="VMNic";e={$physnic.Device}},DevId,PortId

        }

      }

    }

  }

}

}

To run this, simply copy this script to a file, open up powershell, connect to vcenter, and run it with:

.\VMHostCDPInfo.ps1 | Format-Table -AutoSize |Out-File cdpinfo.txt

It ignores all nics where it cannot retrieve any CDP info from, but other than that, you get a nice list, which you can put with your documentation which looks like:

VMHost          VMNic   DevId    PortId
------          -----   -----    ------
server01        vmnic0  switch01 GigabitEthernet1/25
server01        vmnic10 switch02 GigabitEthernet2/0/31

TCPDump of NFS traffic that is using jumbo frames

I had an issue a while ago where I needed to troubleshoot poor NFS performance, so VMWare support asked me to do a tcpdump of the traffic. I was using jumbo frames and the performance was poor after about 5 minutes of not using the NFS.

On the vswitch that has the NFS VMKernel (you ARE using a dedicated vswitch for NFS, right?) create a new service console.

Use an unused IP address from the range that you use on your storage LAN. If you are using a different vlan, don't forget to set that too. You will be using the vswif that is created with this previous action to do the tcpdump. Go to the properties of the vSwitch and set promiscuous mode to "Accept".

Now to do a tcpdump of your traffic.

tcpdump -i vswif2 -w ./dump_1 -s 256 (-s 256 is a buffer of 256MB I believe)

You will get a lovely SMALL file.. Not what you want. This is because vswif2 (the newly created vswif) is not using jumbo frames. To get vswif2 on jumbo frames type the following in the command prompt:

ip link set dev vswif2 mtu 9000
service network restart vswif2

Now do the tcpdump again, and watch your harddrive fill up with data. Stop tcpdump with CTRL-C (and don't wait too long, because it fills up quickly).