HP Servers disconnecting

We come across an issue lately, with several types of HP servers that have QLogic/NetXen NC375i networkcards in them. They disconnect, causing a disruption of service. You can imagine that having an NFS mount or iSCSI target with that happening is less than desirable and has caused Windows clusters to fail over and ESX/ESXi hosts to go crazy. This problem is solved by rebooting the host. This issue is very much OS independent!

In windows eventlog you may see things like:
DEVICE: HP NC375i Integrated Quad Port Multifunction Gigabit Server Adapter #4
PROBLEM: Tx path is hung. The device is being reset.

In ESX you see things in /var/log/vmkernel like:
Jul 31 21:02:12 server01 vmkernel: 165:01:40:09.914 cpu19:4295)<5>nx_nic[vmnic8]: Device is DOWN. Fail count[8]
Jul 31 21:02:12 server01 vmkernel: 165:01:40:09.915 cpu19:4295)<3>nx_nic[vmnic8]: Firmware hang detected. Severity code=0 Peg number=2 Error code=1 Return address=0


HP has brought out an advisory saying that indeed there are problems:

Network Adapters and Affected Firmware Versions

Network Adapter	Affected Firmware Versions
CN1000Q Dual Port Converged Network Adapter	EARLIER than firmware version 4.8.22
NC375i Integrated Quad Port Multifunction Gigabit Server Adapter	EARLIER than firmware version 4.0.585
NC375T PCI Express Quad Port Gigabit Server Adapter	EARLIER than firmware version 4.0.585
NC522m Dual Port Flex -10 10GbE Multifunction BL-c Adapter	EARLIER than firmware version 4.0.585
NC522SFP Dual Port 10GbE Server Adapter	EARLIER than firmware version 4.0.585
NC523SFP 10Gb 2-port Server Adapter	EARLIER than firmware version 4.9.81

The NC375i adapter is integrated on the following servers and storage systems:

• ProLiant DL370 G6 Server
• ProLiant DL580 G7 Server
• ProLiant DL585 G7 Server
• ProLiant DL980 G7 Server
• HP Business Data Warehouse Appliance
• StorageWorks D2D4312 Backup System
• StorageWorks D2D4324 Backup System

Servers manufactured after 1 april 2012 are not affected by this, but check the firmware level if you suffer from this issue. An older interface may still have this issue in your newer machine.

How to check the firmware version:

Windows:
Go to the HP network utilities, and click on the network interface you are having issues with, and click Properties. The Information tab will show the Boot Code, which is the firmware version:

Alternatively, you can run the update tool, and it will tell you which version you are currently running as well.

Linux:

Type "modinfo netxen_nic" and look for the firmware line.

[user@server-01 ~]$ modinfo netxen_nic | grep firmware

firmware: phanfw-4.0.579.bin   <--------  version 4.0.579, so needs an update

ESX/ESXi:
VMware have released a KB article to get the firmware and driver version, available here.

Resolution:

The resolution is to update the firmware of the network cards. The advisory lists the latest drivers and firmware. For Windows and Linux, there are proper update tools, but unfortunately for VMware, no firmware update utility is given, and the Linux firmware utility does not work.

On ESX/ESXi you have to make use of a Linux LiveCD and boot from it (ESX-server in Maintenance mode and reboot). In our case we used Novell SLES11 CD (free ISO download, registering necessary) as the Rescue-CD for RHEL5 gave several errors running the firmware update-utility. Perhaps a OpenSUSE, Fedora, Ubuntu or other distro LiveCD can be used as well, but we haven't tested those.

Many thanks go to my colleague Sven for the info :-)

No Remote Desktop client access licenses available

That's what a colleague of mine got when trying to connect to a terminal server. I had seen this message a long time before, and found a solution back then, but every time I get this message, I need to go through my extensive mailbox and search for the answer, so I thought I'd post it here:

Here’s how you fix the error message:

-    Open up regedit on your machine (Start->Run->Type: regedit.exe)
-    Go to HKEY_LOCAL_MACHINE\Software\Microsoft\MSLicensing\Store
-    Click the LICENSE00x folder (probably LICENSE000)
-    Click File->Export, and export the LICENSE00x folder to a registry file
-    Now delete the LICENSE00x folder.
-    Try to log on again.

So far it worked every time for me and my colleagues.

Cloned VM's and WSUS

 When you clone Windows machines, you may run into the issue where WSUS updates are not working. Even though sysprep has been run, the wsus authorization appears to not have been reset with it. I use this script below to reset the Windows Update to a fresh state.

net stop wuauserv
del c:\windows\WindowsUpdate.log
REG DELETE "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate" /v AccountDomainSid /f
REG DELETE "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate" /v PingID /f
REG DELETE "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate" /v SusClientId /f
REG DELETE "HKLM\Software\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update" /v LastWaitTimeout /f
REG DELETE "HKLM\Software\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update" /v DetectionStartTime /f
REG DELETE "HKLM\Software\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update" /v NextDetectionTime /f
REG DELETE "HKLM\Software\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update" /v AUState /f
net start wuauserv
wuauclt /resetauthorization /detectnow
pause

Paste this into a batchfile, run it, and wait a short while, and then the updates should start again. Note that this deletes the WindowsUpdate.log file. If you want to do troubleshooting of previous WSUS errors, either delete that line, or copy the file to a safe place. I delete it so I get a fresh view of what is happening with the Windows Updates.

Doing work on remote Windows servers not in a domain

Having to manage multiple servers from multiple customers that are not in a domain, you sometimes need to get creative when you need to script batchfiles.

Luckily, Microsoft has something that worked with Windows NT4 and still works with Windows 2008R2, provided there are no firewalls blocking you.

From the command prompt of your administrative pc, type:

net use \\hostnametomanage\ipc$ /user:adminuser password

If you get a "Command completed successfully" back, you are in luck. With psexec, available from the excellent sysinternals pstools suite you can now do everything on the remote server, as if it were done locally.

With that one command and psexec you can use batchscripting to do almost anything you want.

Using Perfmon for SQL Performance Tuning

A while ago I found this older but still fantastic blog post about SQL performance tuning:

http://www.brentozar.com/archive/2006/12/dba-101-using-perfmon-for-sql-performance-tuning/

Excellent video in there, well worth watching. Yes, the video is from 2006, but even now, it is just as relevant.

A little after I saw the blog post, I got the chance to put this in practice too; A customer suspected his SQL cluster wasn't performing too well. I added the counters from the video to the perfmon counter log, and kept the perfmon running.

Then our customer came back after a half a year; he found the SQL performance had degraded even more. We had a nice list of performance statistics now that went deeper than just CPU and memory usage. Now, I got the number of User Connections, and saw that it went up 4 times.

The perfmon counters Brent Ozar was talking about:

• These are listed OBJECT first, then COUNTER
• Memory – Available MBytes
• Paging File – % Usage
• Physical Disk – Avg. Disk sec/Read
• Physical Disk – Avg. Disk sec/Write
• Physical Disk – Disk Reads/sec
• Physical Disk – Disk Writes/sec
• Processor – % Processor Time
• SQLServer: Buffer Manager – Buffer cache hit ratio
• SQLServer: Buffer Manager – Page life expectancy
• SQLServer: General Statistics – User Connections
• SQLServer: Memory Manager – Memory Grants Pending
• SQLServer: SQL Statistics – Batch Requests/sec
• SQLServer: SQL Statistics – Compilations/sec
• SQLServer: SQL Statistics – Recompilations/sec
• System – Processor Queue Length

To find out what they all mean, see the video:

http://www.brentozar.com/archive/2006/12/dba-101-using-perfmon-for-sql-performance-tuning/

Removing old hardware in Windows VM's

If you P2V (physical to virtual) a system, or even when you create a VM on your VMWare workstation and move that to an ESX host (V2V, virtual to virtual), old hardware is "left behind" in Windows. The goal of a VM is to make it use as little resources as possible, so it is best you clean that up. Secondly, if the hardware in question is an old network interface, the IP address on that interface still exists, so Windows starts moaning there's a duplicate IP address.

There's a relatively simple way of fixing that: First, open up a command prompt and type

set devmgr_show_nonpresent_devices=1

Next, type: devmgmt.msc

The device manager will start up. In there, go to the menu "View" and click on "Show hidden devices". Now all sorts of previously hidden hardware will show up. Right mouseclick on the old hardware you want to remove (the icons that are of a lighter color than the existing hardware).

Now in Windows 2008/2008R2, if you do the same with a command prompt, you will get nothing. Fix that issue by running the command prompt as an administrator.