Some searching gave the tip to remove it via PowerCLI (Remove-VMHost), but that did not work either:
It turned out the ESXi host could not be removed because it was part of a cluster.
Solution:
Move the ESXi host entry (which is disconnected) to the top of your vCenter tree, out of any cluster. Then re-run the Remove-VMHost command (most likely the Remove function will be visible in the GUI as well, but I just pressed up and enter in my PowerCLI screen, and it started to remove)
