I ran into a weird problem a few nights ago during a change that took a while to resolve so I thought it was worth posting about.
I was tasked with changing the IP address of a few ESXi servers as they were migrated into a new network segment. Simple enough.
I made the necessary changes on the ESXi hosts, updated DNS, and then removed them from the inventory of vCenter (wasn’t fussed about losing any historical data). I waited a short while and then attempted to add them back in using the appropriate FQDN. Didn’t work. Bugger.
vCenter was complaining about being unable to connect to the host. I was able to add them back in using the IP address though. So I can’t connect with the FQDN or DNS host name but IP address works OK. Hmmm sounds like a DNS problem…..
To cut a long story short, after some Google Fu I discovered that the vCSA has an internal DNS service (running dnsmasq) that seems to aggressively cache lookups far beyond the TTL of the A record. My vCSA was trying to connect to the ESXi host with the old IP address.
To fix it up I had to SSH onto the vCSA as root and run the following to clear the DNS cache:
systemctl restart dnsmasq
This didn’t have any ill effects on the infrastructure so safe to do anytime.
You can check that it has restarted successfully by running the following:
systemctl status dnsmasq
and then a ping to the host will let you see if it has updated.