Recently I ran into a problem on an LPAR where apparently nothing was changed. The problem as always is, how to determine if nothing was changed. After all an IPL of an LPAR is change.

Let me tell you what happened. For a customer running an independent Auxiliary Storage Pool (iASP) in a Cluster environment using a SAN, the firmware of his Flexible Service Processor (FSP) needed to be upgraded.

As you may know this is disruptive and all LPARs, including the VIOS LPARs all need to shut down. After the FSP firmware upgrade was completed, all LPARs were restarted and everything looked fine.

To test as thoroughly as possible a switch was made moving from LPAR ACCCBU to the ACC LPAR.

If you are not familiar with an iASP in a Cluster environment. This concept is in fact very simple. As with a windows server, there is a drive, a C drive, containing the Operating system. For the data and applications there is a D drive.

For IBM i we use the term *SYSBAS for the operating system and *IASP for the data and applications. When a switch is made, IBM i will connect to the SAN and tells it that it wants to use the disks on the other SAN. What needs to be done in detail, when doing this, is beyond the scope of the article.

If you want to learn more about this concept, please have a look here: IBM i Cluster technology.

Anyway, when initiating the switch an error occurred, the message was:

ssh: connect to host 10.200.5.75 port 22: Connection refused.

Finding the cause of a problem, is always difficult. So, the best approach is to exclude possible causes. As the two Flash Systems in use by IBM i where in a different VLAN, checking the firewall was the most obvious choice. When doing my homework, I discovered that the IBM i itself was the cause.

In a cluster environment, it is common practice to define an extra IP address on every Node in the Cluster, which are part of the Cluster Member list. This IP address is different from the Host IP address.

When defining the host IP address

the Host table is used in combination with the search order using DNS:

As you can see during an IPL both the interfaces 10.10.5.54 and 10.10.5.55 are started and although the host ACC.MYDOMAIN.NL is known by the first IP address the SSH Connection Refused was caused by the fact that the Node was reaching out using the 10.105.55 address instead of the one defined for the host.

According to IBM support, changing the second IP address and removing the AUTOSTART option would not solve this problem as a round robin is used to set the outgoing IP address.

The only good solution, according to IBM i Support, is to solve this random issue by forcing the IBM i to use the host IP address by making an entry in the routing table.

Below is an example of adding a default route:

The lesson learned here is that even when you think or believe in the rule “if it ain’t broken don’t fix”, maybe it was already broken, but you were always lucky.