Ever wondered about the firmware level of any adapter present in a POWER server? My guess is that you probably never have or never needed to.

Allow me to tell you a story which took place at the beginning of April.

It all started with messages in QSYSOPR at the weekend. Messages with ID CPPEA33 and   CPPEA35 started popping up. For those familiar with IBM i and a SAN you will recognize the first message as the one telling you that a path to a disk is lost, the second is for the path being restored.

The IBM i environment consisted of a production POWER8 with Storwize V5000 and a CBU POWER8 also with Storwize V5000. Hardware replication on the SAN, on both sides redundant VIOS servers, to complete the picture.

The VIOS on the CBU side had been upgraded in January from version 2.2.6.31 to 3.1.1.10 and in March on the production system. A first analysis of the problem did reveal a code we noticed in VIOS “ECCE4018”. As a search on the Internet did not bring us any closer to a solution we opened a support case with IBM. After mailing them the results of the SNAP data of every VIOS with the error we were soon told which iFix we needed to apply which did solve the problem.

The exact instructions were:

• IJ22290 and IJ23222 must be installed to fix the shown FCA Errors for the NPIV Adapters hosted by the EN0A FC Adapter.
  
  
• EN0A FC Adapter Microcode must be updated to latest Level, since the installed Level is no longer supported.
  
  
• The patches we were told to apply were not suggested when we used the IBM i Fix Level Recommendation Tool for POWER systems. They were however listed in the tool for the version level of the VIOS under the section “The following HIPER APARs apply”.

So, my advice is to check FLRT when preparing for a VIOS update or upgrade and take a look at the HIPER APARs or even better, simply apply them before updating or upgrading. The link below will bring you to a section of FLRT where all the VIOS Hipers can be found:
https://www14.software.ibm.com/webapp/set2/flrt/doc?page=hiper&os=vios_hiper

The instructions to update the microcode of the adapter did puzzle me a little. IBM i is my area of expertise, VIOS for me is a black box. We were dealing here with a POWER8 environment with redundant HMC’s and frequent FSP firmware updates/upgrades. It turns out that the way to update the microcode of the fibre channel (FC) adapter is through the VIOS LPAR which owns the adapter.

In my mind I had to take the following candidates off the list as being able to update the FC adapter microcode: the Hardware Management Console a FSP firmware update and a VIOS update/upgrade. Kind of funny that the word “Hardware” in Hardware Management Console is present, but is unable to complete this task!

I also learned that for some FC adapters you need to update the microcode of every port of that adapter, for some updating just one port will do. Also you are instructed to save the configuration of the adapter, just in case. With my IBM i background I have to pinch myself and remember that we are living in 2020 and updating an adapter is something we in the IBM i world never bothered about.

Updating the adapters given to VIOS to ensure that they are not running with microcode no longer supported by IBM, is task on it’s own.

I wonder if it’s possible to have one additional LPAR running IBM i which you initially give all the adapters configured for a VIOS LPAR, which you then shut down.

Would that IBM i LPAR be able to update the microcode for any adapter and keep them on a supported version? As adapters can be exchanged between VIOS, Linux, AIX and IBM I, for sure there will never be a version conflict I guess.

So one small IBM i LPAR to perform this simple adapter microcode task by an IPL of the system in an automated process. Just giving it the extra adapters and removing them after a shutdown. IBM i as the captain of the ship, one LPAR to update them all, one LPAR to control them all.

I wonder if one of the readers can give a comment and let me know if I am seeing ghosts in thinking that this can be done?