Informing the IBM Community

Masterclass: VIO Server Networking Tips & Techniques

3
(1)

Over the past few years, I have picked up a number of tips and techniques to help when working with LPARs on Power Systems that use VIOS for networking. VIOS can provide highly performant, resilient and redundant networking capabilities to IBM i, AIX and Linux LPARs.

While customers are comfortable with VIOS networking capabilities once they understand their features, there are, inevitably, many questions:

  • How do I know which VIOS my traffic is flowing through?
  • How can I prove that the HA capabilities of VIOS networking behave as expected?
  • How can I view network statistics for a single LPAR rather than all network traffic?
  • How can I retrieve network stats for diagnostics, e.g. using WireShark?

In this article, I’ll cover the above plus a few other handy hints. Be prepared, though, you’ll be getting your hands dirty with the VIOS CLI.

VIOS uses one or more physical Ethernet NICs (network interface cards) to connect to your core network so that the client LPARs can be reached. We refer to these NICs as Shared Ethernet Adapters (SEA). An SEA is basically a trunk which carries network traffic between the client LPARs and the devices on the core network.

entstat command

The most useful command in the networking armoury is the entstat command. This is a native AIX command but also has a VIOS version too. In this article, my SEA has a device name of ent10. To access the stats for the SEA, I’d enter:

entstat -all ent10

Pages and pages of data will fly past your eyes so I normally control this by using:

entstat -all ent10| more

VIOS Masterclass Figure 1


Now I’ll get the first page of stats and I can peruse the data at my leisure. As I’m looking at stats for an SEA, it will show me data for the physical and virtual devices configured for this SEA. If I have an EtherChannel device configured as my SEA, it will also show the stats for the EtherChannel device too.

With entstat, I can see the how much data is flowing through the SEA, which VLANs are being served by that SEA and the virtual adapters using the SEA as a trunk.

seastat command

Another really useful command is seastat. This allows us to view the stats for the individual client LPARs. By default there won’t be any information available as we need to switch this on:

chdev -dev ent10 -attr accounting=enabled

This will instruct VIOS to start recording the client LPAR stats. Personally, I’d only turn this on if you need to do some troubleshooting and then I’d switch it off again. I’ve not noticed it causing any delays when I have switched it on but there’s probably a good reason IBM has the default as disabled.

To see the stats for my SEA, I would enter:

seastat -d ent10|more

VIOS masterclass figure 2


This is slightly different from other VIOS commands as I’d expect it to use -dev rather than -d.

Again, you will probably get pages of information so we can reduce this by using the search parameter, -s.

seastat -d ent10 -s ip=192.168.1.2
seastat -d ent10 -s vlan=1234
seastat -d ent10 -s host=mylpar

Testing VIOS HA capabilities

This is something that all my customers want to test. They’ve spent good money on additional hardware to provide VIOS resiliency so they need to see it working.

Before you start your test, get a ping running to one or more of your LPARs from a PC, server or any other device not associated with the Power System you’re working with. Also, run the command on VIOS to check the current SEA ha setting:

lsdev -dev ent10 -attr

Take note of the ha_mode attribute. This should be “sharing” or “auto”.

Once you have your SEA set up in each VIOS, it really is as simple as running the following command on one of your VIOS LPARs:

chdev -dev ent10 -attr ha_mode=standby

This will put the SEA into standby mode so that it no longer accepts network traffic. All the network traffic should be flowing through your second VIOS. You may notice a couple of packets being dropped in your pings, but nothing significant.

I have provided a script I use which summarises the most relevant data when doing your tests in the references section at the end of this article.

lsseastats.ksh will show the following information:

VIOS Masterclass figure 3


To reinstate the SEA, enter:

chdev -dev ent10 -attr -ha_mode=auto/standby

Make sure you put in the value you recorded from the lsdev command above.

Once again, you may see some dropped packets on your pings. I would then log in to the second VIOS and repeat the test.

tcpdump command

The previous two commands provide you with stats as, at the moment, you run the commands. I had a customer who was struggling to find where a network performance issue was occurring and had checked their firewalls and routers with no success.

The only piece in the jigsaw they hadn’t carried out any detailed analysis on was VIOS (I did tell them it wasn’t VIOS or IBM i that was the problem; turns out I was right). So we ran the tcpdump command to gather real-time packet statistics which we could analyse with WireShark software.

This is a native AIX command so needs to be executed outside of the VIOS shell by entering:

oem_setup_env

This is really useful but bear in mind that tcpdump works on tcpip packets so requires that the adapter we analyse has an IP address assigned. I would not normally assign an IP address to an SEA so, for this, I added a dummy address to en10 (NB, not ent10):

ifconfig en10 192.168.199.10 netmask 255.255.255.0
ifconfig en10 up

If you have an IP address on your SEA then just run the tcpdump commands. tcpdump -d will list the interfaces available for analysis.

We ran the following:

tcpdump -n -i 3 -w oracle-test.pcap -s 0 vlan 1301 and host 10.1.2.3

-n    Do not resolve host names
-i    Use interface number 3 (from tcpdump -d)
-w    Name of the output file to store the tcpdump information
-s    Amount of data per line to collect. 0 means all data

The remainder is what we want tcdump to analyse. Packets where the VLAN ID is 1301 and the host IP address is 10.1.2.3

When we’ve collected a sufficient amount of data, we simply CTRL-C to end and then transfer the file from VIOS and import it in to WireShark for analysis.

If you created a dummy IP address for your SEA, you can remove it using the following two commands:

ifconfig en10 down
ifconfig en10 192.168.199.10 detach

Which Cisco switch port is my NIC attached to?

Using tcpdump, we can also retrieve stats from a Cisco switch using the Cisco Discovery Protocol (CDP). I never thought I’d need this until I experienced some documentation issues at a customer with copper cabling.

I found an AIX script online which was a few years old. This script no longer provided the information I needed. After some digging around, I came up with the following:

tcpdump -i en10 -s 1500 -x -c 1 -vv ‘ether [20:2] = 0x2000’ |grep -v “0:  “

I’m not going to explain it. Suffice to say that it requires an adapter with an IP address, as above. The only thing you should need to change is the -i parameter value.

Be aware that it may take up to a minute for any data to be returned. The output from my command was:

tcpdump: listening on en10, link-type 1, capture size 1500 bytes
261 packets received by filter
14:26:52.769283 CDP v2, ttl: 180s, checksum: 692 (unverified)
0       Device-ID (0x01), length: 18 bytes: 'DC1_SWFARM_SW1'
packets dropped by kernel
Version String (0x05), length: 248 bytes:
Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 12.2(55)SE5, RELEASE SOFTWARE (fc1)
Technical Support: https://www.cisco.com/techsupport
 Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Thu 09-Feb-12 18:14 by prod_rel_team
Platform (0x06), length: 18 bytes: 'cisco WS-C3750X-48'
Address (0x02), length: 13 bytes: IPv4 (1) 10.2.3.4
 Port-ID (0x03), length: 21 bytes: 'GigabitEthernet1/0/24'
Capability (0x04), length: 4 bytes: (0x00000029): Router, L2 Switch, IGMP snooping
Protocol-Hello option (0x08), length: 32 bytes:
VTP Management Domain (0x09), length: 0 byte: ''

The two lines we’re interested in are:

Device-ID, which provides the host name of the switch.
Port-ID, which shows which switch port this adapter is physically connected to.

In my next article, I’ll cover how we go about creating an EtherChannel connection in VIOS which we can then use as an SEA. I’ll also provide some information on how we can view the EtherChannel adapter usage and how to see data flowing through the bundled adapters.

References:

Useful seastat information on DeveloperWorks.

IBM KnowledgeCentre: The seastat command.

IBM KnowledgeCentre: The tcpdump command

My script that summarises the most relevant data when testing VIOS HA capabilities:

Bash
#!/bin/ksh
 #
 # Display Ethernet statistics for SEA
 #
 #
 #
 FILE=/tmp/seastats
 
 # Generate a work file with the stats
 entstat -d $(lsdev |grep "Shared Ethernet"| awk '{ print $1 }')>$FILE
 
 clear
 
 # Print the first two lines
 head -2 $FILE
 
 echo "\nHA mode for this SEA"
 awk /"High Availability Mode:"/ $FILE
 
 echo "\nSEA current state"
 awk /"   State:"/ $FILE| tr -d " "
 
 echo "\nCurrent bridge mode for this SEA"
 awk /"Bridge Mode:"/ $FILE
 
 echo "\nPhysical adapter used by SEA:"
 awk /"Real Adapter:"/ $FILE
 
 echo "\nVirtual adapters using SEA:"
 awk /"Virtual Adapter:"/ $FILE
 
 echo "\nVLANs configured to use this SEA"
 awk /"VLAN Tag IDs:"/ $FILE|grep -v None
 
 echo "\nVLANs currently using this SEA"
 awk /"VID shared"/ $FILE
 
 echo "\n"
 
 head -1 $FILE
 
 rm $FILE
 

How useful was this post?

Click on a star to rate it!

Average rating 3 / 5. Vote count: 1

No votes so far! Be the first to rate this post.