Engineering – Part 9 – Perhaps Affinity Nodes Are Interesting After All

I’ve tended to think Affinity Nodes aren’t worthy of study – because there’s little you can do to influence them.

However, that doesn’t mean one shouldn’t understand them.

(In this post I’m going to say “processor” when I mean “logical processor” and “affnode” when I mean “affinity node”. Also “VL” for “Vertical High”, “VM” for “Vertical Medium”, and “VL” for “Vertical Low”. It’s going to get wordy enough as it is.)

What Is An Affinity Node?

While a z/OS LPAR has logical processors in two pools – GCP and zIIP – there is further division into affinity nodes.

Each affnode generally contains a subset of the processors from a given pool. When HiperDispatch was announced that subset was said to be 4 processors.

Work, with the exception of SYSSTC, runs essentially in a single affnode. (SYSSTC work can run anywhere, being generally short running work. And, barring one of my war stories, not at all disruptive.)

So, you could imagine an affnode has queueing characteristics like a smaller LPAR. The payoff – perhaps set against this – is that work tends to be redispatched on the same small set of processors, leading to better cache effectiveness.

There are two mechanisms related to affnodes that are important to understand, at least conceptually:

• In the short term an overloaded affnode can ask for help from other affnodes. This is the “needs help” mechanism you might’ve heard of.
• In the longer term work can be redistributed among affnodes – for balancing purposes.

There is in fact a third mechanism:

• The processors in an affnode can be moved to other affnodes. Indeed affnodes can be formed and dissolved.

How Do I Observe Affinity Nodes?

Prior to z16 SMF 99-14 records were the only SMF way to get processor home addresses. It was also the only SMF way to see affnodes.

In Engineering – Part 7 – RMF Processor Home Address Fields I described how SMF 70-1 support for z16 added home addresses for LPARs’ processors. What I didn’t say was that it added affnode support. That’s because it doesn’t.

So, the source of affnode information is SMF 99-14. For a long time my code that extracted processor home addresses for me also extracted affnodes.

Recently I started enhancing this code (unimaginatively called “XTR9914”).

And taking another look is what got me to this blog post.

It’s important to note that 99-14 doesn’t give me information on what runs in each affnode, nor how busy each processor is. For the latter you need SMF 70-1 or 113. The former isn’t catered for at all.

It’s also important to note that evidence for affnodes helping each other comes from SMF 72-3 at the service class period level. Here you’ll see zIIP-eligible work running on a GCP. What you won’t explicitly see is work not eligible for zIIP running on a helper node. Also, evidence for delay in either getting help or eventually running on the home affnode is in 72-3.

There probably is some other SMF 99 subtype that tells you more – and I’d be delighted if someone pointed me to any such information.

What I’ve Observed

I’ve observed a fair number of things – from SMF 99-14:

• An affnode contains GCPs or zIIPs – but not both. One obvious corollary of this is that work could run in 2 affnodes – even without “needs help”. (I’ve not seen evidence of this.)
• Affnodes can be of mixed polarities.
• Affnodes tend to have fewer processors than 4. If the number is 4 the affnode tends to contain VM or even VL processors.
• Affnodes in z16 are in a single chip – which makes sense from a Virtual Level 3 Cache perspective.
• Affnodes do indeed come and go. But it is very infrequent.
• Similarly, processors can move between affnodes – on a very infrequent basis.

The last two are interesting – and tend to occur when processors change affinity. I know this because I can see the polarities of individual processors and which affinity node they are in.

(Processors changing affinity tends to happen when another LPAR is activated or deactivated, or weights being altered.)

Helper nodes are interesting:

• I can see that no zIIP affnode helps a GCP affnode. This is to be expected.
• I can see GCP affnodes help GCP affnodes.
• I can see each node tends to have multiple helper nodes. And – for a small enough LPAR – some have none.
• I can see a definite preference list.
  • For an LPAR with enough zIIPs a zIIP affnode has another zIIP affnode first on its list, followed by the GCP nodes.
  • At least for GGP affnodes I can see helper affnode preferences rotating. (Well, sort of but close enough.)
  • I haven’t looked at an LPAR with 3 or more zIIP affnodes so I can only assume their preference lists rotate.

Observing these things brings the whole idea of affnodes to life for me.

Here is an example – before I’ve done anything fancy with the reporting:

In the above Affinity Node 4 is a zIIP node. You can see it has the other three as helper nodes. They are GCP nodes – and help each other.

99-24’s are cut on a 5 minute interval – or when the topology changes. So here’s another one:

• While it’s possible affnodes change within the interval and then change back again the long term constancy of what 99-14 shows suggests to me this isn’t happening. In other words, affnodes don’t change much – assuming configuration changes or eg IRD Weight Management aren’t a disruptive factor.

Conclusion

While it’s true you can’t do much to influence the formation of affinity nodes, it might influence your LPAR design a little.

Their behaviour is not what I expected – and not quite what the literature would have you believe.

One final point: If you understand which logical processors are in which affinity nodes perhaps it will help you understand any imbalance in utilisation between an LPAR’s logical processors.

In any case, I’ve found this bit of observing and learning most interesting. It’s enough to make me want SMF 99 Subtype 14 records in each engagement, whether the outcome is some “tourist information” or an actually useful insight.

Making Of

Yet again I find myself on a daytime flight that was supposed to be a nighttime flight. Again to Johannesburg. So what does one do with 11 hours when one is unlikely to sleep? So that’s how this one got written.

And it got finished off at 5AM on the return flight.