SMF 70-1 – Where Some More Of The Wild Things Are

(First posted February 21, 2021)

As I recall, the last time I wrote about SMF 70-1 records in detail was Engineering – Part Two – Non-Integer Weights Are A Thing. Even if it weren’t, no matter – as I’d like you to take a look at it. The reason is to reacquaint you with ERBSCAN and ERBSHOW – two invaluable tools when understanding the detailed structure and contents of an SMF record. (Really, an RMF SMF record.) And it does introduce you to the concept of a Logical Processor Data Section.

This post is another dive into detailed record structure. (The first attempt at the last sentence had the word “derailed”; That might tell me something.) 🙂

In most cases a system cuts a single SMF 70 Subtype 1 record per interval. But this post is not about those cases.

The Structure Of A 70-1 Record

SMF 70-1 is one of the more complex record subtypes – and one of the most valuable.

Here is a synopsis of the layout:

What is in blue(ish) relates to the system cutting the record. The other colours are for other LPARs.

At its simplest, a single 70-1 record represents all the LPARs on the machine. But it’s not always that simple.

Let me point out some key features.

• The CPU Data Sections are 1 per processor for the system that cut the record. In this example there are three – so this is a 3-way.
• zIIPs and GCPs are treated the same, but they are individually identifiable as zIIP or GCP.
• There is one Partition Data Section per logical partition on the machine, plus 1 called “*PHYSCAL”.
• There is one Logical Processor Data Section per logical processor, plus 1 per physical processor.

The colour coding is useful here. Let’s divide it into two cases:

• The processors for the cutting LPAR.
• The processors for the other LPARs.

For what we’ll call “this LPAR”, there are CPU Data Sections for each processor, plus a Partition Data Section, Logical Core Data Sections, and Logical Processor Data Sections.

For each of what we’ll call “other LPARs” there are just the Partition Data Section and its Logical Processor Data Sections.

You’ll notice that the blue Partition Data Section and its Logical Processor Data Sections are the first in their respective categories. I’ve always seen it to be the case that this LPAR’s sections come first. I assume PR/SM returns them in that sequence – though I don’t know if this is an architectural requirement.

The relationship between Partition Data Sections and the corresponding Logical Processor Data Sections is straightforward: Each Partition Data Section points to the first Logical Processor Data Section for that LPAR and has a count of the number of such sections. The pointer here is an index into the set of Logical Processor Data Sections, where the first has an index of 0. (ERBSHOW calls it “#1”.)

(A deactivated LPAR has an (irrelevant) index and a count of 0 – and that’s how my code detects them.)

So far so good, and quite complex.

How Do You Get Multiple 70-1 Records In An Interval?

Obviously each system cuts at least one record per interval – if 70-1 is enabled. So this is not about that.

In recent years the number of physical processors in a machine and logical processors per LPAR have both increased. I regard these as technological trends, driven mainly by capacity. At the same time there is an architectural trend towards more LPARs per machine.

Here are the sizes of the relevant sections – as of z/OS 2.4:

• CPU Data Section: 92 bytes.
• Partition Data Section: 80 bytes.
• Logical Processor Data Section: 88 bytes.
• Logical Core Data Section: 16 bytes.

These might not seem like large numbers but you can probably see where this is heading.

An SMF record can be up to about 32KB in size. You can only fit a few hundred Logical Processor Data Sections into 32KB, and that number might be significantly truncated if this LPAR has a lot of processors.

All of this was easy with machines with few logical processors (and still is).

But let’s take the case of a 100-way LPAR (whatever we think of that.) Its own sections are (92 + 88 + 16) x 100 or 19.6KB plus some other sections. So at least 20KB. And that’s before we consider sections for other LPARs.

Now let’s ignore this LPAR and consider the case of 50 1-way LPARs. There the PR/SM related sections add up to (80 + 88) x 50 = 8.4KB. Of course it’s extremely unlikely many would be 1-way LPARs, so the numbers are realistically much higher than that.

By the way, for a logical processor to count in any of this it just has to be defined. It might well have zero Online Time. It might well be a Parked Vertical Low. It doesn’t matter. The Logical Processor Data Sections are still there.

So, to exceed the capacity of a 32KB 70-1 SMF record we just have to have a lot of logical processors across all the LPARs in the machine, whether in this system or other LPARs. And an exacerbating factor is if these logical processors are across lots of LPARs.

What Does RMF Do If The Data Won’t Fit One Record?

I’ve seen a lot of SMF 70-1 records in my time, and spent a lot of time with ERBSCAN and ERBSHOW examining them at the byte (and sometimes bit) level.

I do know RMF takes great care to adjust how it lays out the records.

Firstly, to state the obvious, RMF doesn’t throw away data; All the sections exist in some record in the sequence.

Secondly, RMF keeps each LPAR’s sections together. So the Partition Data Section and its related Logical Processor Data Sections are all in the same record. This is obviously the right thing to do, otherwise the index and count for the Logical Processor Data Sections could break.

Thirdly, and this is something I hadn’t figured out before, only one record in the sequence contains the CPU Data Sections. (I think also the Logical Core Data Sections.)

How Should I Handle The Multi-Record Case?

Let me assume you’re actually going to have to decide how to deal with this.

There are two basic strategies:

1. Assemble the records in the sequence into one record in memory.
2. Handle each record separately.

Our code, rightly in my opinion, uses Strategy 2. Strategy 1 has some issues:

• Collecting information from multiple records, and timing the processing of the composite data.
• Fixing up things like the index of the Logical Processor Data Sections.

Probably some tools do this, but it’s fiddly.

So we process each LPAR separately, thanks to all the information being in one record. And so we can process each record separately.

Reality Check

If you have only one 70-1 record per interval per cutting system none of the above is necessary to know. But I think it’s interesting.

If you rely on some tooling to process the records – and most sensible people do – you probably don’t care about their structure. Certainly, the RMF Postprocessor gets this right for you in the CPU Activity Report (and Partition Data Report sub-report).

So, I’ve probably lost most of my audience at this point. 🙂 If not, you’re on my wavelength – which isn’t crowded. 🙂 (This is the second “on my wavelength” joke in my arsenal, the other being open to misinterpretation.)

I like to get down into the physical records for a number of reasons, not least of which are:

• When things break I need to fix them.
• It cements my understanding of how what they describe works.

Oh, and it’s fun, too.

Final Thoughts

This post was inspired by a situation that required yet more adjusting of our code. Sometimes life’s that way. In particular, a number of LPARs were missing – because our Assembler code threw away any record with no CPU Data Sections. (This is inherited code but it’s quite possibly a problem I introduced some time in the past 20 years.)

I should point out that – for simplicity – I’ve ignored IFLs, (now very rare) zAAPs, and ICFs. They are treated exactly the same as GCPs and zIIPs. Of course the record-cutting LPAR won’t have IFLs or ICFs.

I have a quite old presentation “Much Ado About CPU”. Maybe I should write one with “Part Two” tacked on. Or maybe “Renewed” – if it’s not such a radical departure. But then I’ve done quite a bit of presentation writing on the general topic of CPU over recent years.