Framed

Increasingly customers are using 1MB and 2GB page frames. Recent experiences with them managing these areas have led me to writing this post. It’s about the two LFAREAs. 1

In the following I’m going to prefer to say “page” instead of “page frame” – unless the meaning would be unclear.

What Is The Point Of A Large Frame?

To speed Dynamic Address Translation (DAT) mainframe cores have Translation Lookaside Buffers (or TLB entries).

The purpose of DAT – a feature of virtual memory operating systems – is to translate virtual storage page addresses to real storage frame addresses. (Of course a virtual storage frame might not have one – if it has been paged or swapped to disk or not yet touched.2)

The TLB is small and cannot cover all of virtual memory. 3

If a translation operation cannot find a TLB entry some extra CPU is consumed performing the translation.

So we want as high a TLB hit ratio as possible.

Traditionally a TLB entry has covered a 4KB page – and most still do.

As z/OS systems’ memory sizes have increased the TLB hit ratio has declined; The larger the system the worse this is likely to be.

First 1MB pages were introduced – where 1 TLB entry covers 1MB of virtual and real storage. This can greatly improve the TLB hit ratio for the affected pages.

Later on 2GB pages were introduced. This has the potential to improve TLB hit ratios for the affected pages still further.

These page sizes are requested by the application, the most common or sizeable being Db2 or JVM heaps.

Most of the rest of this post is about controls on the amount of memory that can be in 1MB and 2GB pages. It’s important to

  • Avoid wastage of real memory.

And

  • Ensure requests for large pages are effective.

The two LFAREAs behave differently. Understanding their differences is key to both of these.

The instrumentation rounds out this post nicely.

1MB LFAREA

Before z/OS 2.3 the 1MB LFAREA carved out real storage for 1MB pages and it couldn’t be used for 4 KB pages.

With z/OS 2.3 it is now a limit:

If applications attempt to acquire 1MB pages and there aren’t sufficient left in the 1MB LFAREA the acquisition will fail. It is up to the requester to handle this situation as they see fit; In all likelihood some or all of the virtual storage will be backed with 4KB ones.

So you need to specify a 1MB LFAREA at least as large as your peak requirement. Importantly, there is no downside to overspecification.

2GB LFAREA

The 2GB LFAREA carves out real storage for 2GB pages and it can’t be used for smaller page sizes.

Overspecification leads to unusable memory – and should be minimised.

Underspecification leads to disappointment – in the same way underspecifying the 1MB LFAREA does.

So here it’s a balance – and quite a difficult one.

Instrumentation

This is perhaps my favourite topic.

There is instrumentation at the system and at the address space level:

System

For memory, SMF Type 71 records are essential.

You get both the LFAREA specifications and usage – for both page sizes. (In fact all 3.) This enables you to assess whether the limits are too high or too low.

For 1MB LFAREA the limit is in field SMF71LFA and the available value is SMF71L7A. Obviously you can get the usage by subtraction. These values are in MB so I turn them into GB. One day even that might not be adequate.

When it comes to service classes and report classes there are no extra fields in the SMF Type 72 Subtype 3 records. My code subtracts the total workload memory usage from what’s left when you take out the other uses of in-use frames. 1MB frames come out of that.

For 2GB LFAREA the limit is in field SMF71GFA and the average used value is SMF71GUA. Obviously you can get the free by subtraction. These values are in 2GB pages so I turn them into GB.

2GB frames are “special”: Unused 2GB page frames are not counted in SMF 71’s Available Frame Queue metrics; I treat them separately (and get their size by subtraction SMF71GUA from SMF71GFA).

For neither page size can you calculate “would be pages” from RMF. Db2 has information in its DISPLAY BUFFERPOOL command response – but that is obviously only one exploiter. (DISPLAY BUFFERPOOL can tell you how many pages were 2GB and how many were 1MB, for instance.)

Address Space

Address spaces are – in a general sense – instrumented with SMF Type 30 records:

  • Interval records are useful for understanding the fluctuations in usage.
  • Step- and Job-End records are useful in the batch context.

Both give you 1MB and 2GB page usage. You get both the “who” and “how much”. This feeds into my quest for taxonomy.

Conclusion

The controls for 1MB and 2GB are different – and more forgiving in the 1MB case.

Most of the benefit is in using 1MB pages and 2GB is just a little bit more. That statement would be misleading for anyone reading this in 2036 – as memory configurations will have vastly grown by then (despite the true-in-2026 AI memory land grab).

For most customers my view is 1MB pages are probably the best choice.

In either case there is instrumentation to help you size the two LFAREAs and figure out who’s using them.

Our code uses this instrumentation – and I’ve seen both underspecification and underuse – across numerous customers.

And I have prototype code that graphs the use of these two areas – but it could use work when I get a spare moment. Perhaps the centrepiece of another post.

Making Of

This was largely written on a plane back from South Africa, where I spent happy days talking to (or more correctly at) two customers I know well.

I actually wrote it on an iPhone – so a glass keyboard. Not ideal and here I rely heavily on its ability to correct typos. (On that last sentence I typed “Inrely” and it couldn’t correct it – a failure I encounter often.)

Oh, and footnotes return. A two edged sword for those who actually read them – as non-readers get a smoother flow and readers of them flip back and forth. (I blame the late great Terry Pratchett.)

  1. Technically I should say “large page frame areas” but most people would go with “LFAREAs”, even though that is the name of the specifications. ↩

  2. For decades virtual storage use exceeded real storage availability and paging was essential. Now real storage is relatively abundant and paging is rare. ↩

  3. Like Level 1 Cache it’s small and fast. ↩

Memories Are Made Of This

Wouldn’t you like to know where in the machine an LPAR’s memory is? Perhaps not but for some customers it’s more than a matter of idle curiosity.

The good news is that RMF will be able to tell you: A recent APAR for z/OS 3.2 Data Gatherer OA68545 exploits information from z17 processors to provide precisely this information.

As I write this “will be able to” seems appropriate – as most customers are either not at z/OS 3.2 or not on z17. Of course the APAR is a further requirement – so subject to maintenance yet to be applied.

Why You Might Want To Know

In a single drawer machine the location of an LPAR’s memory might not be all that important to know.

It’s when the machine has more than one drawer that you might like to know:

  • If you reckon you have an LPAR split across drawers you’d want to know if the logical cores were split or whether it was the memory. And during an LPAR drawer swap the cores move immediately and the memory takes time to move.
  • If you are adding LPARs or increasing their memory footprint you probably want to know if there is any threat of cross drawer or of PR/SM having to rework LPAR resource placement.
  • You might want to explain why one LPAR is in one drawer and another is in another – when there are few enough cores between them to fit in one drawer.

What The APAR Gives You

This new APAR deals in memory increments.

It gives you the increment size and how many increments the LPAR has at each approximate location.

By “approximate location” it is certainly enough to tell you how much is in which processor drawer. It’s actually down to what is known as a “node” – which I think is a DCM.

What you don’t get is any kind of address – but I don’t yet see that as a shortcoming.

In my first (internal) set of data I see how “PHYSICAL” is treated: You can’t tell how much purchased memory is in each drawer. This is a bit of a shame. From Vital Product Data I get quite a good view of this, by the way. But most people don’t have access to that. It’ll remain in the category called “a stunt Martin pulls” – which isn’t actually a category I want it to be in.

I don’t expect the information to include anything to do with Virtual Flash Memory or anything outside customer addressable memory. My one set of data – as I write this – doesn’t include a machine with VFM defined for any of the LPARs.

There is a new record subtype to deal with: SMF 70 Subtype 3. So this isn’t crammed into the already rather complex SMF 70 Subtype 1. I think that’s sensible.

I know CP3000 Development has worked to support this – and I expect the IntelliMagic team to be swift in supporting it. For my part, the record volume is going to be quite low so I can write REXX against the records and expect good performance. I’m increasingly writing REXX code to process raw SMF records, sometimes with Python post-processing the results.

(As an aside, REXX has been able to process VBS records such as SMF since z/OS 2.1.)

Right now I’ve eyeballed – using ERBSCAN and ERBSHOW – the one set of data I’ve got but I haven’t written the REXX; You know I’m itching to.

Conclusion

You will probably have spotted – as I’ve talked about it enough – that SMF 70 Subtype 1 brought Logical Processor home addresses to our instrumentation kitbag. And it did it for all LPARs on the machine. That made this instrumentation better than SMF 99 Subtype 14.

This new support does the same thing for memory.

It will be interesting to see what stories it enables us to tell. But already I have some ideas.

It’s just that at the time of writing I’ve only seen a z/OS 3.2 system, let alone one running on z17. As so often happens, it’s an internal system – so I await a real customer system. I definitely have customer situations that could’ve done with this.

One caution: You can’t necessarily predict what PR/SM might do but sensible LPAR Design can be informed by good instrumentation.

Making Of

I again don’t remember where I started this post but I’m finishing it off on a flight to South Africa. C’est la vie.

Time to publish a few.

Mind The Gap

Or “An Indecent Interval”?

I’ve written about RMF intervals before – when writing about System Recovery Boost.

I suppose one lesson of this post is to not assume that all intervals are of equal length. We’ve got pretty used to assuming that a system’s RMF interval is, say, 15 minutes long. We’ve maybe got used to it coinciding with the SMF interval – though that is a bit of a tenuous assumption.

Maybe another lesson is that there can be gaps in the data.

Gap Analysis

Recently I’ve made progress in analysing gaps in the data customers send me. By a gap I mean a period of time for which there is no RMF interval.

Having a gap can sometimes do strange things to calculations. For example, working out CPU utilisation by summing up the CPU used over, say, an hour and dividing by the number of seconds would underestimate – if an interval were missing.

Gaps can be benign, for example if the system were down for a few minutes. (Though an unintended bounce of a system is not what I’d actually call benign.)

Often, though, a gap in the SMF data a customer sends to me is caused by a problem in their handling the SMF data. (Conversely, I quite often see duplicate data.) If I see either of these I tend to point them out to the customer.

Irregularity

I’m also observing that sometimes the interval length is irregular.

The most frequent cause of this is System Recovery Boost or Recovery Process Boost. Almost as often, though, it’s a WLM policy switch that causes intervals to terminate. (You might expect that one to be regular – as most policy switches happen on a schedule.)

Minor irregularity can happen on a seriously overloaded system. Ultraminor – a second or two – is very common and doesn’t bother me.

A Decent Interval?

So what would a nice, regular, synchronised interval look like?

It would be synchronised with the SMF interval, with the actual quarter of an hour, and there wouldn’t be any gaps or duplication.

Conclusion

I think it pays not to assume that there are no gaps in the data, that intervals are all the same length, and that intervals are perfectly synchronised.

Further, it pays to know what the state of the system was during any interval. For example, whether it represented part of a boost period, and of what kind.

And missing intervals require some kind of explanation.

Making Of

I don’t recall when I started this post. I do know I’m finishing it on a plane to South Africa – to see a couple of customers who I hope would consider me to be a friend.

You might consider this one to be mercifully brief, or maybe a little high level. I’d agree with both sentiments.

Product CPU Usage – A Sometimes Useful Metric

It’s possible I’ve written about this before. If so it’s been a while.

What I certainly have written about in the past is the SMF 30 Usage Data Section. I will have described how it can be used to describe architectural attributes of an address space or batch job. For example:

  • That it really is a CICS region. (Not all CICS regions have program name “DFHSIP”.) And what its version and release is.
  • That the address space ( or job) connects to Db2 or MQ – and which subsystem or queue manager in particular.

(I probably shouldn’t go on a rant about how important an appreciation of architecture is in many Performance and Performance-adjacent situations.)

But let me come to the point: You can sometimes get useful CPU numbers out of the SMF 30 Usage Data Section. This post is about experiences with those numbers. Consider it an experiment.

Usage Data CPU Numbers

Notice I didn’t call this section “SMF 30 Usage Data Section CPU Numbers”. That’s because SMF 89 also has some Usage-related CPU numbers.

  • SMF 89 aggregates usage at the system level by product.
  • SMF 30 aggregates usage at the address space level.

Both are available at the interval level. SMF 30 is, of course, available at the step- and job-end level.

Both provide Product TCB and Product SRB times. This is what the rest of the post is about – but from the SMF 30 perspective (which I personally find more interesting).

I should note that SMF 89 also has Product zIIP CPU – which SMF 30 doesn’t.

Playing It By The Numbers

I should note the only reason a Usage Data Section appears in the SMF 30 record is that the IFAUSAGE macro was invoked by the program.

First I would advocate summing the SRB and TCB numbers in the Usage Data Section by Product ID (or Name) and possibly Product Qualifier. The latter gets you the Db2 subsystem or MQ queue manager name, if relevant. Having done that you might well break out the large individual section numbers.

You might attempt to reconcile Usage TCB / SRB time with other metrics from the same SMF 30 record. I will note the numbers for a CICS region are pretty close – which makes perfect sense but isn’t terribly interesting.

The numbers for MQ and Db2 tend to be substantially smaller – which again makes sense. They do reflect actual usage.

Here’s a sanitised set of data.

You will see counts of sections in Column M, along with some taxonomy. The percentage numbers are percentages of a GCP processor. (In this case the GCPs are full (z15) speed.)

I haven’t acquired useful explanations for the numbers of eg Db2 Usage sections. I’ve been waved off thinking they are concurrent Db2 thread counts.

A Caution On Db2

It is very often the case that Usage TCB / SRB is zero, except for the “Db2 Engine” address space. If this is so it’s because the installation disabled the function – in DSNZPARM. It’s said to consume CPU but I’ve seen no evidence this is substantial – so I consider this a pity.

More On Db2

This customer has enabled Usage Data to reflect Db2 usage, so the Db2 numbers for the 2 CICS regions are non-zero.

Looking at the 2 Db2 Engine address spaces we see a very small Db2 subsystem. The GCP numbers are tiny – in fact showing as 0 for the DBM1 address space. (It’s not really 0 but very small.)

The DBM1 address space has some zIIP time. This shows up as Dependent Enclave time – which is normal and reflects some Prefetch and Deferred Write activity.

The DIST address space has some TCB Usage – which agrees with the address space TCB time and the GCP time. The TCB time is Thread Establishment and Termination – which is never zIIP eligible . It’s also DDF application work. The zIIP time is all DDF application work.

Up to 60% of zIIP application work can be zIIP eligible. If we assumed that then the total application work would be 1.5 / 0.6 or 2.5% and so the non-zIIP portion would be 2.5 – 1.5 or 1.0%. The remaining GCP time would be 0.3% – for Thread Establishment and Termination.

Of course that 0.3% is a lower bound but it probably isn’t far off.

At this point I would say better work is done with Db2 Accounting Trace as well as SMF 30. But at least this has been a reasonable sanity check – without the need for the former.

Side note: IRLM is not Db2 and it doesn’t have any Usage Data Sections.

Recent Discoveries

I’m seeing many more instances of Liberty Profile (in CICS regions) and Node (not always in CICS regions). I would suggest these appearing is a useful piece of taxonomy, Furthering the “Architecture” cause.

You’ll see both of these in the CICS Region example.

You’ll notice in both cases large TCB numbers. (Sections 2 and 3 in the first case, and 3 and 4 in the second.)

When you look at them each section’s TCB and SRB numbers are about the same as for the whole address space and also the CICS section’s number. There is a clear danger of double counting here, so be careful. Clearly the same CPU is being recorded more than once.

The same number appearing more than once does not, however, lead to double charging for software.

Both Node and Liberty are well known to use zIIP, which you see documented in Columns F and G.

The address space TCB and SRB numbers are GCP numbers – and it’s those that are reflected in the Usage numbers. CICS itself is all GCP.

Conclusion

There are TCB and SRB numbers in the SMF 30 Usage Data Section – that can help you understand more about the address space or region.

This experiment has shown that the way the Usage Data Section numbers appear is complex. Specialist instrumentation from Db2 or CICS or MQ or IMS is probably better, alongside SMF 30. But those might not be available; Sometimes SMF 30 and RMF is all you have.

Still, it’s been fun looking at this.

Making Of

I started writing this on a flight to Istanbul – where I met lots of customers. I’m finishing it off on the way back. This post has been on my todo list for a long time, possibly years. I don’t even remember what the original intent was. The acronym DRY (“Don’t Repeat Yourself”) is often used. In this case I might have fallen foul of it – but I probably have something different now to say than what I might’ve said before. Such brings experience.

I would also note this line of enquiry has caused some tweaks to my Usage Data reporting code. I’m open minded enough to think there might be more to come. In fact the tweaks aren’t for the Usage Data Section itself, but rather adding zIIP and GCP numbers – to help illuminate the Usage numbers.

Im using the Apple Pencil and this time it / I seem(s) to be behaving fairly well. It could be I’ve got better or maybe the smoothness of the flight. Of course technical terms are still a challenge; iPad OS doesn’t expect them at all.

Prepare To Evacuate

It is rare to need to evacuate hardware. But being rare is not an excuse to neglect to plan for it.

Obviously I’ve been sensitised to the question by real customer situations – though I don’t intend to describe the situations in any detail; This post should prove useful enough without doing that.

When I say “rare” hardware generation upgrades usually require giving the machine over. So I suppose one could make a distinction between such things and repair actions and emergencies.

Perhaps more subtly, concurrent repair and upgrade actions are possible. With these you don’t hand the entire machine over. In fact it’s a valid question whether to or not.

So let’s discuss two options:

  • Evacuating a whole machine
  • Evacuating a processor drawer

(Obviously for a single drawer machine the two are the same. I tend to advocate customers buy machines with at least two drawers.)

Evacuating A Whole Machine

Evacuating a whole machine means taking down all of the LPARs of whatever description on the machine. Some of those LPARs might be running specific workloads, but others might be integral to the whole enterprise, for example coupling facility LPARs with unduplexed structures in or LPARs with DVIPAs in.

The former lead to homes being needed for the work they would’ve run, or for that work to be foregone. The latter need the vital components moved – whether explicitly or automatically.

Most customers I work with have more than one machine for a given workload; Obviously if you don’t the situation is more serious.

Evacuating A Processor Drawer

I won’t go into the mechanics of how to evacuate a drawer; In fact I don’t know how it’s done. (To state the obvious, I’m not a hardware planner nor a customer engineer. I try to nod wisely, though. 😊)

It should be noted that a drawer might need to be pulled out for reasons other than repairs. The prime example I can think of is adding memory cards. Though not all increases in purchased memory require additional hardware, some do. The memory cards have to be accessed from the top of the drawer so it would need to be pulled out – and would need to be deactivated.

PR/SM allocates IFLs and ICFs from the top drawer down. It allocates GCPs and zIIPs from the bottom drawer upwards. (Nomenclature for drawers varies – even within IBM infrastructure – so it might be best to refer to them by their location in the frames, but that’s cumbersome.)

When a drawer is removed any characterised processors would have to be relocated (if possible). Likewise any memory (if possible). It should be expected that PR/SM will rework physical resources as well as LPAR allocations. (Lots of other things cause similar rework – but that’s beyond the scope of this post.)

A well planned configuration will have paths from each processor drawer to each I/O device – though not necessarily directly to each I/O drawer. While the standard planning tools strive to achieve redundant connectivity it is worth checking. (Again not my domain, if you’ll pardon the pun.)

While individual z17 DPUs (Data Processing Units) own specific channels, the loss of a DPU (actually most likely its PU chip) does lead to reworking. Perhaps one day I’ll write about SMF 73 Channel Path information and DPUs.

When I talk about connectivity and processor drawers similar things could be said about I/O drawers.

Capacity Planning

While evacuating a whole machine might lead to more loss of capacity than a single drawer, both scenarios need planning for. There needs to be enough remaining capacity to run the important workloads. I’m finding the conversation about shedding workload in an emergency is very similar to the conversation about making work displaceable. Many customers find it very difficult to find work to shed.

Of course, with the exception of older machines, it might be feasible to temporarily add capacity on surviving machines.

You would want to schedule such activities when there is little work on the affected machine – but such a time is becoming increasingly difficult to find. Particularly if you make the pessimistic assumption that it will take longer than scheduled.

Conclusion

Whether evacuating the whole machine or a single processor drawer planning for it is important. I’d advocate doing the planning at a time when the question is theoretical; You don’t want to do it in an emergency.

I would also advocate keeping an eye on where LPARs’ logical processor home addresses are in the machine. This is much easier with z16 and later machines.

One final thing: A long time ago, when Concurrent Drawer Repair was new, I presented it to a customer. Their response was “That’s very interesting, Mr Packer, but if it’s all the same to you, we’d like to continue with the practice of handing over the whole machine to you”. I wonder how many customers feel the same today.

Making Of

I wrote this post on two flights between London and Barcelona, my first time to that city. And with a new-to-me customer. My aim – for once, it seems – is to publish almost immediately.

What’s Important? CICS Transaction vs Region Goals

When assessing how well a WLM policy protects key work it’s necessary to understand how much CPU is used at each importance level. \ If there is very little work at a lower importance (higher importance number) a CPU crunch will leave work at this importance level vulnerable to CPU queuing.

Let me give a real, recent example:

Db2 “Engine” (DBM1, MSTR, DIST address spaces) were in a service class at Importance 1 – with CPU Critical enabled. This is good. However, these address spaces were the only users of zIIP. (DBM1 mostly for Deferred Write and Prefetch, MSTR for Log Writes.) It is fortunate the zIIPs were lightly used. If they had been heavily used Db2 might well have slowed down. And that would cause Db2 clients e.g. CICS to slow down – even though these clients weren’t directly using zIIP.

In that example, Db2 being the only zIIP user is what I call “a fact of life”. You can’t tune that situation away. Fortunately, adding newer kinds of work that exploit zIIP almost certainly would create displaceable work. So growth in zIIP usage might cure the problem all by itself.

So the key thing we seek is indeed displaceable work.

And I deliberately gave a zIIP example to show it’s not just GCP CPU.

Measuring Displaceable Work

So, how do we measure displaceable work?

Let’s keep this simple by only looking at GCP CPU. (My standard code graphs CPU, zIIP-on-zIIP, and zIIP-on-GCP the same way.)

With RMF Workload Activity Report data (SMF 72-3) we can see CPU By Service Class Period. We can also see Importance for that work’s goal.

So, my code sums CPU by Importance – over all the service class periods. It plots it by time of day – as the picture in the batch window is often very different from the online peak.

But you mustn’t forget SYSTEM and SYSSTC service classes. And you mustn’t forget Discretionary. On the latter you could further divide between SYSOTHER and other discretionary service class periods. Personally I don’t.

A Twist With CICS

(What I’m about to say is true of IMS as well.)

CICS allows you to define (single period) transaction service classes. If you do so the regions’ own goals will be ignored and CICS work will be managed to the transaction service class’ goals.

Actually the dispatching priority of the region will be in support of the transaction goals. If there are multiple transaction service classes served by the same region the most important one will drive things. And the other transactions will come along for the ride. This is because it is the address space that has a dispatching priority, not an individual transaction.

So, if the transaction goal overrides the region goal, how do you tell what Importance the CICS CPU is at?

Unless you can tell the Importance of the predominant transactions in the region and the region’s CPU it’s difficult. The latter can, of course, be obtained from SMF 30. But the correspondence between transactions and regions isn’t in RMF or in SMF 30. You could get it from SMF 110 CICS Monitor Trace – but that would be expensive. And it doesn’t speak to CICS transactions’ service classes.

Fortunately, in most CICS customers, the service class of the transactions has the same WLM Importance as the regions. But not always. For example, a transaction might flow from a TOR (perhaps at Importance 1) to an AOR (perhaps at Importance 2).

And this, in a nutshell, is why I am careful about assessing displaceable work when CICS and IMS are involved.

Conclusion

Understanding the amount of CPU at the various Importance levels is important when understanding how resilient the WLM setup is. But it’s not always straightforward.

A few other notes:

  • Importance is not the same as dispatching priority – though it is the most scalable way of assessing displaceability.
  • Having little displaceable work should suggest you can’t run the machine all that busy. And you have to look at GCPs and zIIPs separately.
  • I’ve not really talked about what Importance work should be classified to. You can have all the displaceable work in the world but if work is misclassified it could still avail you little.

One final point: I often ask customers questions like “if you lost a machine what work could you sacrifice?” To my mind this is similar to which work is displaceable. I have in mind another blog post on this subject.

Making Of

Again another “written on a plane” post. For the first time ever, though, I had to restore the Markdown from a prior version. Somehow the text had mostly disappeared. Thankfully Drafts stores versions in iCloud. So, no harm done. Still, it freaked me out. Be careful to have automatic backups, folks.

Actually it was written on two plane journeys, some months apart. I’d had it on my task list to complete but somehow never got round to it. Well, now I have.

Remember Memory

In Engineering – Part 8 – Remote Access Detection I talked about reasons why LPARs might go cross-drawer.

One reason I didn’t give was about memory.

(I should perhaps remind you that on modern machines a logical processor accessing Level 4 Cache or memory in another processor drawer incurs a significant performance penalty, showing up as an increase in Cycles Per Instruction (or CPI) in SMF 113. Engineering – Part 8 – Remote Access Detection talks about techniques to look at cross-drawer.)

I was reminded recently of something I already knew but hadn’t seen much incidence of: LPAR memory configuration leading to LPARs split across drawers.

An Example

Here is a theoretical example that illustrates the point:

A 2-drawer machine has 2TB of memory. (Actually this is a little modest these days.) There are three LPARs:

  • LPAR A with 800GB of memory and fewer than half the number of online logical cores that would fit in a drawer.
  • LPAR B with 400GB of memory and even fewer logical cores than LPAR A.
  • LPAR C with 700GB of memory and again fewer logical cores than half a drawer’s worth.

(When counting cores you need to add together all the cores in all the pools. So, for z/OS that would be both GCPs and zIIPs.)

Ignore the cores, except to note that any pair of LPARs taken together has fewer cores than would fit in a drawer.

With two drawers and three LPARs PR/SM has an interesting challenge: There are three possible pairs of LPARs to try to fit into a drawer. (More than one LPAR In a drawer is not a problem, in principle.) The three combinations are:

  • A + B: 1200GB
  • A + C: 1500GB
  • B + C: 1100GB

In fact all three LPARs add up to 1900GB, so there is enough memory on the machine to fit all three in the two drawers.

It is obvious that some LPAR will have to have memory allocated in both drawers.

PR/SM Decision Making

In the z17 Technical Guide I clarified the following thing: When allocating machine resources – memory and cores – PR/SM allocates them to the LPAR with the highest entitlement first, then the next highest, and so on. So far no clarification. The clarification is what the term entitlement means:

  • For dedicated core LPARs think of it as the number of cores (in all pools).
  • For shared core LPARs think of it as the number of cores’ worth of weight ( in all pools).

A general piece of advice is not to try too hard to second guess what PR/SM would do – but to aim for sensible LPAR design.

The example in this post does not represent the best possible design, though it might be necessary.

How Could This Situation Be Improved?

There are two main approaches, and each has its own downside:

  1. Buy more memory. That obviously costs money – so maybe not popular. But when buying a new machine this sort of thing should be taken into account.
  2. Reduce the memory footprint of the LPARs – but that has performance implications.

When I look at LPAR memory I generally see a lot spare – but not always. This can be established with SMF 71. But it is important to consider things like recovery scenarios: That apparently spare memory might be there to allow, for example, for a Db2 subsystem to be recovered, along with maybe some CICS regions. This is one reason I take an interest in Architecture and what uses installations put their hardware and software to.

Conclusion

LPAR Design is an interesting topic – but it can’t be treated as an exact science.

I would also note in closing two things:

  • You can’t from RMF tell how much memory is purchased in each drawer. Vital Product Data (VPD) is something a Customer Engineer has access to – and it does document this.
  • If an LPAR moves drawer it can take some time to move. The larger the LPAR’s memory (including Virtual Flash) the longer the more it can take – with a potential for cross-drawer memory access.

Perhaps you shouldn’t just assign all purchased memory to an LPAR.

Making Of

I’m writing this on a plane journey home, having spent the week with two rather sophisticated customers. Some of what I’ve written is inspired by them, some not. Writing, to me, is an assimilation of what I’ve experienced and thought about – often over many months. These two might just recognise themselves in bits of this but nobody else would. And the example is a simplified version of something real.

That last sentence was hard fought as I experimented with picking the iPad up and writing on it directly with the Apple Pencil. That didn’t work very well. In general, though, writing with the Pencil is getting a little easier.

And something is causing suggested text replacements to be better – and the acronym “LPAR” to be better recognised. It’s probably AI as I think it’s hallucinating words into existence. 😀

Two Of These Are Not Like The Others

They were simpler times back then.

“Back when?” You might ask.

When PR/SM got started – in the mid 1980’s – a machine might have two or three LPARs. Similarly, when Parallel Sysplex got started the number of members was very small.

For reference, a z17 ME1 can have up to 85 LPARs, and a Parallel Sysplex up to 32 members. I rarely see a customer approach 85 on a machine and I’ve never seen a 32-way Parallel Sysplex.

However, I am seeing increasingly complex environments – both from the point of view of machines having more LPARs and Parallel Sysplexes having more members.

And, of course, Db2 was brand new – and now look at how complex many customers’ Db2 estates have become. A dozen or more members in a Datasharing Group is not unheard of, and multiple Datasharing Groups is very common. (And it’s not just Development vs Production.)

So How Do You Handle Such Complexity?

That’s the important question. I’ll admit I’m slowly learning – by doing. I’d like to think I do each study better than the last – especially where the environments are complex.

What I don’t want to do is to say the same thing over and over again for each of the systems or Db2’s. I think I would at the machine level – as that’s relatively few times.

But I want any conversation to flow nicely – for all concerned.

There are a couple of approaches – and I try to do both:

  • Establish Commonality
  • Discern Differences

So let’s talk about them.

Establish Commonality

If you have 10 systems they might well have things in common. For example, they might have Db2 subsystems in the same Datasharing Group. Or cloned CICS regions.

There might be symmetry between LPARs on a pair of machines. This is very common – though asymmetry tends to creep in, particularly with older systems.

By finding commonality and symmetry it’s possible to tell the tale with economy of effort and reduced repetition.

Discern Differences

But symmetry might be broken and often Parallel Sysplexes are pulled together from disparate systems. This was particularly so in the early days – to take advantage of Parallel Sysplex License Charge, as much as anything.

Nowadays I’m seeing a growth in differentiated systems within a Parallel Sysplex. Thankfully I’m seeing pairs or quartets of such systems. Examples include:

  • DDF workloads, with their own Db2 subsystems alongside CICS or IMS systems. (Alongside meaning sharing data.)
  • Different CICS applications. Quite common is the “Banking Channel” model.

So “spot the difference” is a good game to play.

The Importance Of Good Tools

Good tools enable me to see, among other things:

  • Architectural structure
  • Differences in behaviour

This post is not to boast about the quality of my tools – as most of them are in no fit state to sell or give away.

Further, I wouldn’t say my tools are perfect. Which is why I have to maintain a posture of continual improvement. You’ll see an example of that later on.

Architecture

Over the years I’ve taught my tools to produce architectural artefacts, such as which Db2 subsystems on which LPARs are members of which Datasharing Group. Further, a view of each Db2 is accessed by what. Likewise, which LPARs have which Service Classes and Report Classes being used.

Very recently I’ve got interested in the proliferation of TCP/IP stacks – which I see as different address spaces, plus Coupling Facility structures.

Right now my nursery of “interesting address spaces” is growing.

Difference

You probably wondered about the title of this post.

Seeing differences in behaviours between supposedly similar things can be instructive.

Take this graph. It’s brand new – and the only editing is removing system names and the title.

A few notes on how to read the graph:

  • Each series is for a different system.
  • Each data point is for a different RMF interval.
  • It shows how the velocity of a service class period varies with the GCP CPU used.
  • The green datum line is the goal velocity for the service class period. (If it varies it’s suppressed.)

You might have seen something like this before – but then each series would’ve been for a different day, not a different system. The question I’m solving with this one is “do all the systems behave the same?” rather than “does this system behave the same way every day?”

(The idea of plotting a three-dimensional graph where the two horizontal axes are GCP CPU used and zIIP CPU used had occurred to me – but I consider it problematic both presentationally and technically. Maybe I’ll experiment one day. And I did try out a 3D column chart in a recent engagement.)

But what does it show?

I see a number of things (and you might see others):

  • Two of these systems perform worse than the others. Hence the blog post title.
  • These two systems perform worse for the same sized workload.
  • These two systems have – much of the time – much more CPU consumption.
  • Even the better-performing systems struggle to meet goal.
  • You could argue all the systems scale quite nicely – as their velocity doesn’t drop much with increasing load.

With such a systematic difference you have to wonder why. A couple of thoughts occur:

  • System conditions might be different for these two systems. They are in fact larger LPARs – with lots of other things going on.
  • These two systems might be processing different work in the same service class. (I’m not going to say “period” anymore as these is a single period service class.) This is indeed a “Banking Channel” customer.

I’ve encouraged customers to judiciously reduce the number of service classes. The word “judiciously” is doing a lot of heavy lifting in that sentence. This might be a case where an additional service class is needed.

Still, vive la difference! It certainly shows the value of this graph.

One final point: The graph is for a velocity goal. Doing something similar for response time goals might be a bit more fiddly.

Here we have two subtypes: Average and Percentile variants. Compared to Velocity. So that’s two more graphs to teach my code to construct. If I only want one it’d have to be Performance Index that is plotted – but that’s too abstracted, I feel. Perhaps I’ll experiment with this – probably in early 2026.

Conclusion

It is possible to tell the story of more complex environments in a relatively succinct way – and thus make discussions more consumable. But it takes some thought – and some code.

And my storytelling continues to evolve – which helps me want to keep doing this.

Making Of

This post started out as wanting to show off that graph. While I do like it a lot my thoughts went a lot wider in writing this. And I had the time for them to go wider as I’m on a flight to Istanbul, to meet with a couple of my regular customers.

I was going to try my handwriting out again but somehow I lost the tip of the Apple Pencil on the plane before I got started. I did find it on landing – so all good now.

I still think some automation in my writing tool – Drafts – could help tidy up what I wrote. I’ll have to think about that. That’s probably a good thing to play with on my flight home. Javascript at 35,000 feet.

Modern Machines, Modern Metrics

Modern Machines, Modern Metrics

In z17 Sustainability Metrics – Part 0 I wrote about the new z17 Sustainability Metrics, or “Power Consumption”, if you prefer.

This post isn’t part 1 – as I don’t intend to go into much detail about what I’ve learnt so far. I have learnt things, of course.

During the Summer it occurred to me that there are a number of things that are new in z/OS instrumentation on z16 and z17. Regular readers will know that I like to write about them.

I suggested to my friend John Baker of the IntelliMagic team that we could do a conference presentation about them. He readily accepted. (John has about the same level of interest in such things as I do – and sees different customers soour collective experience base is wider.)

And so Modern Machines, Modern Metrics was born.

As I alluded to “modern” is both z16 and z17. At least for now.

So what does “M4”1 cover?

It starts with a pair of z17 topics:

  • Sustainability Metrics
  • DPU ( Data Processing Unit) or “I/O Engine” as some of us like to call it

Then we talk about a couple of z16 topics.

As l write this, they are:

  • Home Addresses
  • AIU (Artificial Intelligence Unit)

I say “as I write this” because, as with all my presentations, this one will evolve – as we gain experience. We’re expecting the z17 topics to gradually crowd out the z16 ones.

I think it would be ambitious to hope to schedule this as a two parter, but you never know – and we’re still learning things about how z16 metrics behave. In fact older metrics can still spring surprises.

We gave this presentation a few weeks ago at GS UK Annual Conference. It went very well, I think. We’re about to give the first topic in a meeting with the Development team for Sustainability Metrics. I think they’ll find it interesting. We do have some questions to ask them, around data sources and interpretation – but I think you’d expect that.

And I’d like to think we’d evolve this for future machines.

Making Of

I’m writing this on a plane back from Madrid, mostly using an Apple Pencil Pro. I seemed to have it tamed. But then what is euphemistically called “light chop” 😊 intervened. I was doing well, really I was. 😊 The Bay Of Biscay is a harsh mistress.2

This is a new keyboard on the iPad Mini. More expensive than the previous one but it’s well worth the money.

But, again, some exotic (not really in my opinion) keys require strange finger acrobatics.

One cute thing is the keyboard can be made to light up – in different colours. And cycle between them. 😊

  1. M4 is of course the name of a motorway, running west from London. But it’s a handy shorthand. ↩

  2. Hopefully some of you will get the reference. 😊 ↩

md2pptx 6 Is Another Big Step Forward

If the purpose of Version 5 was to add the ability to use Python in the context of md2pptx the purpose of version 6 is to extend md2pptx’s automation capabilities still further.

This time instead of Python it’s AppleScript. The context is different too:

  • With Version 5 the Python support runs the user-supplied code as part of the main md2pptx run – before the PowerPoint presentation is saved.
  • With Version 6 the AppleScript code is generated by md2pptx, saved, and then run – after the PowerPoint presentation has been saved.

But why do it this way?

The answer is that it enables some things that md2pptx can’t do – because the python-pptx package it builds on can’t enable them. And AppleScript – as a postprocessor – can.

It’s unfortunate that this is AppleScript only – restricting the function to Macs. That’s because I don’t have a Windows machine to test or develop Powershell scripts on. I might stand more of a chance with Linux – and something like Open Office. That’s because I have Raspberry Pi’s galore.

So, what can you do with the new support? As I write this you can – with v6.1:

  • Reopen the presentation in PowerPoint to a specific slide
  • Insert slides from other presentations
  • Run your own arbitrary AppleScript

Each of these has their own motivation:

  • When I rebuild a presentation I often want to reopen it in PowerPoint at a specific slide. So I created a bookmarking capability.
  • Users of python-pptx often want to embed slides from other presentations – so I built this more for them than me. But I actually do have standard educational slides in my workshop presentations – so I might well use it myself. A prime example of this is my WLM presentations: I start by explaining terms such as Performance Index and Velocity. I include some optional topics – which often become live – at the end.
  • As with Python for Version 5, I often want to experiment with AppleScript – so I made that easy to do. I also think some adventurous users will write their own.

As with Python, there is an obvious health warning: Running scripts you didn’t inspect can be dangerous. Any I ship will readable and well commented. If not that’s a defect and can be raised as an issue. As can any coding inefficiencies.

I actually ship an AppleScript file that contains a routine to copy slides from another presentation. I plant calls to it – if the user requests md2pptx to do so.

One other limitation of python-pptx is it can’t emulate the PowerPoint layout engine; By opening a presentation, navigating to a slide (or maybe through all slides), and then saving it, AppleScript could force PowerPoint to lay out the slide, text and all. I don’t know how useful it would be – but people have complained of such things. So I’ll have to experiment with this. And now I can.

The net of this is Version 6 opens up yet more automation possibilities for creating PowerPoint presentations.

One final thought: I prefer to add capabilities in Python rather than AppleScript. Further, I would prefer people not to have to use RunPython but rather use Markdown or metadata semantics. This is more user friendly and more widely applicable.