Periodicity

When I examine Workload Manager for a customer a key aspect is goal setting. This has a number of aspects:

  1. How work gets classified – to which Service Class and Report Class by what rules.
  2. What the goals are for each Service Class Period.
  3. What the period boundaries should be.

This post focuses on Aspect 3: Period Boundaries.

What Is A Service Class Period?

When a transaction executes it accumulates service. Generally this is CPU time, especially with modern Service Definition Coefficients.

For transactions that support it you can define multiple Service Class Periods. Each period – except the last – has a duration.

Some transaction types, most notably CICS, only have a single period. For them the discussion of period durations is moot.

The z/OS component that monitors service consumption is System Resources Manager (SRM). SRM predates Workload Manager (WLM) by decades. (It’s important not to see WLM as replacing SRM but rather as supervising it. WLM replaces human-written controls for SRM.) Periodically SRM checks work’s consumption of resources. If the transaction has exceeded the relevant period duration the transaction moves to the next period.

It isn’t the case that a transaction using more service than its current period’s duration directly triggers period switch – so it would be normal for a (generally) slight exceeding as there is some slight latency to detection.

The purpose of multiple periods is, of course, to give good service to light consumers of service and to progressively slow down heavier consumers.

Note: A common mistake is to think that transactions fall through into later periods because of their elapsed time. They don’t; It’s about service. Granted, a long running transaction might be long running because of the CPU it’s burning. But that’s not the same thing as saying it’s the elapsed time that drove it to later periods.

Two Examples Of A New Graph

Here are two example graphs from the same customer. They are new in our code base, though Service Class Period ending proportions are something we’ve talked to customers about for many years. I’m pleased we have these as I think they will tell some interesting new stories. You’ll get hints of what I think those stories might be based on the two examples from my “guinea pig” customer.

Each graph plots transaction ending rates for each period of the Service Class across a day. In the heading is information about period boundaries and how many service units the transactions ending there consumed on average. I feel the usefulness of that latter will emerge with more experience – and I might write about it then. (And graph headings is one place my code has a high tendency to evolve, based on experiences with customers.)

Though the two examples are DDF I don’t intend to talk much about Db2 DDF Analysis Tool – except to say, used right, it would bring some clarity to the two examples.

DDFMED – A Conventional-Looking Service Class

This Service Class looks like many Production transaction service classes – with the classic “double hump” shape. I consider that an interesting – if extremely common – architectural fact. There’s something about this workload that looks reminiscent of, say, CICS transactions.

Quite a high proportion of the transactions end in Period 2 and a fair proportion in Period 3. Those in Period 3 are, on average, very heavy indeed – consuming an average of 162K service units. (This being DDF, the transaction ends when the work is committed – which might not be the end of the transaction from the client’s point of view.)

It seems to me the period boundaries are reasonable in this case, but see “Conclusion” below.

DDFLOW

This Service Class looks quite different:

  • The transaction rate is more or less constant – with two spikes, twelve hours apart. I consider both the constant transaction rate and the twelve-hourly spikes to be interesting architectural facts.
  • Almost all transactions end in Period 1. In fact well within Period 1. The very few Period 3 transactions are extremely long.

Despite the name “DDF Low” I think we have something very regular and well controlled here. I say “despite” as, generally, less well understood / sponsored work tends to be thought of as “low”.

Conclusion

I will comment that, when it comes to goal setting, business considerations play a big part. For example, some of the effects we might see at the technical level could be precisely what it needed. Or precisely what is not needed. So I tend not to walk in with recommendations for things like transaction goals – but I might walk out with them. Contrast this with what I call my “Model Policy” – which I discussed in Analysing A WLM Policy – Part 1 in 2013. Core bits of that are as close to non-negotiable as I get.

However, it is – as I think this post shows – very useful in discussions of period durations to know the proportions of transactions for a Service Class that end in each period. If everything falls through into Period 2, for example, Period 1’s duration is probably too short. And not just the proportions but the transaction rates across, say, the day.

One other thing, which I’ll leave as a question: What happens if you slow down a transaction that, say, holds lots of locks?

Four Coupling Facilities

This isn’t following on from Three Billboards? 🙂 but rather Shared Coupling Facility CPU And DYNDISP from 2016. I’m not sure it adds much that’s new but this set of customer data was an opportunity too good to miss…

… It enables me to graph most of the Coupling Facility LPAR types you’re likely to see.

I won’t repeat the contents of the 2016 post but I will repeat one thing: There are two views of Coupling Facility CPU:

  • SMF 70 Partition
  • SMF 74–4 Coupling Facility

That 2016 post talked at length about the latter. This post is more about the former.

Different Coupling Facility LPAR Types

There are different kinds of coupling facility LPAR, and this customer has several of them:

  • Dedicated
  • Shared – without Dynamic Dispatch (DYNDISP=NO)
  • Shared – with Dynamic Dispatch (DYNDISP=YES)
  • Shared – with DYNDISP=THIN

The latter two are similar but, in essence, Thin Interrupts (DYNDISP=THIN) shortens the time a CF spends polling for work, releasing the physical CPU sooner. This is good for other ICF LPARs, but maybe not so good for the LPAR with DYNDISP=THIN.

While this customer’s data doesn’t exemplify all four types it is a useful set of data for illustrating some dynamics.

About This Customer’s Mainframe Estate

I’m only going to describe the relevant bits of the customer’s mainframe estate – and I’m going to remove the names.

There are four machines, each running a mixture of LPARs in sysplexes and monoplexes. The sysplex we were most interested in had four LPARs, one on each machine. Also four coupling facilities, again one on each machine. There were no external coupling facilities.

Those of you who know a bit about resilience are probably wondering about duplexing coupling facility structures but this post isn’t about that.

I don’t think it makes any difference but these are a mix of z13 and z14 machines.

We had SMF 70-1 and SMF 74-4 from these z/OS LPARs and the four coupling facilities, but little from the others.

Here are the four machines’ ICF processor pools, across the course of a day.

The top two look significantly different to the bottom two, don’t they?

Machines A And B

These two machines have multiple ICF LPARs, each with some kind of DYNDISP turned on. We can see that because they don’t use the whole of their PR/SM shares – as their utilisation from the PR/SM point of view is varying.

Each machine has two shared ICF processors.

We have SMF 74-4 for the blue LPARs. So we can see they are using DYNDISP=THIN. We can’t see this for the other LPARs as we don’t have SMF 74-4 for them. (SMF 70-1 doesn’t have a DYNDISP indicator.)

The blue LPARs are also much busier than the other LPARs in their pool. While one might consider dedicating one of the two ICF processors we wouldn’t ideally define an ICF LPAR with a single logical processor.

Machine C

Machine C looks different, doesn’t it?

Again we have 2 physical processors in the ICF pool.

Here we have four ICF LPARs, each with DYNDISP=NO. We know three facts to establish this:

  • From SMF 70-1 we know that none of the LPARs has any dedicated ICF engines. We know this two ways:
    • As this graph shows, none of them has the share of a single ICF engine.
    • We have Dedicated and Shared processor information explicitly in SMF 70-1.
  • These LPARs use their whole share – judging by the constant CPU use in SMF 70-1.
  • From SMF 74-4 we know the blue LPAR in particular has DYNDISP=NO. (We don’t have SMF 74-4 for the others.)

Machine D

This looks similar to Machine C but it isn’t quite.

Yet again the ICF pool has 2 processors.

  • The Red LPAR has dedicated processors – from both SMF 70-1 and SMF 74-4. DYNDISP doesn’t even come into it for this LPAR.
  • The other LPARs have DYNDISP=NO, judging by their (SMF 70-1) behaviour.

A minor footnote: As I articulated in A Picture Of Dedication (in 2015) I sort the LPARs so the Dedicated ones appear at the bottom of the stack up. (Even below *PHYSICAL – which is a very thin blue veneer here but actually red in the case of the other three machines.)

But Wait, There’s More

When I started writing this post I thought it was just going to be about showing you four pretty pictures. But, partially because some blog posts get written over an extended period, something came up meanwhile that I think is worth sharing with you. Besides, electronic words are unlimited – even if your patience isn’t.

Having shown you some graphs that depict most of the ICF LPAR CPU situations I experimented with DYNDISP detection for ICF LPARs we don’t have SMF 74-4 for.

Got right, this could make the story of an ICF pool with shared logical engines much nearer to complete.

The current algorithm – now in our Production code – assumes DYNDISP if the ICF LPAR uses less than 95% of its share over the focus shift (set of hours). Otherwise it’s Dedicated or DYNDISP=NO. I still can’t tell whether it’s DYNDISP=THIN or YES without SMF 74-4.

While ideally I still want data from all z/OS LPARs and coupling facilities in a customer’s estate, this technique fills in some gaps quite nicely.

Well, it works for this customer, anyway… 🙂

Three Billboards?

You could consider presentations as advertising – either for what you’ve done or for what something can do. Generally my presentations are in the category of “Public Service Announcement”. Which rather combines the two:

  • What I’ve found I can do – that you might want to replicate.
  • What I’ve found a product can do – that you might want to take advantage of.

Sometimes it’s a “Public Health Warning” as in “you’d better be careful…”

Anyhow, enough of trying to justify the title of this post. 🙂

(I will say Three Billboards Outside Ebbing, Missouri is an excellent movie. I first saw it on an aeroplane on the usual tiny screen and it worked well even there.)

So, I have three presentations, two of which are brand new and the other significantly updated this year:

  • What’s The Use? With Scott Ballentine
  • zIIP Capacity And Performance
  • Two Useful Open Source Tools

I’ll post the abstracts below.

The first two are for IBM Technical University Virtual Edition (registration here). This is run by IBM and is premier technical training, including over 600 sessions on IBM IT Infrastructure.

As I’m a big fan of user groups I’m delighted to say the first and third are for GSE UK Virtual Conference 2021. In both cases these are “live” – which, frankly, I find more energising.

Scott and I recorded last week – as the Tech U sessions will be pre-recorded with live Question And Answer sessions at the end. So these are “in the can” – which is a big relief.

The third I’ve just started writing but I have a nice structure to it – so I’m sure it’ll get done, too.

So here are the abstracts…

What’s The Use – With Scott Ballentine

For many customers, collecting the SMF type 89 subtype 1 Usage Data records are an important and necessary part of Software Licencing, as they are used by SCRT to generate sub-capacity pricing reports.

But Usage Data has many more uses than that – whether in SMF 89 or SMF 30.

Customers can get lots of value out of this data – if they understand what the data means, and how it is produced.

Vendors can delight their customers – if they produce the right usage data, knowing how it can be used.

This presentation describes how Usage Data is produced, how a vendor can add value with it, and how customers can best take advantage of it.

zIIP Capacity And Performance

zIIP Capacity Planning tends to be neglected – in favour of General-Purpose Engines (GCPs). With Db2 allowing you to offload critical CPU to zIIPs, and the advent of zCX and z15 Recovery Boosts, it’s time to take zIIP capacity and performance seriously.

You will learn how to do zIIP capacity planning and performance tuning properly – with instrumentation and guidelines.

Two Useful Open Source Tools

You will learn how to use two open source tools many installations will find extremely useful, covering System and Db2 Performance:

Db2 DDF Analysis Tool – which uses Db2 SMF 101 Accounting Trace to enable you to manage DDF work and its impact on your system.

WLM Service Definition Formatter – which uses your WLM XML file to create some nice HTML, enabling you to see how the various parts of your Service Definition fit together.

Time For Action On Virtual Storage?

I wrote How I Look At Virtual Storage in 2014.

But since then my stance has shifted somewhat. A recent study I was part of made me realise my tone on z/OS virtual storage should probably be a little more strident. In that post I was somewhat even handed about the matter, just describing my method.

This study also involved the CICS Performance team. They pointed out that some of this customer’s CICS regions had substantial use of 24-Bit virtual storage. If those CICS regions were to be able to scale (individually) a limiting factor might be 24-Bit virtual storage.

Obviously working on CICS regions’ use of virtual storage is the correct way forward under these circumstances.

But it got me thinking, and doing a little investigating.

Why Virtual Storage Matters

Almost 35 years ago a sales rep in my branch suggested selling a customer more virtual storage. 🙂 (Yes, he really did.)

Joking apart, if you want more virtual storage you can’t buy it; You have to work for it.

But why would you want more 24-Bit virtual storage anyway?

  • Older applications might well be 24-Bit and converting them to 31- or even 64-Bit might be difficult or not economically viable.
  • Some things, such as access method buffers, might well be in 24-Bit virtual storage. (To be fair, high level languages’ support for buffers above the 16MB line is good.)

I raise the subject of access method buffers because one of the ways of tuning I/O performance is to increase the number of access method buffers. To optimally buffer a single QSAM data set, for example, takes a substantial proportion of an address space’s 24-Bit region – unless the buffers have moved above the 16MB line. So one is sparing of buffers under these circumstances, perhaps sacrificing some performance. (You would choose high I/O data sets to buffer, obviously.)

My own code – at least the bits I have the excuse of having inherited – is probably predominantly 24-Bit. I might fix that one day, though it doesn’t seem like a good use of time. (It would probably be coupled with removing SIIS (Store Into Instruction Stream) horrors.)

What’s New?

When my CICS colleagues mentioned managing the use of 24-Bit virtual storage within the CICS regions a thought occurred to me: I could examine whether the 24-Bit region size could be increased. Both are useful approaches – reducing the demand and increasing the supply.

The same is probably true of 31-Bit, of course. For 64-Bit I’m more concerned with getting MEMLIMIT set right.

I think I’ve observed something of a trend: It seems to me many customers have the opportunity to increase their 24-Bit region – probably by 1MB but sometimes by 2MB. This would be a handy 10 – 20% increase. I doubt many customers have examined this question in a while – though most have SMF 78-2 enabled all the time. It’s rare to get a set of customer data without it in – and we always pump out the report outlined in How I Look At Virtual Storage.

Examining And Adjusting Virtual Storage

In what follows I’m describing 24-Bit. Similar actions apply for 31-Bit.

  1. Check that your SQA definition is small enough that there is little or no SQA free. SQA can overflow into CSA but not vice versa – so overspecifying SQA is a waste of virtual storage.
  2. Check how much CSA is free.
  3. Adjust SQA and CSA so that there is generally 1MB CSA free but little if any SQA free, but so that you are raising the private boundary by an integer number of MB – probably 1MB, possibly 2MB.
  4. Monitor the resulting 24-bit virtual storage picture.

You need to do the analysis with data over a reasonable amount of time. I’d say at least a week. You want to know if SQA and CSA usage are volatile. In most customer situations I’ve seen they are relatively static, but you never know.

One of the key points is that the boundary between common and private is a segment boundary i.e. 1MB. There is therefore no point trying to claw back, say, half a megabyte. (The “generally keep 1MB free” above is not related to this; It’s just a reasonable buffer.)

For 24-Bit I said “adjust SQA and CSA so that there is generally 1MB free”. For 31-Bit I’ve generally said “keep 100MB free”. I think that’s fair but 200MB might be better. Most middleware and applications that use Below The Bar virtual storage have substantial amounts of 31-Bit. If their use is volatile – again when viewed over a reasonable period of time – then that might guide you towards 200MB free or more. The “100MB” is not structural; That’s 100 segments and is just a reasonable rule of thumb. At least for 31-Bit the 1MB segment boundary doesn’t seem nearly so coarse grained as it does for 24 Bit, so adjustment needn’t be in 100MB increments.

Talking of 31-Bit virtual storage, one of the biggest users of ECSA usage has been IMS. Until a few years ago the biggest users of 31-Bit private were Db2 and MQ. Db2 is now almost all 64-Bit and MQ has done lots of 64-Bit work.

Conclusion

I would urge customers to examine their virtual storage position. In particular, 24-Bit might well contain good news. 31-Bit might also, but there I’m more inclined to check there’s enough left. It’s not difficult to examine this, even if all you do is run off an RMF Virtual Storage Activity report.

I, for one, will make a point of examining the virtual storage reports we produce. Generally I do, anyway. Else I wouldn’t be seeing this trend.

Finally, a sobering thought: If, as has been said, customers are using 1 bit of addressability a year we are 21 years into a 33-year 64-Bit lifespan. I’m pre-announcing nothing by saying this. And the “1 bit a year” thing might not actually hold. But it does make you think, doesn’t it.

Another Slice Of PI

This post follows on from my 2019 post A Slice Of PI. Again, not Raspberry Pi but Performance Index PI. 🙂

I’ve become sensitised to how common a particular Workload Manager (WLM) problem is since I created some handy graphs.

Some Graphs Recent Customers Have Seen

When I examine a customer’s WLM Service Definition part of what I do is examining SYSTEM Workload, and then successive importances.

(SYSTEM Workload is mainly SYSTEM and SYSSTC Service Classes, of course. And if these are showing a drop off in velocity you can imagine what is happening to less important work. This is when I consider Delay For CPU samples – to figure out how much stress the system is under. But I digress.)

By “successive importances” I mean:

  1. Graph importance 1 Performance Index (PI) by Service Class by hour.
  2. Graph importance 2 Performance Index (PI) by Service Class by hour.

And so on.

Sometimes there is little work at eg Importance 1, so I might stop after Importance 2 or I might not. (Service definitions vary in what is at each importance level – and I graph CPU for each importance level to understand this.)

The above approach gives me some orientation. And it has shown up a phenomenon that is more common than I had supposed: Percentile goals where the service class period ends up with a PI of 0.5.

The Top And Bottom Buckets

Recall the following from A Slice Of PI: For a Percentile Goal when an individual transaction ends its response time relative to the goal is used to increment the transaction count in one of 14 buckets. There are two buckets of especial interest:

  • Bucket 1 – where transactions whose response times are no more than 50% of the goal time are counted.
  • Bucket 14 – where transactions whose response time are at least 400% of the goal time are counted.

These buckets feed into the Performance Index (PI) calculation. Let’s deal with Bucket 14 first:

Imagine a goal where the goal is “85% in 15ms”. If fewer than 85% of transactions end with response times short enough to land them in Buckets 1 to 13 the PI is 4. We don’t have much clue as to quite how long the more than 15% in Bucket 14 took, but we know they all took at least 4 × 15 = 60ms. So we don’t know how much to adjust the goal by. (Of course, we might want to tune the transactions or the environment instead – and we don’t know how much to speed things up by.)

A note on terminology: I’m going to use the term goal percent for the “85%” in this example. I’m going to use the term goal time for the “15ms” part.

The Bottom Bucket

Bucket 14 was the “warm up act” for Bucket 1.

With our “85% in 15ms” goal a transaction ending in Bucket 1 has a response time of no more than 0.5 × 15 = 7.5ms. Again, we don’t know how close to the bucket boundary the transaction response times tend to be. Here are two possibilities (and there are, or course, myriad others):

This first graph shows a transaction distribution where the transactions have response times tending towards way shorter than 50% of the goal time.

This second graph shows a transaction distribution where the transactions have response times tending towards just short of the 50% of goal time marks.

Both could have a PI of 0.5.

So what should we adjust the goal time to? I think we should be prepared to be iterative about this:

  • Set the goal time to somewhat less than the current goal time. Maybe as aggressively as 50% of the current goal time – as that is safe.
  • Remeasure and adjust the goal time – whether up or down.
  • Iterate

I think we have to get used to iteration: Sometimes, from the performance data, the specialist doesn’t know the final value but does know the direction of travel. This is one of those cases, as is Velocity.

In most cases we should be done in a very few iterations. So don’t be unimpressed by a specialist having the maturity to recommend iteration.

An alternative might be to tighten the goal percent, say to 95%. This has a couple of issues:

  1. it tells us nothing about distribution.
  2. It is more or less problematic, depending on the homogeneity of the work. At the extreme, if all transactions have the same measured response time adjusting the percent is a blunt instrument.

Problem 2 is, of course, more serious than Problem 1.

Of Really Short Response Times

Really short response times are interesting: Things being really fast is generally a good thing, but not always.

Consider the case of JDBC work where Autocommit is in effect. Here every single trivial SQL statement leads to a Db2 Commit and hence a transaction ending. This can lead to extremely short response times. More to the point, it’s probably very inefficient.

(You can observe this sort of thing with Db2 DDF Analysis Tool and Db2 Accounting Trace (SMF 101).)

But transactions have become faster. It’s not uncommon for goals such as our example (85% in 15 ms) to be lax.

Faster against the expectations of WLM goal setters. In the DDF case it’s quite likely such people won’t have been told what to expect – either in terms of performance or how to classify the work. Or, fundamentally, what the work is.

I see a lot of DDF Service Classes with 15ms for the goal time. For many years you couldn’t set it lower than that. Relatively recently, though, the lower limit was changed to 1ms. I’m not suggesting a transaction response time of 1ms is that common, but single digit is increasingly so.

So this 1ms limit stands to fix a lot of goal setting problems – both for percentile and average response time goal types.

The analogy that comes to mind is 64-Bit: We don’t – for now – need all 64 bits of addressability. But we certainly needed many more than 31. We might not need 1ms right now but we needed considerably less than 15ms. (I would guess some technological improvement meant this became feasible; I really should ask someone.)

Why Is A PI Of 0.5 A Problem?

You would think that doing better than goal would be a good thing, right?

Well, not necessarily:

  1. Not what was contracted for
  2. Not protective

Between the two of them there is a Service Delivery hazard: When things slow down an overly lax goal won’t stop this work from slowing down. And this is where Service Level Expectation – what we’ve delivered so far – clashes with Service Level Agreement – what the users are entitled to.

I often like, when we have a little time in a customer workshop, to ask where the goal come from. Perhaps from a contract, whether internal or external. Or maybe it’s the originally measured response time. Or maybe it’s just a “Finger in the air”. This all feeds into how appropriate the goal now is, and what to do about it.

Walking This Back A Tiny Bit

It’s not hopeless to figure out what’s going on if everything ends in Bucket 1, with a PI of 0.5. You can – again from Workload Activity Report (SMF 72-3) data – calculate the average response time. If the average is well below the “50% of goal time” mark that tells you something. Equally “not much below” tells you something else.

So, I think you should calculate the average, even though it’s not that useful for goal setting.

Conclusion

You probably know I’m going to say this next sentence before I say it: Revisit your WLM Service Definitions – both goal values and classification rules – on a regular basis.

And, now that I have PI graphs for each importance level, I’m seeing a lot of “PI – 0.5” situations. Hence my concentrating on the bottom bucket.

One final thing: With Db2 DDF Analysis Tool I am able to do my own bucketing of DDF response times. I can use whatever boundaries for buckets I want, and as many as I want. This helps me – when working with customers – to do better than the “14 buckets”. Or it would if I weren’t lazy and only had 10 buckets. 🙂 I can do the same with CPU time, and often do.

Hopefully this post has given you food for thought about your percentile goals.

One Service Definition To Rule Them All?

This is one of those posts where I have to be careful about what I say – to not throw a customer under the bus with over-specificity. So, bear with me on that score.

Actually it’s two customers, not one. But they have the same challenge: Managing Workload Manager (WLM) service definitions for multiple sysplexes.

These two customers have come to different conclusions, for now. I say “for now” because, in at least one case, the whole design is being reconsidered.

Some Basics

Even for those of us who know them well, the basics are worth restating:

  • A sysplex can have one and only one service definition.
  • A service definition cannot be shared between sysplexes, but it can be copied between them. These are separate installations and activations.
  • In a sysplex different systems might not run work in all the service classes.
  • The ISPF WLM application has been used since the beginning to edit service definitions.
  • You can print the service definition from the ISPF WLM application.
  • Relatively recently the z/OSMF WLM application became available – as a more modern (and more strategic) way of editing service definitions.
  • You can export and import the service definition as XML – in both the ISPF and z/OSMF applications.

What I’ve Been Doing Recently With WLM Service Definitions

For many years I relied entirely on RMF Workload Activity SMF records (SMF 72-3) when tuning WLM implementations. This simply wasn’t (good) enough, so a fair number of years ago I started taking WLM service definitions in XML form from customers when working with them.

Why wasn’t RMF SMF good enough? Let’s review what it buys you:

  • It tells you how service class periods behave, relative to their goals and with varying workload levels.
  • It allows you to examine report class data, which often is a good substitute for SMF 30 address space information.

So I’ve done a lot of good work with SMF 72-3, much of which has surfaced as blog posts and in conference presentations. So have many others.

But it doesn’t tell you what work is in each service class, nor how it got there. Likewise with report classes. Admittedly, SMF 30 will tell you an address space’s service and report classes, but it won’t tell you why. Likewise, SMF 101 for DDF work will give you a service class (field QWACWLME) but it won’t tell you why, and it won’t tell you anything about report classes. And SMF won’t tell you about history, always a fascinating topic.

To understand the structure of the service definition you need the XML (or the print, but I don’t like the print version). So that’s what I’ve been looking at, increasingly keenly.

A Tale Of Two Customers

  • Customer A has multiple sysplexes – on two machines. They sent me multiple XML service definitions – as they have one per sysplex.
  • Customer B has multiple sysplexes – on more than two machines. They sent me a single XML service definition – so they are propagating a single service definition to multiple sysplexes.

As it happens, in both customers, there are application similarities between some of their sysplexes. In Customer B there are different application styles between the LPARs in a single sysplex – and this fact will become important later in this discussion.

So, which is right? The rest of this post is dedicated to discussing the pros and cons of each approach.

Multiple Service Definitions Versus A Single Propagated One

The first thing to say is if you have an approach that works now think carefully before changing it. It’ll be a lot of work. “A lot of work” is code for “opportunity for error” rather than “I’m too lazy”.

If you have one service definition for each sysplex you might have to make the same change multiple times. But this would have to be an “egregious” change – where it’s applicable to all the sysplexes. An example of this would be where you’d misclassified IRLM address spaces and now needed to move them all into SYSSTC (where they belong). There’s nothing controversial or specific about this one, and not much “it depends” to be had.

For changes that are only relevant or appropriate to one sysplex this is fine. For example, changing the duration of DDFHI Period 1 for a Production sysplex.

But if you had a DDFHI in each of several sysplexes this could become confusing – if each ended up with a different specification. Confusing but manageable.

If you had one service definition for a group of sysplexes you’d make a change in one sysplex and somehow propagate that change around. You’d use literally the same XML file (assuming you go the XML route), export it from the first sysplex, and import it into the rest. The mechanism, though, is less important than the implications.

Any change has to be “one size fits all”. Fine for the IRLM example above, but maybe not so good for eg a CICS Region Service Class velocity change. In the “not so good” case you’d have to evaluate the change for all the sysplexes and only if it were good (enough) for all of them make the change.

But, you know, “one size fits all” is a problem even within a sysplex: Customer B has different applications – at quite a coarse-grained level – in different LPARs within the same sysplex:

  • The common componentry in the LPARs – e.g Db2 or MQ – probably warrants the same goals.
  • The application-specific, or transaction management, componentry – e.g. Broker, CICS, IMS, Batch – probably doesn’t.

There are some ways of being “differential” within a service definition. They include:

  • Treating one transaction manager differently from another.
  • Classification rules that are based on system.
  • Classification rules that are based on sysplex.

That latter – classification rules that are based on sysplex – is something I addressed very recently in my sd2html WLM Service Definition XML Formatter open source project. If you run it there’s a “Service Definition Statistics” table near the top of the HTML it produces. If there are classification rules based on sysplex name the specific sysplexes are listed towards the bottom of the table. This will save me time when looking at a customer’s service definition. You might use it , for example, to check whether such rules are still in place – when you thought they’d been removed. The specific use of the “Sysplex Name” qualifier will appear in either the Classification Groups or Classification Rules table, depending.

(I could use a good XML comparer, by the way. Because today I can’t easily tell the differences between Service Definition XML files – and I don’t think I should be teaching sd2html to do it for me.)

Conclusion

There is no conclusion, or at least no general firm conclusion; You really have to think it through, based on your own circumstances. Most notably two things:

  • Whether “one size fits all” works for you.
  • Whether operational procedures in WLM service definition management are effective.

But, still, it’s an interesting question. And it’s not hypothetical:

  • Plenty of customers have separate Sysprog, Development, Pre-Production, and Production sysplexes. In fact I’d encourage that.
  • Quite a few customers have separate Production sysplexes, especially outsourcers.

And this is why I find “Architecture” such fun. And why WLM is anything but simple – if you do it right.

Mainframe Performance Topics Podcast Episode 29 “Hello Trello”

This was a fun episode to make, not least because it featured as a guest Miroslava Barahona Rossi. She’s one of my mentees from Brasil and she’s very good.

We made it over a period of a few weeks, fitting around unusually busy “day job” schedules for Marna and I.

And this one is one of the longest we’ve done.

Anyhow, we hope you enjoy it. And that “commute length” can be meaningful to more of you very soon.

Episode 29 “Hello Trello” long show notes.

This episode is about how to be a better z/OS installation specialist, z/OS capture ratios, and a discussion on using Trello. We have a special guest joining us for the performance topic, Miroslava Barahona Rossi.

Follow up to Waze topic in Episode 8 in 2016

  • Apple Maps Adds Accident, Hazard, and Speed Check Reporting using the iPhone, CarPlay, and Siri to iOS.

What’s New

  • Check out the LinkedIn article on the IBM server changing for FTPS users for software electronic delivery on April 30, 2021, from using TLS 1.0 and 1.1 to using TLS 1.2, with a dependency on AT-TLS.

  • If you are using HTTPS, you are not affected! This is recommended.

Mainframe – Being a Better Installation Specialist

  • This section was modeled after Martin’s “How to be a better Performance Specialist” presentation. It’s a personal view of some ideas to apply to the discipline.

  • “Better” here means lessons learned, not competing with people. “Installation Specialist” might just as well have used “System Programmer” or another term.

  • There’s definitely a reason to be optimistic about being a person that does that Installation Specialist or System Programmer type of work:

  • There will always be a need for someone to know how systems are put together, how they are configured, how they are upgraded, and how they are serviced…how to just get things to run.

    • And other people that will want them to run faster and have all resources they need for whatever is thrown at them. Performance Specialists and Installation Specialists complement each other.

  • Consider the new function adoption needs too. There are so many “new” functions that are just waiting to be used.

    • We could put new functions into two catagories:

      1. Make life easiesr: Health Checker, z/OSMF, SMP/E for service retrieval

      2. Enable other kinds of workload: python, jupyter notebooks, docker containers

    • A good Installation Specialist would shine in identifying and enabling new function.

      • Now more than ever, with Continuous Delivery across the entire z/OS stack.

      • Beyond “change the process of upgrading”, into “driving new function usage”.

      • Although we know that some customers are just trying to stay current. Merely upgrading could bring new function activation out of the box, like SRB on z15.

    • A really good installation specialist would take information they have and use it in different ways.

      • Looking at the SMP/E CSIs with a simple program in C, for example, to find fixes installed between two dates.

      • Associating that with an incident that just happened, by narrowing it down to a specific element.

      • Using z/OSMF to do a cross-GLOBAL zone query capability, for seeing if a PE was present within the entire enterprise quickly.

      • Knowing what the right tool is for the right need.

    • Knowing parmilb really well. Removing unused members and statements. Getting away from hard-coding defaults – which can be hard, but sometimes can be easier (because some components tell you if you are using defaults).

      • Using the DISPLAY command to immediately find necessary information.

      • Knowing the z/OS UNIX health check that can compare active values with the hardened used parmlib member.

    • Researching End Of Service got immensely easier with z/OSMF.

    • Looking into the history of the systems, with evidence of merging two shops.

      • Could leave around two “sets” of parmlibs, proclibs. Which might be hard to track, depending on what you have such as ISPF statistics or comments. Change management systems can help.

      • Might see LPAR names and CICS region naming conventions

    • Might modern tools such as z/OSMF provide an opportunity to rework the way things are done?

      • Yes, but often more function might be needed to completely replace an existing tool…or not.

    • You can’t be better by only doing things yourself, no one can know everything. You’ve got to work with others who are specialists in their own area.

      • Performance folks often have to work with System Programmers, for instance. Storage, Security, Networking, Applications the list goes on.

      • Examples are with zBNA, memory and zIIP controls for zCX, and estate planning.

    • Use your co-workers to learn from. And teach then what you know too. And in forums like IBM-MAIN, conferences, user groups.

    • Last but not least, learn how to teach yourself. Know where to find answers (and that doesn’t mean asking people!). Learn how to try out something on a sandbox.

Performance – Capture Ratio

  • Our special guest is Miroslava Barahona Rossi, a technical expert who works with large Brasilian customers.

  • Capture Ratio – a ratio of workload CPU to system-level CPU as a percentage. Or, excluding operating system work from productive work.

    • RMF SMF 70-1 versus SMF 72-3
      • 70-1 is CPU at system and machine level
      • 72-3 is workload / service class / report class level
      • Not in an RMF report

  • Why isn’t the capture ratio 100%?

    • There wouldn’t be a fair way to attribute some kinds of CPU. For example I/O Interrupt handling.

  • Why do we care about capture ratio?

    • Commercial considerations when billing for uncaptured cycles. You might worry something is wrong if the capture ratio is low.

    • Might be an opportunity for tuning if below, say, 80%

  • What is a reasonable value?

    • Usually 80 – 90. Seeing more like 85 – 95 these days. It has been improved because more of I/O-related CPU is captured.

    • People worry about low capture ratios.

    • Also work is less I/O intensive, for example, because we buffer better

  • zIIP generally higher than GCP

  • Do we calculate blended GCP and zIIP? Yes, but also zIIP separately from GCP.

  • Why might a capture ratio be low?

    • Common: Low utilisation, Paging, High I/O rate.

    • Less common: Inefficient ACS routines, Fragmented storage pools, Account code verification, Affinity processing, Long internal queues, SLIP processing, GTF

  • Experiment correlating capture ratio with myriad things

    • One customer set of data with z13, where capture ratio varied significantly.

      • In spreadsheet calculated correlation between capture ratio and various other metrics. Used =CORREL(range, range) Excel function.

      • Good correlation is > 85%

      • Eliminate potential causes, one by one:

        • Paging, SIIS poor correlation

        • Low utilisation strong correlation

      • It has nothing much to do with machine generation. The same customer – from z9 to z13 – always had a low capture ratio.

        • It got a little bit better with newer z/OS releases

        • Workload mix? Batch versus transactional

      • All the other potential causes eliminated

      • Turned out to be LPAR complexity

        • 10 LPARs on 3 engines. Logical: Physical was 7:1, which was rather extreme.

        • Nothing much can be done about it – could merge LPARs. Architectural decision.

    • Lesson: worth doing it with your own data. Experiment with excluding data and various potential correlations.

  • Correlation is not causation. Look for the real mechanism, and eliminate causes one by one. Probably start with paging and low utilisation

  • Other kinds of Capture Ratio

    • Coupling Facility CPU: Always 100%, at low traffic CPU per request is inflated.

    • For CICS, SMF 30 versus SMF 110 Monitor Trace: Difference is management of the region on behalf of the transactions.

    • Think of a CICS region as running a small operating system. Not a scalable thing to record SMF 110 so generally this capture ratio is not tracked.

  • Summary

    • Don’t be upset if you get a capture ratio substantially lower than 100%. That’s normal.

    • Understand your normal. Be aware of the relationship of your normal to everybody else’s. But, be careful when making that comparison as it is very workload dependent.

    • Understand your data and causes. See if you can find a way of improving it. Keep track of the capture ratio over the months and years.

Topics – Hello Trello

  • Trello is based on the Kanban idea: boards, lists, and cards. Cards contain the data in paragraphs, checklist, pictures, etc.

  • Can move cards between lists, by dragging.

  • Templates are handy, and are used in creaing our podcast.

  • Power-ups add function. A popular one is Butler. Paying for them might be a key consideration.

  • Multiplatform plus web. Provided by Atlassian, which you might know from Confluence wiki software.

    • Atlassian also make Jira, whch is an Agile Project Management Tool.

  • Why are we talking about Trello?

    • We moved to it for high level podcast planning

      • One list per episode. List’s picture is what becomes the cover art.

      • Each card is a topic, except first one in the list is our checklist.

    • Template used for hatching new future episodes.

      • But, we still outlining the topic with iThoughts

    • We move cards around sometimes between episodes.

      • Recently more than ever, as we switched Episode 28 and 29, due to the z/OS V2.5 Preview Announce.

      • Right now planning two episodes at once.

  • Marna uses it to organise daily work, with personal workload management and calendaring. But it is not a meetings calendar.

    • Probably with Jira will be more useful than ever. We’ll see.

  • Martin uses it for team engagements, with four lists: Potential, Active, Dormant, Completed.

    • Engagement moves between lists as it progresses

    • Debate between one board per engagement and one for everything. Went with one board for everything because otherwise micromanagement sets in.

    • Github projects, which are somewhat dormant now, because of…

    • Interoperability

      • Github issues vs OmniFocus tasks vs Trello Cards. There is a Trello power up for Github, for issues, branches, commits, and pulls requests mapped into cards.

      • However, it is quite fragile , as we are not sure changes in state reliably reflected.

    • Three-legged stool is Martin’s problem, as he uses three tools to keep work in sync. Fragility in automation would be anybody’s problem.

      • iOS Shortcuts is a well built out model. For example, it can create Treello cards and retrieve lists and cards.

        • Might be a way to keep the 3 above in sync

    • IFTTT is used by Marna for automation, and Martin uses automation that sends a Notification when someone updates one of the team’s cards.

      • Martin uses Pushcut on iOS – as we mentioned in Episode 27 Topics

      • Trello provides a IFTTT Trigger, or Pushcut provides a Notification service, which can kick off a shortcut also.

      • Encountered some issues: each list needed its own IFTTT applet. Can’t duplicate applets in IFTTT so it’s a pain to track multiple Trello lists, even within a single board.

    • Automation might be a better alternative to Power-ups, as you can build them yourself.

  • Reflections:

    • Marna likes Trello. She uses it to be more productive, but would like a couple more function which might be addded as it becomes more popular.

    • Martin likes Trello too, but with reservations.

      • Dragging cards around seems a little silly. There are more compact and richer ways of representing such things.

      • A bit of a waste of screen real estate, as cards aren’t that compact. Especially as lists are all in one row. It would be nice to be able to fill more of the screen – with two rows or a more flexible layout.

In closing

  • GSE UK Virtual Conference will be 2 – 12 November 2021.

On the blog

So It Goes

Pi As A Protocol Converter

I wrote about Automation with my Raspberry 3B in Raspberry Pi As An Automation Platform. It’s become a permanent fixture in my office and I’ve given it another task. This blog post is about that task.

Lots of things use JSON(Javascript Object Notation) for communication via HTTP. Unfortunately they don’t all speak the same dialect. Actually:

  1. They do; It’s JSON pure and simple. Though some JSON processors are a little bit picky about things like quotes.
  2. It’s just as well there is the flexibility to express a diverse range of semantics.

This post is about an experiment to convert one form of JSON to another. When I say “experiment” it’s actually something I have in Production – as this post was born from solving a practical problem. I would view it as more a template to borrow from and massively tailor.

The Overall Problem

I have a number of GitHub repositories. With GitHub you can raise an Issue – to ask a question, log a bug, or suggest an enhancement. When that happens to one of my repositories I want to create a task in my task manager – OmniFocus. And I want to do it as automatically as possible.

There isn’t an API to do this directly, so I have to do it via a Shortcuts shortcut (sic) on iOS. To cause the shortcut to fire I use the most excellent PushCut app. PushCut can kick off a shortcut on receipt of a webhook (a custom URL) invocation.

Originally I used an interface between GitHub and IFTTT to cause IFTTT to invoke this webhook. This proved unreliable.

The overall problem, then, is to cause a new GitHub issue to invoke a PushCut webhook with the correct parameters.

The Technical Solution

I emphasised “with the correct parameters” because that’s where this gets interesting:

You can set GitHub up – on a repository-by-repository basis – to invoke a webhook when a new issue is raised. This webhook delivers a comprehensive JSON object.

PushCut webhooks expect JSON – but in a different format to what GitHub provides. And neither of these is tweakable enough to get the job done.

The solution is to create a “protocol converter”, which transforms the JSON from the GitHub format into the PushCut format. This I did with a Raspberry Pi. (I have several already so this was completely free for me to do.)

Implementation consisted of several steps:

  1. Install Apache web server and PHP on the Pi.
  2. Make that web server accessible from the Internet. (I’m not keen on this but I think it’s OK in this case – and it is necessary.)
  3. Write a script.
  4. Install it in the /var/www/html/ directory on the Pi.
  5. Set up the GitHub webhook to invoke the webhook at the address of the script on the Pi.

Only the PHP script is interesting. You can find how to do the rest on the web, so I won’t discuss them here.

PHP Sample Code

The following is just the PHP piece – with the eventual shortcut being a sideshow (so I haven’t included it).

<?PHP

$secret = "<mysecret>";
$json = file_get_contents('php://input');
$data = json_decode($json);

if($data->action == "opened"){
  $issue = $data->issue->number;
  $repository = $data->repository->name;
  $title = $data->issue->title;
  $url = $data->issue->html_url;

  $pushcutURL = "https://api.pushcut.io/" . $secret . "/notifications/New%20GitHub%20Issue%20Via%20Raspberry%20Pi";

  //The JSON data.
  $JSON = array(
      'title' => 'New Issue for ' . $repository,
      'text' => "$issue $title",
      'input' => "$repository $issue $url $title",
  );


  $context = stream_context_create(array(
    'http' => array(
      'method' => 'POST',
      'header' => "Content-Type: application/json\r\n",
      'content' => json_encode($JSON)
    )
  ));

  $response = file_get_contents($pushcutURL, FALSE, $context);
}

?>

But let me explain the more general pieces of the code.

  • Before you could even use it for connecting GitHub to PushCut you would need to replace <mysecret> with your own personal PushCut secret, of course.
  • $json = file_get_contents('php://input'); stores in a variable the JSON sent with the webhook. Let’s call this the “inbound JSON”.
  • The JSON gets decoded into a PHP data structure with $data = json_decode($json);.
  • The rest of the code only gets executed if $data->action is “opened” – as this code is only handling Open events for issues.
  • The line $pushcutURL = "https://api.pushcut.io/" . $secret . "/notifications/New%20GitHub%20Issue%20Via%20Raspberry%20Pi"; is composing the URL for the PushCut webhook. In particular note the notification name “New GitHub Issue Via Raspberry Pi” is percent encoded.
  • The outbound JSON has to be created using elements of the inbound JSON, plus some things PushCut wants – such as a title to display in a notification. In particular the value “input” is set to contain the repository name, the issue number, the original Issue’s URL, and the issue’s title. All except the last are single-word entities. If you are adapting this idea you need to make up your own convention.
  • The $context = and $response = lines are where the PushCut webhook is actually invoked.

As I said, treat the above as a template, with the general idea being that the PHP code can translate the JSON it’s invoked with into a form another service can use, and then call that service.

Conclusion

It was very straightforward to write a JSON converter in PHP. You could do this for any JSON conversion – which is actually why I thought it worthwhile to write it up.

I would also note you could do exactly the same in other software stacks, in particular Node.js. I will leave that one as an exercise for the interested reader. I don’t know whether that would be faster or easier for most people.

On the question of “faster” my need was “low volume” so I didn’t much care about speed. It was plenty fast enough for my needs – being almost instant – and very reliable.

One other thought: My example is JSON but it needn’t be. There need not even be an inbound or outbound payload. The idea of using a web server on a Pi to do translation is what I wanted to get across – with a little, not terribly difficult, sample code.

Mainframe Performance Topics Podcast Episode 28 “The Preview That We Do Anew”

(Originally posted 2 March, 2021.)

It’s unusual for us to publish a podcast episode with a specific deadline in mind. But we thought the z/OS 2.5 Preview announcement was something we could contribute to. So, here we are.

I also wanted to talk about some Open Source projects I’ve been contributing to. So that’s in there.

And it was nice to have Nick on to talk about zCX.

Lengthwise, it’s a “bumper edition”… 🙂

Episode 28 “The Preview That We Do Anew” long show notes.

  • This episode is about several of the z/OS V2.5 new functions, which were recently announced, for both the Mainframe and Performance topics. Our Topics topic is on Martin’s Open Source tool filterCSV.

  • We have a guest for Performance: Nick Matsakis, z/OS Development, IBM Poughkeepsie.

  • Many of the enhancements you’ll see in the z/OS V2.5 Preview were provided on earlier z/OS releases via Continuous Delivery PTFs. The APARs are provided in the announce.

What’s New

Mainframe – Selected z/OS V2.5 enhancements

  • Many of the enhancements you’ll see in the z/OS V2.5 Preview were provided on earlier z/OS releases via Continuous Delivery PTFs. The APARs are provided in the announce.

  • We’ve divided up the Mainframe V2.5 items into two sections: installation and non-installation.

z/OS V2.5 Installation enhancements.

  • IBM will have z/OS installable with z/OSMF, in a portable software instance format!

  • z/OS V2.4 will not be installable with z/OSMF, and z/OS V2.4 driving system requirements remain the same.

  • z/OS V2.5 will be installable via z/OSMF, so that is a big driving system change.

    • However, there is a small window when z/OS V2.4 and z/OS V.5 are concurrently orderable in which z/OS V2.5 will have the same driving system requirements as z/OS V2.4. That overlapping window when z/OS V2.5 is planned to be available via both the old (ISPF CustomPac Dialog) and new (z/OSMF format) is September 2021 through January 2022.

    • After that window, be aware! When z/OS V2.5 is the only z/OS orderable release – at that time, all IBM ServerPac will have to be installed with z/OSMF.

    • All means CICS, Db2, IMS, MQ, and z/OS and all the program products.

    • To be prepared today for this change:

      • Get z/OSMF up and running on their driving system.

      • Learn z/OSMF Software Management (which is very intuitive and try to install a portable software instance from this website.

    • This is a big step forward in the z/OS installation strategy that IBM and all the leading software vendors have been working years on.

      • John Eells came to this very podcast in Episode 9 to talk about it.

    • CICS, Db2, and IMS are already installable with a z/OSMF ServerPac. You can try those out right now.

    • CBPDO will remain an option, instead of ServerPac. But it much harder to install.

      • ServerPac is much easier, and a z/OSMF ServerPac is easiest of all.

z/OS V2.5 Non-installation enhancements.

  • Notification of availability of TCP/IP extended services

    • For many operational tasks and applications that depend on z/OS TCP/IP communication services the current message is insufficient

    • New ENF event intended to enable applications with dependencies on TCP/IP extended services to initialise faster

  • Predictive Failure Analysis (PFA) has more checks

    • For above the bar private storage exhaustion, JES2 resource exhaustion, and performance degradation of key address spaces.

  • Workload Manager (WLM) batch initiator management takes into account availability of zIIP capacity

    • Works most effectively when customer has separate service classes for mostly-zIIP and mostly-GCP jobs

    • Catalog and IDCAMS enhancements

      • Catalog Address Space (CAS) restart functions are enhanced to allow you to change the Master Catalog without IPL

      • IDCAMS DELETE mask takes TEST and EXCLUDE. TEST to see what would be deleted using the mask. EXCLUDE is further filtering – beyond the mask.

      • IDCAMS REPRO moves I/O buffers above line. This will help avoid 878 “Insufficient Virtual Storage” ABENDs.
        We think think this might allow more buffers, and multitasking in one address space.

    • New RMF Concept for CF data gathering

      • There is a a new option, not the default, to optimize CF hardware data collection to one system. Remember SMF 74.4 has two types of data: system specific, and common to all systems.

      • This is designed to reduce overhead on n-1 systems.

  • RMF has been restructured, but all the functions are still intact. z/OS V2.5 RMF is still a priced feature.

    • A new z/OS V2.5 base element called “Data Gatherer” provides basic data gathering and is available to all, whether you’ve bought RMF or not. It will cut some SMF records.

    • There is a new z/OS V2.5 price feature called “Advanced Data Gatherer” which all RMF users are entitled to.

    • Marna is mentioning this, as the restructure has brought about some customization changes you’ll need to do one time for parmlib with APF and linklist.

  • More quite diverse RACF health checks for Pass tickets, subsystem address spaces active, and sysplex configuration.

Performance – z/OS V2.5 zCX enhancements.

  • Our special guest is Nick Matsakis, who is a performance specialist in z/OS Development, and has worked on several components in the BCP (GRS, XCF/XES, …). Martin and Nick have known each other for many years, recalling Nick’s assignment in Hursley, UK.

  • zCX is a base element new in z/OS V2.4, and requires a z14. It allows you to run Docker Container applications that run on Linux on Z on z/OS.

  • zCX is important for co-locating Linux on Z containers with z/OS. You can look at zCX like an appliance, which are z/OS address spaces.

  • Popular use cases can be found here and in the Redbook here. Another helpful source is Ready for the Cloud with IBM zCX.

    • Nick mentions the use cases of adding microservices to existing z/OS applications being served by a zCX container, and the MQ Concentrator for reducing z/OS CPU costs, by running it on zCX. Another is Aspera whcih good for streaming-type workloads.

  • zIIP eligibility enhancements

    • Context switching reduction was delivered to typically expect about 95% offload to zIIP.

  • Memory enhancements

    • Originially it was all 4K fixed pages. New enhancements include support for 1 MB and 2 GB large pages (still fixed) for backing guests.

      • Increases efficiency of memory management, with better performance is expected, mainly based on TLB miss reduction.

      • In house, Nick saw .25% up to about 6-12%, depending on what you are running.

    • Note need to set LFAREA as discussed in Episode 26.

      • LFAREA as of z/OS V2.3 is the maximum number of fixed 1M pages allowed on system. 2GB hasn’t changed.

      • zCX configuration allows you to say which page sizes you’d like to try. Plan for using 2GB.

    • Guest memory is planned to be configured up to 1 TB.

      • zCX uses fixed storage so the practical limit may be lower. The limit used to be much lower, at about 100 GB.

      • Now we support up to 1000 containers in a zCX address space. Capacity is increasing.

  • Another relief is in Disk space limits

    • The number of data and swap disks per appliance is planned to be increased to as many as 245. This is intended to enable a single zCX to address more data at one time.

    • Point is you can run more and larger containers.

  • Instrumentation enhanced

    • Monitor and log zCX resource usage of the root disk, guest memory, swap disk, and data disks in the servers job log.

    • zCX resource shortage z/OS alerts are proactive alerts that are sent to the z/OS system log (SYSLOG) or operations log (OPERLOG) to improve monitoring and automated operations. The server monitors used memory, root disk space, user data disk space, and swap space in the zCX instance periodically and issues messages to the zCX joblog and operator console when the usage rises to 50%, 70%, and 85% utilization. When returning below 50%, an information message is issued

    • But still nothing in SMF to look inside a zCX address space

      • There is Docker-specific instrumentation that can provide that for you.

  • SIMD (or Vector)

    • SIMD is a performance feature, and can be used for analytics.

    • Some containers don’t check if they are running on hardware where SIMD is available.

  • Note that most of what’s in the z/OS 2.5 Preview for zCX is rolled back to z/OS 2.4 with APARs.

  • From this, we can conclude zCX wasn’t a “one and done”.

    • z/OS 2.5 might be a good time to try it. There is a 90-day trial period, as there is a cost for it. But, why wait for 2.5?

  • Nick’s presentation (with Mike Fitzpatrick) can be downloaded here.

Topics – filterCSV and tree manipulation

  • Trees are nodes that have zero to many children. You can have a leaf node (zero children), or a non-leaf node (one or more children).

    • Navigation can be recursive or iterative, which makes it nice for programming.

  • Mindmapping leads to trees. Thinking of z/OS: Sysplex -> System -> Db2 -> Connected CICS leads to trees. Also, in Db2 DDF Analysis Tool we show DDF connections as a tree.

  • Structurally, each node is a data structure with fields such as readable names. Each node has pointers to its children and maybe its parent. This gives it its “topology”, and tree levels.

  • iThoughts is a mind mapping tool, and displays a mind map as a tree. Nodes can have colours and shapes, and many other attributes besides.

    • iThoughts runs on Windows, iOS, iPadOS and macOS.

    • Exports and imports CSV files, with a tree topology and also node attributes, such as shape, colour, text, notes.

    • Has very little automation of its own. But crucially you can mangle the CSV file outside of iThoughts, which is what filterCSV does.

  • filterCSV is a python open source program that manipulates iThoughts CSV files.

    • It could address the automation problem, as it mangling automatically.

    • An example: automatically colours the blobs based on patterns (regular expressions).

      • Colouring CICS regions according to naming conventions

  • fiterCSV started simple, and Martin has kept adding function. Most recently find and replace. As it’s an open source project, contributions are welcomed.

On the blog

So It Goes

SMF 70-1 – Where Some More Of The Wild Things Are

(First posted February 21, 2021)

As I recall, the last time I wrote about SMF 70-1 records in detail was Engineering – Part Two – Non-Integer Weights Are A Thing. Even if it weren’t, no matter – as I’d like you to take a look at it. The reason is to reacquaint you with ERBSCAN and ERBSHOW – two invaluable tools when understanding the detailed structure and contents of an SMF record. (Really, an RMF SMF record.) And it does introduce you to the concept of a Logical Processor Data Section.

This post is another dive into detailed record structure. (The first attempt at the last sentence had the word “derailed”; That might tell me something.) 🙂

In most cases a system cuts a single SMF 70 Subtype 1 record per interval. But this post is not about those cases.

The Structure Of A 70-1 Record

SMF 70-1 is one of the more complex record subtypes – and one of the most valuable.

Here is a synopsis of the layout:

What is in blue(ish) relates to the system cutting the record. The other colours are for other LPARs.

At its simplest, a single 70-1 record represents all the LPARs on the machine. But it’s not always that simple.

Let me point out some key features.

  • The CPU Data Sections are 1 per processor for the system that cut the record. In this example there are three – so this is a 3-way.
  • zIIPs and GCPs are treated the same, but they are individually identifiable as zIIP or GCP.
  • There is one Partition Data Section per logical partition on the machine, plus 1 called “*PHYSCAL”.
  • There is one Logical Processor Data Section per logical processor, plus 1 per physical processor.

The colour coding is useful here. Let’s divide it into two cases:

  • The processors for the cutting LPAR.
  • The processors for the other LPARs.

For what we’ll call “this LPAR”, there are CPU Data Sections for each processor, plus a Partition Data Section, Logical Core Data Sections, and Logical Processor Data Sections.

For each of what we’ll call “other LPARs” there are just the Partition Data Section and its Logical Processor Data Sections.

You’ll notice that the blue Partition Data Section and its Logical Processor Data Sections are the first in their respective categories. I’ve always seen it to be the case that this LPAR’s sections come first. I assume PR/SM returns them in that sequence – though I don’t know if this is an architectural requirement.

The relationship between Partition Data Sections and the corresponding Logical Processor Data Sections is straightforward: Each Partition Data Section points to the first Logical Processor Data Section for that LPAR and has a count of the number of such sections. The pointer here is an index into the set of Logical Processor Data Sections, where the first has an index of 0. (ERBSHOW calls it “#1”.)

(A deactivated LPAR has an (irrelevant) index and a count of 0 – and that’s how my code detects them.)

So far so good, and quite complex.

How Do You Get Multiple 70-1 Records In An Interval?

Obviously each system cuts at least one record per interval – if 70-1 is enabled. So this is not about that.

In recent years the number of physical processors in a machine and logical processors per LPAR have both increased. I regard these as technological trends, driven mainly by capacity. At the same time there is an architectural trend towards more LPARs per machine.

Here are the sizes of the relevant sections – as of z/OS 2.4:

  • CPU Data Section: 92 bytes.
  • Partition Data Section: 80 bytes.
  • Logical Processor Data Section: 88 bytes.
  • Logical Core Data Section: 16 bytes.

These might not seem like large numbers but you can probably see where this is heading.

An SMF record can be up to about 32KB in size. You can only fit a few hundred Logical Processor Data Sections into 32KB, and that number might be significantly truncated if this LPAR has a lot of processors.

All of this was easy with machines with few logical processors (and still is).

But let’s take the case of a 100-way LPAR (whatever we think of that.) Its own sections are (92 + 88 + 16) x 100 or 19.6KB plus some other sections. So at least 20KB. And that’s before we consider sections for other LPARs.

Now let’s ignore this LPAR and consider the case of 50 1-way LPARs. There the PR/SM related sections add up to (80 + 88) x 50 = 8.4KB. Of course it’s extremely unlikely many would be 1-way LPARs, so the numbers are realistically much higher than that.

By the way, for a logical processor to count in any of this it just has to be defined. It might well have zero Online Time. It might well be a Parked Vertical Low. It doesn’t matter. The Logical Processor Data Sections are still there.

So, to exceed the capacity of a 32KB 70-1 SMF record we just have to have a lot of logical processors across all the LPARs in the machine, whether in this system or other LPARs. And an exacerbating factor is if these logical processors are across lots of LPARs.

What Does RMF Do If The Data Won’t Fit One Record?

I’ve seen a lot of SMF 70-1 records in my time, and spent a lot of time with ERBSCAN and ERBSHOW examining them at the byte (and sometimes bit) level.

I do know RMF takes great care to adjust how it lays out the records.

Firstly, to state the obvious, RMF doesn’t throw away data; All the sections exist in some record in the sequence.

Secondly, RMF keeps each LPAR’s sections together. So the Partition Data Section and its related Logical Processor Data Sections are all in the same record. This is obviously the right thing to do, otherwise the index and count for the Logical Processor Data Sections could break.

Thirdly, and this is something I hadn’t figured out before, only one record in the sequence contains the CPU Data Sections. (I think also the Logical Core Data Sections.)

How Should I Handle The Multi-Record Case?

Let me assume you’re actually going to have to decide how to deal with this.

There are two basic strategies:

  1. Assemble the records in the sequence into one record in memory.
  2. Handle each record separately.

Our code, rightly in my opinion, uses Strategy 2. Strategy 1 has some issues:

  • Collecting information from multiple records, and timing the processing of the composite data.
  • Fixing up things like the index of the Logical Processor Data Sections.

Probably some tools do this, but it’s fiddly.

So we process each LPAR separately, thanks to all the information being in one record. And so we can process each record separately.

Reality Check

If you have only one 70-1 record per interval per cutting system none of the above is necessary to know. But I think it’s interesting.

If you rely on some tooling to process the records – and most sensible people do – you probably don’t care about their structure. Certainly, the RMF Postprocessor gets this right for you in the CPU Activity Report (and Partition Data Report sub-report).

So, I’ve probably lost most of my audience at this point. 🙂 If not, you’re on my wavelength – which isn’t crowded. 🙂 (This is the second “on my wavelength” joke in my arsenal, the other being open to misinterpretation.)

I like to get down into the physical records for a number of reasons, not least of which are:

  • When things break I need to fix them.
  • It cements my understanding of how what they describe works.

Oh, and it’s fun, too.

Final Thoughts

This post was inspired by a situation that required yet more adjusting of our code. Sometimes life’s that way. In particular, a number of LPARs were missing – because our Assembler code threw away any record with no CPU Data Sections. (This is inherited code but it’s quite possibly a problem I introduced some time in the past 20 years.)

I should point out that – for simplicity – I’ve ignored IFLs, (now very rare) zAAPs, and ICFs. They are treated exactly the same as GCPs and zIIPs. Of course the record-cutting LPAR won’t have IFLs or ICFs.

I have a quite old presentation “Much Ado About CPU”. Maybe I should write one with “Part Two” tacked on. Or maybe “Renewed” – if it’s not such a radical departure. But then I’ve done quite a bit of presentation writing on the general topic of CPU over recent years.