(Originally posted 2018-01-26.)

“What is this DB2 subsystem for?” is an important question – at least to me. And explaining this topic over lunch to a friend of mine who was a DB2 developer, convinced me it has more general importance.

If you’re a DB2 sysprog or DB2 the answer is probably well known to you ¹.

If you’re not a DB2 person it’s probably not so obvious.

Something worth noting at this stage is that very little of what’s in this post is specific to DB2. Much of it applies to IMS and MQ, for example.

Why Do We Care What A DB2 Is For?

In general, the more you know about what is going on on your systems the better.

Certainly, from my perspective as an outsider, I think I’m more effective with a customer if I know what their infrastructural components are for. It’s a kind of architectural thinking – so it might appeal to architects and infrastructure managers in customers as well.

To take a (generalised) example, if I see a DB2 Datasharing group I want to know what each member (DB2 subsystem) is for. Elements of the answer might include statements like:

• This DB2 is for Batch.
• These DB2s are clones of each other, for resilience.

We’ll return to these presently.

One other point: Subsystems arise over time for different reasons. Two examples:

• To provide virtual storage constraint relief.
• Different applications came into being, each with their own DB2. They might even have come from different companies merging. I’ve heard customers refer to these as “heritages”.

Add to that how Datasharing groups come to exist and the picture gets really interesting.

And the reason for all this is to be able to provide governance and manage DB2 subsystems better.

Detecting The Different Roles

This post really emphasises the use of SMF 30 (Address Space) and RMF data. The reason for using this data (and not DB2-specific data) is quite simple: The approach has to scale. Many customers have more than 50 DB2 subsystems, some more than 100. If you do, you’re not going to be keen on processing Accounting Trace for all of them.²

Of course, having formed a high level view this way, you might well want to dive into Accounting Trace – but only for some subsystems and time ranges.

Detecting DB2 Subsystems

This is the easy bit: From SMF 30, if the job name is fredDBM1 and the program name is DSNYASCP then we have subsystem fred. We can see its DB2 version in the Usage Data Section.

Obviously you see its SMFID, Workload, Service Class, Report Class and Reader Start Time.

Detecting Datasharing Groups

You can, most of the time, obtain a good grouping of subsystems into Datasharing groups. The following technique works if each DB2 subsystem has its own IRLM and if the IRLM address space’s job name matches the subsystem. For example the IRLM address space corresponding to DB2ADBM1 (subsystem DB2A) is DB2AIRLM.

A DB2 Datasharing group has two XCF groups, supporting the group’s LOCK1 Coupling Facility structure. One’s name begins with IXCLO. It’s the other one that interests me: It is a proxy for the group name. For example, it might be called DXRPROD when the actual group’s name is DSNPROD or DB2PROD. But it’s close enough. (CF structure names could be used to patch the name, perhaps.)

In any case the IRLM address spaces – as proxies for subsystems – can be used this way to detect Datasharing groups.

I discussed this in detail in DB2 Data Sharing and XCF Job Name back in 2009.

Working Out What Role A Subsystem Plays

This is the bit that really excites me: My bold claim is I can tell you something about the work coming into a DB2 subsystem without using DB2 instrumentation³. And the use of SMF 30 only is important because most people connect SMF 30 across their entire z/OS estate.

The key to this is the Usage Data Section in SMF 30, which I’ve talked about many times.

As an example, take a CICS region. (You detect these by the program name being DFHSIP). If the CICS region connects to a DB2 subsystem you get at least one Usage Data Section that documents the DB2 connection. In particular it tells you the DB2 subsystem it connects to. I’ve discussed this in a 2012 post: Another Usage Of Usage Information.

By the way a CICS region can directly attach to only one DB2 subsystem (likewise only one MQ queue manager).

So you can identify all the CICS regions connecting to a given DB2. You could, in addition to naming them, add up their CPU.

Similarly, for IMS work you’d see a Usage Data Section mentioning IMS.⁴

And almost everything else you could lamely call “Other” or be bold and call it “Batch”. My code counts the distinct job names and sums up the CPU.

DDF is an interesting case: You would find the DIST address space for the subsystem⁵ and examine the Enclave-related fields. I wrote about this in DB2 DDF Transaction Rates Without Tears But essentially you can pick up the DDF Transaction Rate and DDF CPU.

So, let’s take a real example. Here is a DB2 Datasharing group with 6 members, running on 2 LPARs.

The numbers are from my prototype code. All numbers are multi-hour (shift) averages.

Looking at SYSA first:

• DB1A has a large amount of CICS work (over 7 engines’ worth) from 14 regions. It also has a very small amount of DDF work. “Other” here is mainly Batch and it is about an engine’s worth.
• DB2A also has lots of CICS work – over 7 engines’ worth from 16 regions. DDF work is much more significant than on DB1A – at about half an engine. “Other” is again similar.
• DB3A has very little work to it – just a small amount of DDF.

Turning to SYSB:

• DB1B has 11 CICS regions, using 5 engines’ worth of CPU and a small amount of DDF. Again “Other” is just over an engine’s worth.
• DB2B is about as lightly used as DB3A – with a smattering of DDF and a littl bit of “Other”.
• DB3B has 17 CICS regions, using 6.4 engines’ worth of CPU. DDF here is a significant workload, similar to that on DB2A. Here is uses about 1/3 of an engine. “Other” is about 1 engine’s worth of CPU.

So, these are “broad brush” characterisations but I think they are useful. For example, the 2 DDF-oriented subsystems (DB3A and DB2B) might be easy to fold into other subsystems. I say this because they are light on transaction rates and CPU so should pose no threat to the scalability of other subsystems in the Datasharing group.

Conclusion

Much of this instrumentation you might’ve seen before. The trick here is to invert the question “what does this address space connect to?” to yield “what connects to my DB2?” And that way you get a view of the role of the DB2 subsystem.

Of course, this is not nearly as deep an analysis as you could do with DB2 Accounting Trace – such as what transactions get at a DB2. But it is much lighter and more scalable. It’s great for a first pass – which might be all I get to do.

One other thought: This feels like the beginning of a journey to characterise DB2 subsystems. One could, for example, use WLM Service Classes⁶ for classifying each DB2’s role. This wouldn’t work for DDF but for other work you could sum up the 30’s by Service Class instead of e.g. “CICS”.

(Originally posted 2018-01-19.)

So now I get to sit and relax for umpteen hours, while a 787 speeds me to California. So I can write something. And, yes, it’s been a while.

To tell you the truth, I’ve been working on back-to-back engagements and enjoying vacation for what seems like forever. And after a stint of that I generally I end up with a swirl of “if only I had time to write about that” topics in my brain. So here I am ”Going To California On A Big Jet Plane”[1] with time on my hands.

So I have this customer – and it’s not the first like it – who has a Parallel Sysplex with a pair of Coupling Facility LPARs and a couple of Production z/OS LPARs. Not the first, either, to have offline Coupling Facility links. But I didn’t know they had offline links until they told me. Shame on me! 🙂

If a customer tells me things I guess I should know I go on a hunt for how I could have figured it out from the data. And so it was in this case. And, having just been successful at writing code to detect offline links, I’m writing this to show you what’s possible and how.

Three Sources Of Link Data

SMF 74 Subtype 4 is the main Coupling Facility Activity record, and it has three sets of information about links:

• Local Facility Data Section
• Remote Facility Data Section
• Path Data Section

You get at least one 74–4 for each coupling facility from each z/OS connected to it. Ideally you’d enable 74–4 on all systems in the sysplex.[2]

I say “at least” as modern CFs tend to spill to a second and, occasionally, a third. So here’s another thing I’ve absorbed over the last few customer engagements: Customers nowadays have hundreds of structures in each coupling facility.[3]

Local Facility Data Section

You get one of these in the 74–4 record – whether the base or the overflow ones. It has to be there as it has the Coupling Facility name, as well as the machine it’s on.

It also has some relevant things which I’ll touch on now, and come back to later:

• Path masks
• Channel Path Acronyms
• Channel Path IDs (CHPIDs)
• Index into Path Data Sections and count of the number of sections

As examples of Channel Path Acronyms, I’ll take three:

• CIB – Infiniband
• CS5 – ICA-SR
• CL5 – Coupling Express Long Reach

Remote Facility Data Section

You get one or more of these, depending on how many CFs the one in the Local Facility Data Section has duplexing links to. The path-related information in this section is:

• Count of the number of paths
• Channel Path Acronyms
• CHPIDs
• Index into Path Data Sections and count of the number of sections

Pretty similar to Local Facility Data Section information. But not quite. The information is only for online paths.

Path Data Section

I’ve written about this section many times, most recently in And Latency Once More.

You get one section for each link under certain circumstances: The link has to be physical (no IC links) and modern enough (not ICB, for example). Unfortunately, it has to be online. So this section doesn’t help us in our quest for offline links.

And the reason I keep writing about this section is because it is full of value and surprises. But not this time. 😦

Actually There’s A Fourth

I include this for completeness[4]. SMF 73 gives information on CHPIDs – from a z/OS system.

For CF links this information is very basic. Mainly the CHPID, whether it’s shared, whether it’s online, and Channel Path Acronym. [5] For other types of channel, such as FICON, we get a lot more, useful, information.

There is a potential use for this data, quite apart from corroboration: If a CF-related channel path appears in SMF 73 but not in 74–4 it implies it’s not connected. Of course it would be strange if it managed to be online despite that. 🙂 I should probably code to this possibility.

The Truth Is Masked

In the Local Facility Data Section piece I mentioned path masks. It might not have seemed obvious but they are key to this. There are three 8–bit mask fields:

• Path Installed Mask
• Path Available Mask
• Patch Composite Mask

Each bit represents a different path and each path is described by the same bit position in all the masks.

These terms might be familiar in the context of disk controllers.

Consider the following worked example:

This is from a real customer. Reading from top to bottom.

1. The channel path acronyms speak of 4 CS5 (ICA-SR) paths and 4 CIB (Infiniband) paths. Note the order.
2. The Path Installed Mask (Hex FF) has all the bits turned on – so all 8 paths are installed.[6]
3. The CHPIDs are listed. The order here is again important.
4. The Path Available Mask (Hex B1) has some bits on and some off – 10110001 being the bit string. So only some of these paths are available, the rest being installed (in this case) but offline.
5. The Path Composite Mask (also Hex B1) reflects both the Installed and Available Masks.

Taken together, we conclude:

• All 4 Infiniband paths are installed but offline – CHPIDs 0E, 6A, 6F, and 78.
• All 4 ICA-SR paths are installed and online – CHPIDs 04, 24, 25, and C6.

Conclusion

For links between z/OS LPARs and CF LPARs we can see both the online and offline paths. For the CF to CF links we can only see the online ones.

There is an old joke. It goes like this, updated a little: “I wrote a long blog post as I didn’t have time to write a short one”. Well, I did have time to write a short one… 🙂

By the way “Maskerade” is the name of a Terry Pratchett Discworld book, in case you wondered if there was a cultural reference. High time I read it again.

(Originally posted 2017-12-03.)

In my “Parallel Sysplex Performance Topics” presentation I have some slides on Coupling Facility Processor Busy. I’ve worried about including them, considering them borderline boring.¹

In my head I justified them because they:

1. Help people understand Structure Execution Time (R744SETM).
2. Help people see the changed behaviour with Thin Interrupts.

For both of those topics it’s been enough to take a “whole Coupling Facility” view, aggregating over all the Coupling Facility’s processors.

But, and not many people know this, RMF documents the picture for individual processors in SMF 74 Subtype 4. The reason this isn’t widely known is mainly the reports don’t show this level of detail.

One can speculate why this level of detail exists. My take is that it was relevant long ago when we had “Dynamic ICF Expansion”.² This feature allowed an ICF LPAR to expand beyond the ICF pool into the GCP Pool. Performance Impacts of Using Shared ICF CPs describes this feature. (The document is from 2006 but it does describe this one feature quite well.)

There, you’d want a better picture of Coupling Facility processor busy than just summing it all up. In particular you’d want to know if the GCP engines had been used.

What Is Coupling Facility Processor Busy?

This seems like a silly question to ask, but it isn’t.

If you were to look at RMF’s Partition Data Report for the ICF pool you’d find dedicated ICF LPARs always 100% busy. And it’s not just because they’re dedicated. It’s because the Coupling Facility spins looking for work. So that’s not a useful measure – for dedicated ICF LPARs³.

So a better definition is required, and thankfully RMF provides one. There are two relevant SMF 74 Subtype 4 fields:

• R744PBSY – when the CF is actually processing requests.
• R744PWAI – when the CF is not processing requests but the CFCC is still executing instructions.

Using these two the definition of busy isn’t hard to fathom:

Coupling Facility Busy % = 100 * R744PBSY / (R744PBSY + R744PWAI)⁴

This, as I say, is normally calculated across all processors in the Coupling Facility.

By the way, you might find Coupling Facility Structure CPU Time – Initial Investigations an interesting read. It’s only 9 years old. 🙂

Processor Busy Considerations

But let’s come almost up to date.

I recently looked at a customer’s parallel sysplex and got curious about engine-level Coupling Facility Busy, so I prototyped some code to calculate it at the engine level. I not only summarised across all the RMF intervals but plotted by the individual 15-minute interval.

Here is the summarised view for one of their two 10-way z13 Coupling Facilities:

The y axis is Coupling Facility Busy for the engine, the x axis being the engine number.

So clearly there is some skew here, which I honestly didn’t expect. By the way, at the individual interval level the skew stays about the same. Indeed the same processors dominate, to the same degree.

A couple of points:

• At this low utilisation level the skew doesn’t really matter as no engine is particularly busy. However, we like to keep the Coupling Facility as a whole below 50% busy. Part of this is about “white space”⁵ but it’s also about everyday performance. I have to say I’ve not seen a case where Coupling Facility busy caused requests to get elongated, but that means nothing. 🙂 So, I’d like to suggest that individual engine busy needs measuring, to ensure it doesn’t exceed 50%. This is a revision of the “whole CF” guideline. But at least the data’s there.

• This is a 10-way Coupling Facility. It would be better, where possible, to corral the work into fewer engines. Perhaps fitting within a single processor chip. In this customer’s case there’s a spike which means this isn’t possible. Working on the spike’s the thing.

CFLEVEL 22

Now let’s come really up to date.

z14 introduced CFLEVEL 22. One area of change is in the way work is managed by the Coupling Facility Control Code (CFCC). In particular, processors have become more specialised. This is to improve efficiency with larger numbers of processors in a Coupling Facility.

CFLEVEL 22 introduced “Functionally specialized” ICF processors for CF images with dedicated processors defined under certain conditions:

• One processor for inspecting suspended commands
• One processor for pulling in new commands
• The remaining processors are non-specialized for general CF request processing.

This avoids lots of inter-processor contention previously associated with CF engine dispatching.

If there are going to be specialised engines I’d expect more skew than before. At this stage I’ve no idea whether the two specialised processors are going to be busier than the rest or less busy⁶. Further, I don’t know how you would manage down the CPU for either the specialised processors, nor the rest. Maybe the state of the art will evolve in this area.

Note: There’s no way of detecting a processor as belonging to one of these three categories.

So, this makes it even more interesting to examine Coupling Facility Busy at the individual engine level. I’ve not yet seen CFLEVEL 22 RMF data, but at least I have a prototype to work from, whether the customer’s data is at the CFLEVEL 22 or not.

Stay tuned.

(Originally posted 2017-11-19.)

I was in two minds whether to do this as a screencast or a blog post. Obviously I plumped for the latter. There are a few reasons why:

• This is not terribly visual, there being only one graph.
• It’ll reach a wider audience, and the message is quite important.
• I’m feeling lazy. 🙂

Anyway, here we are and I think this is quite an important subject – so I’m glad you’re here.

Usually, when looking at WLM, we tend to ignore service classes such as SYSSTC and SYSTEM. But there are two reasons why you shouldn’t:

• What’s classified to SYSSTC matters.
• It’s not a given that it will perform well.

It’s the latter that concerns us in this post. (The former is touched on in Screencast 12 – Get WLM Set Up Right For DB2.)

From the very same data set in that screencast I saw something I’d not noticed before: SYSSTC velocity is alarmingly low.

The above is one of four systems. Each has a persistently low velocity in the range 25% – 40%.

I didn’t expect this. To be honest, I’ve never looked at SYSSTC Velocity before. And what made me see it was adding the above graph to my kitbag a few months ago ¹ . (That and what happens to Importance 1, 2, etc Goal Attainment.)

Never having looked at SYSSTC velocity I have no real basis for an expectation, but this seems alarmingly low. And as I see more customer data I’ll form some “folklore in my head” 🙂 about it.

So this begs the question “what is it that is making SYSSTC’s velocity so low?” The search for an answer has to start with understanding which Delay component drags the velocity down. In this case it is Delay For CPU (and not zIIP, by the way).

I would’ve thought SYSSTC was relatively protected from CPU queuing ² but the data tells us otherwise. But consider two things:

• The low velocity is pretty consistent across the day.
• We know – from the previous blog post – there can be substantial CPU in SYSSTC at times.

So, it’s not really workload driven. So I suspect the LPAR set up and other things going on in the machine. A vague diagnosis for now. But at least I suspect something 🙂 – and that’s quite an advance.

Now why is this important? If you’ve reviewed Get WLM Set Up Right For DB2 you’ll know that key address spaces, such as DB2 / IMS lock managers (IRLM), run there. Without good access to CPU these address spaces will damage performance across a wide range of work.

So this is an unusual situation – until I keep seeing this in customers. 🙂 But it’s one worth looking out for.

One final thought: I probably should suppress the graph lines for SYSSTC1 – SYSSTC5 if there is no CPU in them. Oh to have some spare time. 🙂

(Originally posted 2017-11-05.)

For once this isn’t a blog post that discusses a podcast episode or a screencast. It is one where I feel a little exposed, but only a little.¹

I just updated my “Even More Fun With DDF” presentation – after a whole two months. You’d think there’d be little new to say after only two months, but you’d be wrong. Quite apart from some other additions, there is a whole new thought² to share. And I’m interested in opinions on it.

I think it’s a quite important thought: WLM Controls Are Not Enough For DDF.

Let me explain.

Traditional DDF Control Mechanisms

As you probably know, you classify DDF work below the DB2 address spaces and based on multiple potential qualifiers. Each transaction is a single commit³. Which is often just a small part of the requestor’s conversation with DB2. And WLM doesn’t know about the whole conversation, just the (independent enclave) transaction.

You can use period aging – so as a transaction accumulates service it can fall into periods 2, 3, and maybe 4⁴. You would expect each successive period to have a looser goal and a lower importance.

You can also use Resource Groups – but I consider that a pretty blunt instrument.

Where Does This Fall Short?

It falls short in two main areas:

• Non-Performance Controls
• Short Transactions

Non-Performance Controls

It might be stating the obvious but this is important: WLM does not control access to DB2.

So, you have to set up Security in DB2 and your security manager (such as RACF). You not only have to decide who can access a given DB2 and from where but also what they can do.

This is not a performance issue as such, but it’s the first clue that DB2 and Security people need to be involved in discussions about how to manage DDF work.

Short Transactions

A very short transaction is difficult to manage in any case. A bunch of all-the-same short transactions doubly so. There’s no period aging to be had there, for one.

This is the new area for me: I’m seeing in several recent cases bursts of short transactions from the same source. Two examples come to mind:

• SAP Batch. Which is really a bulk insertion of short DDF transactions.
• Java Batch. Likewise, quite often.

One thing these have in common is that they are generally⁵ detectable. The Unit Of Work ID⁶ is a constant for the “batch job”.

But this doesn’t really help control them. For that you really need gating mechanisms in DB2 / DDF and maybe outboard of that. And that’s really the new point.

To be fair, for SAP Batch, you generally see a Correlation ID of xyzBTCnnn where xyz is a kind of application code and nnn is a three-digit number. So you could classify it in a separate Service Class from e.g. SAP Dialog transactions (xyzDIAnnn).

Two Parting Thoughts

Another thing that occurred to me is this: Control of DDF is important not just for System purposes but to protect other DDF from rogue⁷ DDF.

Consider DB2 logical resources such as DBATs, which are a subsystem-level resource. If rogue DDF work came in and used them all at once it could crowd out Production DDF. And there’s plenty of that around that matters. So you definitely want to protect Production DDF. Probably with DDF Profiles, but with other DB2 mechanisms as available. And this is where I chicken out and defer to people like Rob Catterall – and posts such as this.

And one final thought: Slowing down DDF work might not be all that helpful, particularly if it holds locks that other work needs. But then this is true of other work, like lower-priority batch jobs. So WLM controls might have an unfortunate effect.

(Originally posted 2017-10-31.)

Hot on the heels of Screencast 11 – DDF Spikes is my latest screencast: Screencast 12 – Get WLM Set Up Right For DB2.

In it I talk about the important topic of ensuring DB2 is protected against shortages of CPU (and zIIP). There are a couple of quite nice examples to illustrate the point.

I should note that I don’t see instrumentation that explicitly shows what happens when IRLM gets heavy competition for CPU. I’m more concerned to show – as one of the examples does – that DBM1 can be significant so it’s worth keeping below IRLM.

Production Notes

This was the first time using Camtasia where I pruned out silence at the final – audio and video recombined – stage. I think I’m finally getting the hang of the tool and there’s a certain amount of muscle memory developed now.

Also, and I hope it doesn’t show too much, my editing of the audio (with Audacity) involved getting rid of a lot of huffing and puffing. I’d like to believe I wasn’t getting (too) old (to record) 🙂 and that this was just a cold. It seemed to me the more excited I got the less the huffing and puffing. I’m not sure if there’s a life lesson there or not. 🙂

(Originally posted 2017-10-30.)

It’s been a busy few weeks but we’ve a new episode out.

The experiment with narrowing the “stereoscape” was an interesting one. It is always going to be effort to do this, but having edited all 5 segments I got quite good at it. I think I maybe narrowed it a little too much, but I’d be interested in how it sounds to y’all. To my ears Marna and I (and indeed our guest and Marna) are coming at you from discernably different points, but it’s not too harsh.

This is the second episode where I added chapter markers. In case you’re not able to see them in your podcast client this is how it looks on my phone:

Anyhow, we hope you enjoy this show. Here are the show notes.

Episode 16 “Chapter and Worse” Show Notes

Here are the show notes for Episode 16 “Chapter and Worse”. The show is called this because our Topics topic is about adding chapter markers (and pictures!) to our published MP3 file. We hope those of you that can see them enjoy them.

Where we’ve been

Marna has just returned from conferences and events in Johannesburg ZA, Chicago IL, and Munich Germany.

Martin has been to Munich Germany, and also visited a customer in Siena Italy.

Feedback

We have received feedback (in person, in Munich!) that our stereo separation of the channels was a little dizzying. Martin will be trying to make it less severe to relieve this effect.

Thanks for the feedback; We want to hear more!

Follow Up

Martin and Marna talked to the developers of the DocBuddy app, about their new release.

The latest release is 2.0.1, and has added a lot of social aspects (and fixed reported problems). The ability to look up messages is still there, but z/OS V2.3 isn’t there yet. We anticipate it will be coming soon, though, as it is important.

You can sign into the app (very easy to do!), and subscribe to Products and People, and discover them. People are “Influencers” and can be subscribed to. Products doesn’t include all the core z/OS elements, but Communications Server is there and has been active.

Feedback can be given, it’s an email under “Settings”, which took us a while to find.

Mainframe

Our “Mainframe” topic discusses a new z/OS V2.3 function, Coupling Facility Encryption, with Mark Brooks, Sysplex Design and Development. Mark talked about this latest capability in Sysplex, which has been getting a lot of attention as part of the larger Pervasive Encryption direction.

Mark explained that CF Encryption means that the customer’s data is encrypted by XCF in z/OS, sent along the link as encrypted, and stored as encrypted on the Coupling Facility.

z/OS sysprog needs to set it up by:

• using new keywords on CFRM policy on a structure by structure basis

• putting it in your CFRM couple data sets, and the policy change will be pending

• rebuild the struture (to get it from unencyrpted to encrypted)

• DISPLAY XCF structure commands can be used to see what the policy has, what the structure currently is, and the form of encryption used.

List and Cache structures contain customer sensitive data, the XCF control information will not be encrypted because it is not sensitive customer data. Lock structures don’t contain sensitive customer data, and are not encryptable.

Software requirements:

• z/OS V2.3. Strongly recommend not using CF encryption in production until fully at z/OS V2.3.

• ICSF. Need to have ICSF to generate keys and talk to the crypto cards. Every system in sysplex needs to be running with the same AES master key (meaning, same PKDS), note this requirement!

• XCF generates the key from ICSF services and stores that wrapped key inside the CFRM couple data set.

Hardware requirements:

• CPACF

• Encryption is performance sensitive, because it is extra work to encrypt and decrypt. You want it to be executed quickly. Encryption is host-based, and zEC12 has these facilities, however the older machines are not as fast as a z14. Take that into consideration.

Tooling:

• zBNA looks at new SMF data, so that you can see the amount of data transferred to the CF. From there, you could judge the cost of doing the encryption.

• SMF 74 Subtype 4 records contain the new information on the amount of data, via measurement APAR OA51879 on z/OS V2.2. Planning can begin on z/OS V2.2 with this APAR.

For more information, see the z/OS Setting Up a Sysplex.

Performance

Martin talked about MSU-related CPU fields for doing software pricing analysis. Some of these fields are used by SCRT in support of the new Mobile function.

Most notably the fields cover:

• Mobile Workload Pricing (MOBILE), using a new WLM mechanism, is available to many customers today. (The old way of doing Mobile Workload Pricing has been around for quite a long time.) Note a misrecording of IMS Mobile CPU at the Service Class Period level, which is fixed in IMS V15 APAR PI84889 and IMS V14 APAR PI84838.

• CATEGORYA and CATEGORYB: These are just placeholders for any future additional pricing options that come about.

These categories of CPU/MSUs are brought to life using the Workload Manager ISPF panels, using a new reporting attribute. You scroll twice to the right to get there. The values in the field can be MOBILE, CATEGORYA, or CATEGORYB.

Container Pricing is another pricing model, ….., and maybe another topic on that later.

The overall idea: Be aware of the new fields that you will be analysing. These fields are available at two levels:

• At the System level, as Rolling 4 Hour Average numbers – in SMF 70 Subtype 1.
• At the Service Class Period level, as interval-based numbers – in SMF 72 Subtype 3.

Topics

Our podcast “Topics” topic is about adding chapter markers to the MP3 file for podcast apps. This makes it nice to skip from one section to another easily. Our podcast has five sections, each with its own graphic and chapter.

Copyright on the MP3 specification expired finally in 2017, allowing more things to be done with the format. Chapters is one of them.

Martin adds the chapter markers into the MP3 file after doing the audio editing with Audacity. He then takes the MP3 file, and runs it into another tool on iOS called Ferrite. Audacity doesn’t have the ability to mark chapters (or to add the graphics), but Ferrite does. Hence it has to be processed with this second program to give the final MP3 chapters and graphics! (Ferrite was among the first tools to support Chapter Markers, with or without graphics.)

On Martin’s iOS podcast app, Overcast, he sees the chapter markers and graphics fine. Marna uses Android and CastBox, in which she cannot see them. She then tried another Android app, PodcastAddict, which claimed to have chapter support, and yet she still doesn’t see them. So it goes. 🙂

Customer Requirements

RFE 100505 Specify DSN HLQ for Healthchecker Debugging

The quoted description is: When the Healthchecker DEBUG option is turned on, the Healthcheck needs to write to a dataset. If that HLQ is not defined to security (in the case of Top Secret), the Healthcheck will fail. The customer then has to get this HLQ defined and appropriate access granted. I would like the customer to have the ability to control the DSN that the Healthcheck writes to.

Our discussion:

• Those checks which are added by the check writer themselves, usually during initialization with HZSADDCK service, set the HLQ. Right now, these checks are not changable by the user for DEBUG, which is what the requirement is all about.

• Sounds helpful, and desirable to have control of the high level qualifier(s) of the data sets. We agree.

• A likely solution would be to put it in HZSPRMxx, and allow the customer to control it (hopefully) across several checks.

Where We’ll Be

Martin will be in Whittlebury UK 7-8, November for GSE UK.

Marna will also be in Whittlebury UK 7-8, November for GSE UK with Martin. Then, at the big IBM Z event Systems Technical University in Washington, DC, 13-17 November. Then, in Milan, Italy 28-29 November for a System Z Symposium.

On The Blog

Martin published one blog post, which highlights a screencast of his:

• Screencast 11 – DDF Spikes

Marna has an idea for a blog, but needs more time to do some testing for it. It will be coming!

Contacting Us

You can reach Marna on Twitter as mwalle and by email.

You can reach Martin on Twitter as martinpacker and by email.

Or you can leave a comment below.

(Originally posted 2017-10-24.)

It’s been a month since I last did a screencast – and boy what a busy month I’ve had.

And in that month I had the privilege of working with a very nice customer, exercising my DDF Analysis code.¹

As so often happens, a graph or two come together to tell a nice story. One I shared with the customer, and one I’m sharing with you.

Because graphs are involved the natural medium for sharing it is a short video, so that’s what I made. You can find it here.

I think the background to the topic is quite important, so I preface the actual customer case with some educational material. Indeed there is a key point I’m keen to get across:

Guarding against ill-behaving (or “feral”, as I like to call it) is important. It’s insufficient to rely on Security mechanisms and DB2 controls to avoid feral DDF misbehaving. And misbehaviour matters. In the example I give it’s CPU – both GCP and zIIP – that is consumed by the engineful². But it might be precious DBATs, or other DB2 resources.

To clarify, while most people are interested in CPU as it relates to capacity planning (and software cost), I’m more worried about bursts of CPU affecting critical infrastructure. Though, I’ll admit, a (thick) veneer of “low value” DDF usage through a protracted period is worth managing down.

But this isn’t an easy phenomenon to control, particularly for egregious short-commit-scope work (aka bursts of small transactions).

One of the key sources of data is DB2 Accounting Trace (SMF 101) because it gives much finer timestamp granularity than RMF SMF. Further, the ability to identify a spiky consumer is pretty good. I advocate summarising at the 1-minute level, perhaps breaking out by IP address or Authid. This is realistic for me as I’ve built some nice DFSORT-based code to do the analysis.

The graphs come from CSV files created by this code. The question is whether you, dear reader, have access to a way of summarising individual 101 records in this way. I would assume, for example, MXG would allow you to. But I can say that the DFSORT code I use is very fast and very light.

Anyhow, I hope you enjoy the video (and the others in the series).

(Originally posted 2017-09-06.)

So this is a shorter episode, much to Marna’s pleasure. (Personally I’m indifferent to show length, regularly listening to episodes of other podcasts that run to 1.5 to 2 hours.)

It was very good to have a guest: Barry Lichtenstein. (I kept in the bit where I mispronounced his name, as I thought it a funny mistake[1]. You’ll find another piece of flubbing, again because it was funny.)

Enjoy!

Episode 15 “Waits And Measures” Show Notes

Here are the show notes for Episode 15 “Waits and Measures”. The show is called this because our Performance topic is about LPAR weights, and because this episode was after a seasonal hiatus.

Where we’ve been

Martin has been to nowhere in person, but has talked on the phone to several interesting locations.

Marna has just returned from SHARE in Providence, RI and from Melbourne, Australia for conferences.

Mainframe

Our “Mainframe” topic discusses a small new function in z/OS UNIX that there were customer requirements for, and that might not have been highlighted as much as other new functions in z/OS V2.3: automatic unmount of the version “root” file system in a shared file system environment. Our guest was Barry Lichtenstein, the developer of the function, and he told us all about it:

• there is a new BPXPRMxx VERSION statement parameter, UNMOUNT. This means that when no one is using that “version file system” (the new name for the “version root file system”!), it will be automatically unmounted. This is not the default. Syntax here
• this function is nice, as it will allow an unused version file system to be “rolled off” when you don’t need it anymore. Unused here, means that no system is using it or using any file system mounted under it. z/OS UNIX will do this detection automatically, and unmount not just the version file system, but mounted file systems under it that are no longer used by any systems after an unspecified amount of time.
• you can turn this on and off dynamically with a SET OMVS or SETOMVS command. There is DISPLAY command support of it. And perhaps the best news, the health check USS_PARMLIB will see if the current settings don’t match the used parmlib specification. (Marna thinks this check is the gold standard for using dynamic commands and not regressing them on the next IPL with the hardened parmlib member!)
• we weren’t sure if SMF record 92 would be cut when the unmount happened, but Barry said nothing unique was happening for this function so what happens today is most likely the same behavior. There are messages that are issued in the hardcopy log when the unmounts happen. SMF 90 might be issued for SET changes.

Performance

Martin talked about Weights and Online Engines in LPARs, and Martin again looking at customer information.

• Intelligent Resource Director (IRD) changed PR/SM worked:
  1. Dynamic weight adjustment
  2. Online logical engine management (vary online and offline). Shows minimum and maximum weights when changed by IRD.
• HiperDispatch: took away logical engine management (and manages it better!), and kept IRD dynamic weight adjustment. With HiperDispatch’s parking of engines, no work is directed towards it. An affinity node is a small group of logical engines to which a subset of the LPAR’s work is directed.
• More instrumentation was introduced, such as Parked Time and refined instrumentation on weights (vertical weights, by engine).
• Customer situation was that they did their own version of IRD and HiperDispatch: Varying logical engines online and varying weights (not using IRD itself). Martin expected IRD to change weights, but he saw the IRD weight fields were all zero. You must look at the “inital weights”, which means initial since you last changed it.
• Why not let IRD do it? Martin thinks there was something in the customer’s mind to control it themselves, perhaps somethind other than WLM goal attainment (which IRD would adjust weights for).
• Why not use HiperDispatch? Martin thinks that maybe a subtle difference might be needed, but LPARs should be designed properly. One possible aim might be to fit in one drawer, for instance. Maybe it’s not understanding what HiperDispatch does.
• How did the customer adjust the weights? It was an open question. Probably via BCPii? Feedback would be welcome on this.
• As an aside, with IRD a change in weights would lead to HiperDispatch recalculating how many Vertical High, Medium, and Low logical engines each LPAR has.

Lesson learned: Assumption on how something has dynamically changed may not always be correct.

Topics

Our podcast “Topics” topic was “Video Killed the Radio Star?” and about screencasting.

Martin has been trying to post screencasts to YouTube. Here’s one.

Screencasts are not videos where you see the speaker. It’s just a visual of what is happening on a screen with a talkover.

The best candidates are graphs and program output. Martin uses these steps to create these screencasts:

1. First, make a set of slides or images. Annotations are good to use, to point to a particular feature on the screen. (They don’t have to be animated.)

2. Use the screen recorder to add sound to the slide. (In PowerPoint, record the slides.)

3. Editing, with proper fadeouts. Split the audio out and clean it up with Audacity.

4. Re-unite the audio and the video. Camtasia, while expensive, has some promise.

5. Publish on Youtube.

Customer Requirements

As well as Barry’s mention on the v/OS V2.3 automatic unmount of version file system requirement (RFE Number 47549, “Automatic disposal of z/OS UNIX version root”), there was another customer requirement we discussed:

RFE 97101 Make /dev/random available on z/OS without ICSF dependency

The quoted description is:

/dev/random is a special file that serves as a psuedo-random number generator source for applications. On z/OS, this special file is only provided if ICSF is started. If ICSF is not available, we need to resort to some other source of random numbers (which will have to be implemented within applications). Goal here is to make /dev/random available on z/OS, independently of whether ICSF is available or not.

Our discussion:

• It’s a major migration action in z/OS V2.3 to have ICSF available for /dev/random.
• ICSF is a dependency for many functions. It’s important to have on every single z/OS system.
• Another aspect: each user had to have authority to use these ICSF services for certain of the functional depedencies (including /dev/random).

Martin mentioned that random number generators vary in quality, and behaviour. He hopes, if this were done, it would be high quality. One criterion would mean a close enough match to the ICSF-based algorithm, distributionwise.

Where We’ll Be

Martin will be in middle of Italy in August, 2017. He is threatening to drive from Italy to the Munich zTechU conference.

Marna will also be in Munich too. Marna is going to Johannesburg for the IBM Systems Symposium, aka IBM TechU Comes to You. She is also going to Chicagoland area on Sept 26 and 27, 2017 for some area briefings.

Both Martin and Marna are hoping to do a poster session in Munich, which should be jolly good fun.

On The Blog

Martin has actually not published a blog recently!

Marna actually did publish a blog recently!

• Some handy z/OS V2.3 and z14 information

Contacting Us

You can reach Marna on Twitter as mwalle and by email.

You can reach Martin on Twitter as martinpacker and by email.

Or you can leave a comment below.

(Originally posted 2017-07-14.)

If a customer has Transaction goals[1] – for CICS or IMS – it’s possible to observe the CPU per transaction with RMF.

But you have to have:

• Transaction rate
• CPU consumed on behalf of the transactions.

This might seem like stating the obvious but it’s worth thinking about: The transaction rate and the CPU consumption have to be for the same work.

Now, a Transaction service class doesn’t have a CPU number. Similarly, a Region service class doesn’t have transaction endings.

So you have to marry up a pair of service classes:

• A Transaction service class for the transaction rate.
• A Region service class for the CPU.

Operationally this might not be what you want to do. Fortunately, you can do this with a pair of report classes.

There’s another advantage to using report classes: You can probably achieve better granularity – as you can have many more report classes than service classes[2].

So I wrote some code that would only work if the above conditions were met[3].

Unimaginitively my analysis code is called RTRAN; You feed it sets of Transaction and Region class names.

Perhaps I should’ve said you could have e.g. a pair of Transaction report classes and a single Region report class and the arithmetic would still work[4].

But why do we care about CPU per Transaction?

There are two reasons I can think of:

• Capacity Planning – by extrapolation
• Understanding what influences the CPU cost of a transaction

From the title of this post you can tell I think the latter is more interesting. So let’s concentrate on this one.

In what follows I used RTRAN To create CSV files to feed into spreadsheets and graphs[5]. Over the course of a week I captured four hills while developing RTRAN.

The first thing to note is that CPU per Transaction is not constant, even for the same mix of transactions.

This might be a surprise but it makes sense, if you think about it. But let’s think about why this could be.

Two Important Asides

But first a couple of asides on this method:

• Take the example of a CICS transaction calling DB2. While most of the work in DB2 is charged back to CICS not all is: There is a significant amount of CPU not charged back[6]. It’s highly likely the DB2 subsystem is shared between CICS and other workloads, such as DDF and Batch; It gets much less satisfactory trying to apportion the DB2 cost so I simply don’t.

• Likewise, I’m ignoring capture ratio. While it would be wrong to believe it’s constant, for most of customers’ operating range it’s a fair assumption to go with a constant value for capture ratio.

In a nutshell, both these asides amount to “this is not a method to accurately measure the cost of a transaction but rather to do useful work in understanding its variability.”

Why CPU Per Transaction Might Vary

I’m going to divide this neatly in two:

• Short-Term Dynamics
• Long-Term Change

Short Term Dynamics

CPU per transaction can, demonstrably vary with load. There are a couple of reasons, actually probably more. But let’s go with just two:

• Cache effects, that is more virtual and real storage competing for the same scarce cache.
• If a server becomes heavily loaded it might well do more work to manage the work.

But it’s not just homogenous variation; Batch can impact CICS, for example.

Look at the following graph:

In this case it’s the lower transaction rates that are associated with the higher CPU per transaction. But not all low transaction rate data points show high CPU per transaction.

A tiny bit more analysis shows that the outliers are when Production Batch is at its heaviest, competing for processor cache. It’s also the case that these data points are at very high machine utilisation levels, so the “working more to manage the heavy workload” phenomenon might also be in play.

Long Term Change

“The past is a foreign country; they do things differently there” L. P. Hartley The Go-Between.

Well, things do change, and sometimes it’s a noticeable step change, like the introduction of a new version of an application, where the path length might well increase. Or, perhaps, a new release of Middleware[7]. Or, just maybe, because the processor was upgraded[8].

But often, perhaps imperceptibly, CPU per transaction deteriorates. For example, as data gets more disorganised.

Conclusion

If it’s possible to do, there’s real value in understanding the dynamics of how the CPU per transaction number behaves.

Try to understand “normal” as well as behavioural dynamics, and watch for changes.