(Originally posted 2018-05-27.)

I must be mad to do what I do… 🙂

Specifically, I’m talking about maintaining code to process SMF records, rather than relying on somebody else to do it.

Recently I sat down with someone who is getting started with processing SMF. In fact they’re building sample reporting to go against a new piece of technology that already maps the records. My contribution was to help them get started, given my knowledge of the data.

Our Infrastructure

Our infrastructure consists of two main parts:

1. Building databases
2. Reporting against those databases

This post is almost entirely about the former. In passing, I would note any SMF processing tools have to sit in a framework where whatever queries you run can turn into useful work products – such as tabular reports, graphs, and diagrams.¹ In our case our reporting is essentially a mixture of REXX and GDDM Presentation Graphics Facility (PGF), both of which the query engine integrates well with.

An Example: SMF 70 LPAR Data

SMF 70 Subtype 1 is the record that contains system- and machine-level CPU information. It’s a pretty complicated ² but very valuable record.

So let me describe to you how to get LPAR-level CPU, which is a fundamental report – and the one I showed my colleague how to do.

SMF 70–1 has a section called the “Logical Processor Data Section”. In a record you get one of these for each logical engine configured to each LPAR on the machine.³ You also get one for each physical processor -in pseudo-LPAR “PHYSICAL”.

Essential fields are the Virtual Processor Address (VPA), Physical Dispatch Time (PDT), and Effective Dispatch Time (EDT). PDT is EDT plus some PR/SM cost. Depending on your needs, you probably want PDT when calculating how busy the logical processor is. You probably also want to know which processor pool it is in – CIX, and how much of the RMF interval the processor was online for (ONT).

Most customers define some offline engines for each LPAR – so actually ONT is important.

For a modern z/OS LPAR GCPs have CIX=1 and zIIPs have CIX=6.

This is really valuable information, but there’s a catch: The Logical Processor Data Section doesn’t tell you which LPAR each section is from. So you have to get that from elsewhere:

For each LPAR on the machine there is a Logical Partition Data Section. As well as the name, there is a field that tells you which Logical Processor Data Section is the first one for this LPAR, and another that tells you how many there are.⁴ So we use this to put LPAR-level information into a virtual record we build from the Logical Processor Data Section.

We store information summarised at the LPAR level, as well as at the individual logical processor level. Importantly, the LPAR-level summary is a the Pool level, rather than conflating GCPs and zIIPs.

PDT, EDT, and ONT are times. To convert these to percentages of an interval I need to extract the Interval length (INT. But this is in neither the Logical Processor Data Section nor the Logical Partition Data Section. It is in the RMF Product Section. So I need to extract it from there. But INT is a time in a very different format from the others. So some conversion is necessary.

I’ve grossly simplified what’s in the relevant sections – restricting myself to the one problem area: LPAR Busy.

Decisions Decisions

We do our data mangling as we build our database.

Our code, by the way, is Assembler. I inherited it about 15 years ago and have extensively modified it as the CPU data model has evolved. If the above sounds complicated I’d agree with you. Register management is a real issue for us. Perhaps I should convert to baseless. But of course that carries a major risk of breakage.

So that’s our design: “Mangle and store”. But that’s not the only design: You could do the mangling every time you run the query. I would suggest that can lead to a maintenance burden and a risk of loss of consistency. So, if you were doing this in SQL you’d probably want a view.

And if you did mangle the data every time you run such a query you’d want to be careful about performance. Fortunately, SMF 70 isn’t a high volume record. So maybe performance isn’t critical.

Of course with SQL you could mangle and store, querying the resulting table. That’s probably the best design for any new SMF processor. By the way, I’ve no idea what other SMF processing tools do.

And do you throw away the raw, unprocessed data tables? I would say not; We don’t.

Parting Shorts 🙂

There are lots of people out there, including several from vendors, who claim to be able to process SMF data. If all they do is map the records, be very careful as you’ll have to do the hard work yourself.⁵

In any case, there is mapping and there is mapping right. I’m not sure how you’d know the difference – until you installed it and tried to use it.

I would claim – for the reporting and SMF processing I maintain⁶ – my experience of the actual data is invaluable. And this experience transfer session proved to me that even beginning to replicate what the RMF Postprocessor produces takes all that experience. Anyone who just knows RMF reports will have a hard time with the raw data.

Beyond the mapping is, of course, reporting. Sample reports are vital here – and well-documented ones at that. And a thriving community of people using and sharing comes a close second.

I think my contribution to “community” is through blog posts like this one, podcasting and presenting at conferences⁷. Little of that would be possible without my real life experience of mangling 🙂 data. And, no, I don’t think I’m giving anything regrettable away by telling you how I process data.

In short, I must be mad to maintain our code – until you consider the alternative: Blissful ignorance. Blissful right up to the point where real work has to get done. 🙂

(Originally posted 2018-05-12.)

In my “How To Be A Better Performance Specialist” presentation I have a bullet point that says ‘there’s no value in being “The Man Who Knew Too Little”’.

Well, it astonishes me how often I am the man who knew too little. 🙂 I guess we keep on learning OR ELSE.¹

In the spirit of “I’ve become sensitised to something so you should, too” I’ll confess that I live in a comfortable bubble where ignorance of some realities is bliss. This post is about that, but also a wider point.

I don’t know if you’ve ever seen a real mainframe, preferably with its clothes off 🙂 (or doors open at least). If you haven’t, I really think you should. We talk about machines in an idealised way: “This many engines” or “that much memory”. But it’s nice to actually see one – and not just the hollowed out ones on display at conferences or IBM labs.

Recently I enjoyed some time in the Singapore Plant (which I will call “84” – as that’s the only manifestation of it in the data). I was shown a ZR1 – which is indeed narrow², being in a 19” rack. I also got to see – and not for the first time – a z13 and also a z14. I can now tell one from the other, without seeing the badge.

By the way, there is a “16U” space for you to potentially put your own stuff in the ZR1. It’s time us mainframers learnt a bit about (the aforementioned) 19” rack terminology. “16U” is 16 standard units high, by the way.

But let me come to the point.

Lesson 1: Physical Hardware Considerations Matter

We’ve been talking to a customer about coupling facility ICA-SR links. We think they should ideally have double the number. But two bites of, ahem “reality bites”:

1. They are on zEC12 hardware now and doubling the number of links can’t be done.
2. When they get to z14 hardware there are drawer-level limits to the numbers of links.³

Figuring out which device type a machine is from SMF is trivial. ⁴ What isn’t trivial is working out how many drawers. You would have to do it from the model. At last, with z14 it’s straightforward:

• If ZR1 it’s 1 drawer.
• If M01, M02, M03, M04, it’s 2, 3, 4 drawers, respectively.
• If M05 it’s also 4 drawers. But these are different drawers.

And we have considerations for upgrades. So, M01, M02, M03, M04 can’t be upgraded to M05.

We know, based on the model number, the limit of characterisable or “customer” PUs.

ZR1 poses an additional difficulty: There are four different feature codes for the maximum number of customer PUs: 4, 12, 24, 30. So it’s not good just saying “it’s a ZR1”.

Fortunately there is a new field in SMF 70, introduced by APAR OA54914. It is SMF70MaxPU, which is how many processor cores are physically available for this particular machine.⁵

Another example of “hardware considerations matter” is the upgradability of processors. For example, from December 31, 2016 you couldn’t order upgrades to zEC12 that required physical hardware.

Lesson 2: How Something Actually Behaves Matters

Some people would consider I was in Marketing. While I don’t dissent from that, I’d like to think I was quite technical⁶. One of the things you get “in the field” is lots of marketing material, much of it reasonably technical, on product capabilities. While we can readily absorb quite a lot of this, it doesn’t really come to life until you see it in action. In a real customer situation.

I’m not sure this is a good example of it, but it’s one I came across recently: Mobile Workload Pricing (MWP) is, in principle, a nice software pricing scheme to reduce the cost of Mobile work on z/OS. You agree with IBM how Mobile work will be measured and this is factored into your software bill.

That last sentence is incredibly high level – and that is deliberate. It’s actually pretty close to how some people understand MWP. But to really understand what it can do for you takes a much deeper understanding.

In particular, what counts is how big the Mobile work is at the Rolling 4 Hour Average CPU peak. In a recent case this was in the middle of their nightly batch. This is an environment that serves essentially a single time zone. Unsurprisingly, there isn’t a lot of Mobile work at that time of day, so the benefit of Mobile is relatively small.

So this is an example of how something actually behaves mattering. Fortunately we have instrumentation that speaks to this.

By the way, Mobile is an example of something where a financial benefit might distort the architecture: Do you really want separate Mobile CICS or IMS regions? Fortunately, WLM APAR OA47042 and corresponding CICS TS 5.3 support made Mobile classification of transactions much easier. But many installations implemented Mobile with separate regions. A few went with the more intensive method of measuring individual transactions. I don’t know, by the way, of customers using separate Mobile LPARs – which remains a possibility.

The reason for that last paragraph is there’s another point: Architecture matters – and especially how it behaves.

Minor factoid: The XML element in the WLM policy that governs this new Mobile capability is Reporting Attribute. I know this because the customer kindly sent their WLM policy to me in XML format – and I coded my parser accordingly.

How Hard Are These Lessons To Learn?

I always feel it’s lame to end a piece – whether a presentation or a blog post or anything else – with the words “In Conclusion” so I won’t.

I’ll just reflect where I am with it:

I’ve said I’m more a software person than a hardware one – but that’s because I can readily make software do my bidding. I’m increasingly sensitised to hardware configuration aspects but there’s a lot to learn there and not all of it is relevant. Further, I like machines to be self documenting – and the data model is not exactly complete. But APAR OA54914 takes us a little further.

“How stuff actually behaves” is one where you really have to see data. In the case of Mobile I’d say that data is a combination of what the terms and conditions say, in particular the Mobile contract exhibit and the scheme itself, plus performance data.

In both cases this is really about experience. More than 30 years in, I’m still learning at a very rapid rate through formal materials and, equally importantly, customer experience. I guess this is my version of On Practice. And I guess learning is the real point. Which is handy, as we’re just about to kick off System Z Technical University in London – and then I go to do the other thing with a customer workshop.

And I count myself as very lucky to know so many diverse customers around the world.

(Originally posted 2018-04-21.)

Maybe it’s because I know nothing about DB2 buffer pools that I’ve never written about them.¹

Actually, that wouldn’t be true. I would say I know a fair amount about DB2, but I’ve taken a bit of a step back from it – as we have a DB2 Performance specialist in the team. But not too much of a step back, I hope.

So, why write about DB2 buffer pools now? Well, I had an interesting conversation about them the other night.²

The actual conversation was about Buffer Pool Hit Ratio calculations – which are a little weird, to say the least.

But, having discovered I hadn’t blogged on the subject, I’ve decided to braindump on the subject.³

A Simple Buffer Pool Model

In a simple buffer pool setup, an application requests pages⁴ from the buffer manager. The buffer manager retrieves them from the buffer pool if they’re there. Otherwise they are retrieved from disk.⁵ If a page is retrieved from disk it is loaded into a buffer, potentially displacing another page. In simple buffer pool models, the buffer pool manager deploys a Least Recently Used (LRU) algorithm for discarding pages.

So a hit percentage is easy to calculate: It would be hits/(hits+misses) turned into a percentage. A hit here is where the page was retrieved from the buffer pool and a miss is where it came from disk. It’s meaningful, too.

But DB2’s Buffer Pool Model Is Anything But Simple

Here are some ways that it isn’t simple:

• Pages aren’t necessarily read on demand.
• Buffer pool management isn’t necessarily LRU.

This is not an exhaustive list of the ways that DB2 buffer pool management is complex; I just want to give you a flavour.

Prefetch

Suppose you knew that the page that was asked for was the first of several neighbouring pages that would need to be retrieved from disk. Then you could more efficiently retrieve them as a block. Not just because you could do it in fewer I/Os but also because you could retrieve the second and subsequent pages while the first was being processed. Or something like that.

This is actually what DB2 will do, under certain circumstances. This technique is called Prefetch. DB2 has three types of Prefetch: Sequential, List and Detected (or Dynamic).

The first two are generally kicked off because DB2 knows before it executes the SQL that prefetching is required. The third is more intriguing. As the name suggests, it happens because DB2 detects dynamically that there is some benefit in prefetching.

In any case, more pages are read than the initial application request asked for.

Suppose DB2 read a page that the application never wanted. That would do something strange to our naive buffer pool hit ratio calculation. And the various flavours of prefetch can lead to that. There is even the phenomenon of a negative hit percentage.

But relax; These are just numbers. 🙂

By the way, just to complicate things, “neighbouring” might not mean “contiguous”. It might mean “sufficiently close together”. (Contiguous is, of course an example of that.) List Prefetch, in particular, is gaining efficiency without contiguousness.

Buffer Pool Page Replenishment

I mentioned the Least Recently Used (LRU) algorithm above. While DB2 generally does use this method of page management it needn’t and you can control this. There are two other algorithms available to you:

• Preloading
• First In First Out (FIFO)

The LRU algorithm is computationally a little expensive – as you have to keep track of the “page ages” – that is how long it was since each page was last referenced. The other two algorithms are simpler and thus less expensive.

To be fair, most customers stick to LRU buffering, which is the default.

DB2 Buffer Pool Instrumentation

As you probably know, DB2 has instrumentation at various levels of detail. Two are of general interest:

• Statistics Trace – at the subsystem level
• Accounting Trace – at the application level

Let me summarise what these can tell us about buffer pools – and how they behave.

Statistics Trace

Statistics Trace (SMF 100 and 102) is the main instrumentation for understanding subsystem performance themes, for example virtual storage.

Of most relevance to this post is the instrumentation in support of buffer pools. You get detailed instrumentation on each individual buffer pool. By the way a typical subsystem can have anywhere from half a dozen to dozens of buffer pools.

As well as configuration information – sizes, names and thresholds – you get counts of events. For example Synchronous Read I/Os.

If you were processing raw Statistics Trace records you might easily get confused: The counters are totals from when the subsystem was started up to the moment the record was cut. So, to get rates, you have to subtract the previous record’s counts from those in the current record. A typical calculation would be the delta between the count in the first record in an hour and the last in the same hour. There used to be a fairly severe problem with this:

Record cutting frequency is governed by the STATIME parameter. It used to default to 30 minutes. With a STATIME of 30 if you subtracted a counter in the first record from the same counter in the last record in the hour you’d underrepresent the hour’s activity by about 50%. Fortunately the default value of STATIME dropped to 5 and finally to 1 minute. So the underrepresentation is under 2%. Dropping the STATIME default from 30 to 1 does produce 30 times as many SMF records but they are cheap to produce and small⁶.

This might be extraneous detail but it used to be possible for counters to overflow – in fact wrap – but the counters are now 64-bit instead of 32-bit.

One stunt I used to pull was to detect if the value of a counter dropped. That would indicate a DB2 restart had occurred.⁷ Nowadays, of course, I use SMF 30 for the same purpose. And in fact I don’t tend to see DB2 restarts outside of IPLs – much.

Statistics Trace doesn’t contain any timings; For that read on.

Accounting Trace

Accounting Trace (SMF 101) is much more straightforward. And more detailed.

Let’s take a simple example. A CICS transaction, accessing DB2, will generally cut a single “IFCID 3” Accounting Trace record. As well as many other useful things, such as timings, the record contains Buffer Pool Accounting sections.

Each Buffer Pool Accounting section documents the activity to a single buffer pool from this transaction instance. So you can tell, for example, how many synchronous read I/Os were performed – at the buffer pool level – for this transaction.

If you wanted to analyse the buffer pool behaviour as it relates to a transaction you would:

1. Ascertain the time spent waiting for synchronous read I/Os, the time waiting for asynchronous read I/Os, and the time waiting for asynchronous write I/Os.
2. Examine the counters in the Buffer Pool Accounting sections to understand how each buffer pool’s behaviour might contribute to each of the times gleaned in Step 1.

There are a couple of things to note:

• The word “asynchronous” cropped up a couple of times above. A general life lesson is that asynchronous activities aren’t necessarily fully overlapped or done “for free”. Hence the time buckets in Accounting Trace.
• The counters in Accounting Trace are not cumulative – so no subtraction is required.

In short, Accounting Trace is very useful. Obviously, with its level of granularity, it is high volume. But it’s not terribly CPU intensive and is essential for proper DB2 performance analysis.

Establishing A Working Set

If you look at a DB2 subsystem in the first few days and even weeks of its life you’ll generally see something that looks like a memory leak. Namely, that more and more memory is used. I’ve demonstrated this many times to customers. (Apart from bugs) this isn’t the case. It’s what I would call “establishing a working set”. Here are two examples of how this happens:

• Buffer pools generally need populating. In the most common buffer management schemes pages are read in more-or-less on demand. So buffer pools start empty and, as data is accessed, fill up. Perhaps to full, perhaps not.
• Other caches, such as the Prepared Statement Cache (for Dynamic SQL) are similarly populated as used.

The whole area of DB2 Virtual Storage is fascinating, and again well documented by Statistics Trace. However, nowadays (since Version 10) what is of greater interest is how use of Virtual drives use of Real. Hence this section of the post.

Conclusion

This has necessarily been a brief introduction to DB2 buffer pool management. I wanted to introduce you to a few concepts:

• Ways in which DB2’s buffer pool management algorithms are more sophisticated than a simple scheme.
• How the instrumentation works and can be used.
• How it takes some time for DB2 buffer pools to populate and settle down.

I’ve deliberately steered clear of discussing updates and DB2’s strategies for managing writing updates out. I’ve also not covered logging nor DB2 Datasharing – as I wanted to keep it simple.

Wow! “Braindump” is the right word: This got verbose. Well done if you got all the way through it. 🙂 But I’ve covered a lot of ground at a high level. DB2 Performance is a very interesting topic – and really quite extensive. Which is one of the reasons I got into the subject in the first place. And why I like to keep my hand in still.

(Originally posted 2018-04-15.)

This week I went from a very fuzzy “I’ve a sneaking suspicion this WLM policy doesn’t make sense” to a (clear as mud)¹ 🙂 “it makes perfect sense” – so I suppose it’s been a good week. 🙂

Something Clever Our Reporting Does

I keep banging on about² the value of SMF 30, don’t I?

One of the things our code does is to pull out the CICS regions (PGM=DFHSIP) and reports on them. (I’ve mentioned this many times in this blog and elsewhere, so I’ll focus on an aspect you might not know about.)

In Workload Manager (WLM) terms you can manage regions to either a Region goal, or a Transaction goal³. I’ll talk about what these mean in a minute.

So, our code detects which kind of a goal a CICS region is being managed to.

• If all the CICS regions are managed to Transaction goals, we display something like “All Regions Managed To Transaction Goals”.
• Similarly, if all are managed to Region goals, we display “All Regions Managed To Region Goals”.
• Otherwise, our little table gets more crowded as we display the goal type for each CICS region individually.

And it’s all done using SMF 30 (Subtypes 2 and 3) Interval records.

CICS WLM Goal Types

There are, to recap, three CICS WLM goal types. Two have been around for many years; The third is relatively recent. I’ll briefly summarise them here.

Region Goal

Here the CICS region is managed using its own goal, without heed to transaction performance. Generally this is a velocity goal.

The address space, in this case a CICS region, owns the CPU. With Region goals the CPU is associated with the region’s Service Class.

For velocity goals we get samples from RMF. It’s not directly translatable into response times.

Transaction Goal

Here the CICS region is managed to support the goals of the transactions executing in it. Generally these goals are single-period response time goals – and there might be multiple transaction service classes for transactions executing in a single region.

You might ask how this could work, especially with multiple Transaction service classes’ transactions running in the one region. What is really happening is an internal service class is constructed, based on the Transaction service classes. This is what WLM services.

Here the Transaction service class does not own the CPU, the region still does.

In RMF we get a nice response time distribution, around the goal response time value.

Managed To Both

This is relatively new. This is only intended to be used for Terminal-Owning Regions (TORs). In this case the region (TOR) is managed to a velocity goal but the transactions that run in it have their response times recorded as part of their Transaction Goals. This makes sense if the AORs are managed to Transaction goals.

The point here is to manage the TOR above the AORs (Application-Owning Regions) with a velocity goal but still have WLM understand (and RMF report) the actual transaction response times.

To be honest, my code hasn’t yet detected cases of Both.⁴

Something Fishy?

In this customer case I saw all the CICS regions were “Managed To Transaction Goals” – which generally indicates to me I can say all the work in the region is executed with that region’s goal. This matters to me because our code also constructs graphs of how much CPU is used at each importance level. If the region is managed with a Transaction Goal that breaks down.

But

1. I saw no CICS Transaction Class in the policy.
2. I saw no evidence of CICS transactions executing.

2 is obviously a consequence of 1.

So how come we have CICS regions managed to Transaction goals but no actual Transaction goals?

You might construct a hypothesis that the nature of the CICS transactions was that they never end. Mumble mumble CICS Conversational – but I think that went the way of the Dodo. ⁵ And anyway that wouldn’t explain the lack of CICS Transaction classes in the policy.⁶

Hegel might have something to say about that. 🙂

Insufficiently Clever By Half

As so often happens, it was an issue in my code. If there was a bug it was the text it decided to put out, not the logic behind it. And, yes, that would be as a result of fuzzy thinking. 2018 Martin probably has 2015 Martin to thank for that. 🙂

At this point I have to describe the two bit flags in SMF30PF2 that describe this area:

• X‘40’ (SMF30SME) Address space cannot be managed to transaction goals, because “manage region to goals of region” was specified in the WLM service definition.

• X‘04’ (SMF30CRM) If this bit is on, it indicates that the address space matched a classification rule which specified “manage region using goals of both”, which means it is managed towards the velocity goal of the region. But, transaction completions are reported and used for management of the transaction service classes with response time goals. This option should only be used with CICS TORs, the associated AORs should remain at the default “manage region using goals of transaction”.

You can probably tell those two descriptions were taken straight from the SMF manual.

So my code’s logic was:

1. If SMF30CRM was set we say the region is managed to Both. And the “all/some/none” logic applies to the set of CICS regions we see.
2. Otherwise, if SMF30SME is set Region goals are indicated.
3. Otherwise Transaction Goals are in play.

It’s entirely possible to have a region allowed to support Transaction goals but for none to be defined in the Policy. That is exactly what happened here.

And it happened at another customer. I’d publicly thank their sysprog who told me this, but it’s probably better not to do that here, thus identifying the customer. And she knows who she is anyway. 🙂

So how do you get to be in this state? Well, you mark the region as liable to being managed to Transaction goals and then just don’t define any CICS Transaction service classes.

Presumably the legitimate reason for doing that would be in preparation for actually having some Transaction goals. But plans change and some things don’t actually get built.

Meanwhile, the customer is falling back to the Region goal – so I correctly know it’s in the Importance 2 tier of e.g. CPU usage, and it really does have a Velocity goal of 60. With actual Transaction service classes Importance 2 might well be wrong. And it’s even more likely the Velocity goal would not be 60.

Conclusion

One of the lessons from this is life is untidy. WLM policies certainly grow to be. And if you assume, when analysing data, that things are tidy your assumptions can be proven wrong.

Now, I’m not going to rush to make my code look across into SMF 72-3 to look for Transaction goals. It might find some that aren’t relevant to the region at hand. Sorting that out would be rather tricky. Read “error prone”.

So, a human eye can be wary of the “transaction goals in play” statement, and apply judgment.

I probably should also point you to a 2013 blog post of mine: Are You Being Served? It talks about the relationship between serving – in this case Region Goal (serving) service classes and Transaction Goal (served) service classes. More to the point what you can get out of RMF.

And the title of this post? More fully it should be “Just because an address space says it’s managed to transaction goals doesn’t mean it is.”

(Originally posted 2018-03-05.)

”… And then two come along at once.” 🙂

Unlike Billboards or London Buses, only two here folks. 🙂

So, we had ideas (and somehow time) to record two episodes almost back to back. So we did.

But we have no chance of recording together for a while, sadly. Which is a bit of a shame as we have quite a lot we could talk about. Maybe some of mine will come out in other ways. After all it’s been quite a busy few weeks, bloggingwise. The really neglected medium has been screencasting, but I have an idea for something to do with video. ANd a little time on my hands.

Anyhow, I hope you enjoy this episode.

Episode 18 “What we won’t have in common anymore”

Here are the show notes for Episode 18 “What we won’t have in common anymore”. The show is called this because our Mainframe topic is about some removals in common storage planned for the release after z/OS V2.3.

Where we’ve been

This episode was completed right after Episode 17, so we haven’t been anywhere special:

• Marna has been to home and to the office. Several times.

• Martin has been travelling up and down stairs a lot.

Mainframe

Our “Mainframe” topic discusses a future planned removal coming in the release after z/OS V2.3.

• Four items actually form what we’ve titled “User Key In Common Storage Removal”. This has been outlined in a Statement of Direction, and is documented in the z/OS V2.3 Migration Workflow and book.

  1. ALLOWUSERKEYCSA in DIAGxx will be treated as NO.

  2. Removal of support for obtaining User Key CSA

  3. Can’t change ESQA storage to User Key

  4. Can’t create a SCOPE=COMMON Dataspace in User Key.

• New capabilities with OA53355, which is at z/OS 2.1 and above:

  1. SMF 30 has been enhanced to identify jobs/steps that use user key common storage. There is a new set of bits whose names begin with SMF30_USERKEY*, specific to each disallowed item after z/OS V2.3.

  2. SLIP Trap Capability. Single slip trap covering the removed items. Provides relief from “not multiple PER Type SLIPs” constraint.

  3. A new z/OS Health Checker for z/OS health check, ZOSMIGV2R3_NEXT_VSM_USERKEYCOMM. Issues an exception message when use of user key common storage is detected.

• An important point is that when this happens CICS users will have to be at CICS V5.2 or later. CICS V5.1 will be out of service by that time, on July 1, 2019. Laggard regions will need to be upgraded. The SMF 30 record’s Usage Data Section helps you – across your estate – check CICS levels. (Product Version is correctly filled in for CICS regions.)

Performance

Martin talked about what’s in a data set name and this topic is about getting insight by parsing data set names. His reporting code, which processes SMF 14 (OPEN for read) and SMF 15 (OPEN for write) records, bolds data set qualifiers (or segment names) if they match any of a bunch of criteria. Now, new criteria have been added to investigating data set names.

The purpose of bolding in the reporting is to alert the specialist and the customer to significant portions of a data set name.

Previously, the only criterion was job name, which most notably show up in temporary data sets.

Also previously, the code played formatting games with GDG generations – converting to relative generations – and listed all the generations seen in each running of the job.

The new criteria are job step name, dates (in lots of formats), job step program name, “SORT” in the qualifier, and system SMFID.

While the code doesn’t see “partner” job names (for instance, where one writes to a data set and the partner reads from the data set), those could be detected using the Life Of A Data Set (LOADS) technique, which we can talk about more one day.

And surely there will be more interesting finds in other customers’ data set names. As they are stumbled upon the code can be enhanced to learn these new tricks – as Martin refactored it to make it easier to add new criteria.

For example, in the “real life” example in the cited blog post (What’s In A Name? – Revisited) “PR” might very well mean production. and the low level qualifier “T1700” might or might not mean a time. These he considers tenuous parsings right now.

Topics

Our podcast “Topics” topic is about two cheap and fun pieces of hardware Martin has been playing with: Raspberry Pi and Arduino

• Arduino is more hardware oriented. It is an open source hardware board. It is a microcontroller you download small programs to from e.g. a PC, written in a flavour of C. It is optimised for driving electronics, e.g. on a breadboard (which you can plug components into) and boards you plug into the top of the Arduino (called shields).

  Martin has three Arduino shields: An SD Card reader/writer, a 7-segment display, and graphical LCD display.

• Raspberry Pi is more for software. It is a small cheap computer on a card.

  Martin’s has 4 USB ports, HDMI, Ethernet, Wifi, and runs Raspbian (a flavour of Linux (Debian)). You can run other builds. He bought an HDMI monitor for it (HP 27es).

  He also found a Logitech K780 keyboard and a Logitech M720 mouse. Both of these can be switched instantaneously between 3 computers, connected via Bluetooth or USB. They can be used with Windows or Linux or Mac, and others.

• Spending on accessories can add up considerably, with three shields, and lots of electronic components. He had to work out how a switch worked, for example, bringing him back to his youth. Resistor bands came back to him quickly. Soldering was difficult then and now . Martin considered electronics to be more like following recipes. One learning point is he was more interested in software in his youth. He cut his “assembler teeth” programming the Z80 processor then (in the early 1980s). Also the Z80 support chips – PIO, SIO, CTC etc. His first computer was a Z80-based Amstrad CPC 464.) Raspberry Pi fits the bill now.

• He hasn’t discovered many practical uses. But it’s something to practice things like SSH with. However, it’s feasible to run “jobs” on the Raspberry Pi, “submitted” from iOS that can’t actually run on iOS.

• Raspberry Pi has captured Martin’s heart more than Arduino, but he’s intrigued at the idea of combining the two.

Where We’ll Be

Martin will hopefully be in Copenhagen, Denmark visiting a customer.

Marna will be at SHARE in Sacramento March 12-16 , and in Cairo IBM TechU April 15-17.

On The Blog

Martin referenced one blog post from the Performance topic:

• What’s In A Name? – Revisited

Contacting Us

You can reach Marna on Twitter as mwalle and by email.

You can reach Martin on Twitter as martinpacker and by email.

Or you can leave a comment below.

(Originally posted 2018-03-01.)

This episode really has been a long time coming. If I reflect on why, there are two things…

• The main one is our schedules. Marna and I, as you’ll see, have been all over the place.

• The minor one is that – until about a month ago – I wasn’t on a creative streak. Nothing major, it happens. And I’m certainly on one now.

We certainly hadn’t got bored of doing this – and I hope you, dear listener, hadn’t got bored of us.

We enjoyed recording this one, so much so that we’re “going back into the studio” 🙂 straightaway.

In the meantime, I hope you’ve all caught up with previous episodes. There’s quite a lot of evidence of that from the statistics we see. (To repeat something I’ve said before, we’re not even close to obsessed with the statistics but enjoy the fact we continue to have a good audience.)

Episode 17 “Two Good, Four Better?”

Here are the show notes for Episode 17 “Two Good, Four Better?”. The show is called this because our Performance topic is about aspects of whether more LPARs in the same CECs in a sysplex is better or worse.

Here are the show notes.

Where we’ve been

It’s taken a long time to get back together to do another episode!

• Marna has been to the zTechU Washington DC, and Milan, Italy for a z Sysposium for Italian customers, and at GSE UK at Whittlebury Hall.

• Martin has been travelling a lot too. As well as being in Whittlebury Hall, Martin has been to Hamburg, Germany and Istanbul, Turkey to visit customers, IBM Silicon Valley Lab and Poughkeepsie Lab, and in New York City for a customer visit.

• While in Poughkeepsie he dropped in on the Terminal Talk folks, in the “T4” (“Terminal Truck Talk Thing”).

Mainframe

Our “Mainframe” topic discusses the addition of a new parmlib member for RACF, IRRPRMxx.

• It replaces ICHRDSNT (the RACF Data Set Name Table), and ICHRRNG (the RACF Range Table). These previously could be provided as usermods, maintained as assembler source. There are two levels of pain here:

  1. Having to know how to code and assemble them.

  2. Having to manage them as a usermod each release.

• If you use both, the load modules for ICHRDSNT and ICHRRNG and the IRRPRMxx parmlib member, the parmlib member takes precedence.

• IRRPRMxx is found at IPL within IEASYSxx RACF= statements. You can have up to three members. ICHRDSNT replaced by DATASETNAMETABLE statement, and ICHRRNG replaced by RANGETABLE statement in the parmlib member. Complete syntax is found here.

• You don’t need to code the new IRRPRMxx from scratch, you can use a nicely provided REXX exec to convert from memory on what you are currently using, or from a load module on DASD with DSNT2PRM. Be careful, though, as any RVARY commands might change settings won’t be reflected. This tool is informal and off the web, and it can be run pre-V2.3 just to see what it produces.

• You can’t use the IRRPRMxx until z/OS V2.3. But when you do, use the handy supplied TSO command RACPRMCK to verify the syntax of the parmlib member. Put it in the parmlib concatention, and try it out.

• Then, when you’re ready to use your real IRRPRMxx members, just update your IEASYSxx and IPL.

Performance

Martin talked about whether 2 LPARs (on 2 CECs) or 4 LPARs (on 2 CECs) is better.

• This is related to some trends, but it’s a big subject.

• There are tradeoffs between 2 vs. 4. Better availability with 4, but would it perform better or worse? It might be better if you get to a state where each LPAR is contained in a single drawer (on z13 or z14). (At least two customers he knows does this.) There is more PR/SM overhead with more LPARs, which was discussed in a 1990 Washington Systems Center Orange book (“PR/SM Performance in LPAR Mode”, ZZ05-0453).

• Also the topic of memory duplication, with more DB2 images. CICS and MQ would similarly have memory considerations too. There might be some scope for consolidation of DB2s.

• Operational and software considerations need to be thought through. How much harder would it be to manage it, and also keep the software up to date across more images? Of course, much of the thinking would already have been done for 2-way.

• Martin is planning on doing a presentation on this with Anna Shugol. This LPAR configuration design does remain in interesting topic.

Topics

Our podcast “Topics” topic is a follow up and a short discussion on what to do if your wifi isn’t performing as you wish.

• It’s been a year since Marna’s son built his “gaming” personal computer. She asks him about what has gone well with it, and what he might save in the future for.

• One thing that has been a problem is the wifi signal in the house. It is not consistently strong, and he would like an ethernet connection directly into the computer. That isn’t happening (with a cord down the hallway), so a different solution was found: Powerline Ethernet solution, which plugs into two electric outlets.

• This was less than $100 and is working well. Other solutions do exist, but this cost effective one has been this gamer’s delight.

Customer Requirements

• RFE 76283 KC4Z sysplex support

  Need KC4z to support sysplex environments - data should be stored at the sysplex level, i.e. /sharedapps and any system specific/runtime info stored in system level directories.

Our discussion:

This is in z/OS V2.3, and after performing a migration action, you’ll have it. It was nice to have a requirement that was delivered already (although it wasn’t marked as so at the time of the recording)!

• RFE 109248 IEBUPDTE Support for VB and LRECL > 80

  This utility is limited to 80 bytes, but PDS/PDSE are not limited to 80 bytes, so we can not use it to add members to a PDS/PDSE that exceed the 80 byte limit.

Our discussion:

Indeed, this sounds to be like a good utility extension. Limited to FB and no larger than 80 bytes does mean that it can’t be used for some solutions today. Although Martin points out that other capabilities exist (for instance in DFSORT) to do this without the FB 80 restriction. Marna pointed out this very old utility was wonderful for doing quick additions in parmlib (which is what ServerPac does with customized data). We’ll have to see the response on this one, and how it ends up.

Where We’ll Be

Martin will hopefully be in Copenhagen, Denmark visiting a customer.

Marna will be at SHARE in Sacramento March 12-16 , and in Cairo IBM TechU April 15-17.

On The Blog

Martin published three blog posts:

• REST, JSON, And DB2 – Some Initial Thoughts

• What Is This DB2 Subsystem For?

• Maskerade

Contacting Us

You can reach Marna on Twitter as mwalle and by email.

You can reach Martin on Twitter as martinpacker and by email.

Or you can leave a comment below.

(Originally posted 2018-02-24.)

I’ve used a title like this before.

It was in 2011 in What’s In A Name?¹. But this post bears only a slightly resemblance to that one.

This post is about gleaning information from names, specifically data set names.

Consider this one…

PR04.XYZ892.XYZ893.D171130.T1700

So what are we to make of this data set name?

It’s a real data set name, by the way, with only one obvious substitution.

What if I tell you the step name is XYZ892?

That gets us to

PR04.~step name~.XYZ893.D171130.T1700

where I’m using the convention ~...~ to denote something that’s been decoded.

What if I tell you this data set appears in a job that is part of a stream that kicked off on 30 November, 2017?

Then, and you probably guessed this, D171130 probably refers to a specific iteration of the job, namely the one that started on 30 November, 2017.

That gets us to

PR04.~step name~.XYZ893.~run date~.T1700

and the data set name makes a lot of sense.

Now, it turns out that the XYZ892 step name is also the main DB2 package name, with a launcher program as the actual job step name. This is encoded in the job name PXYZ8924.

From SMF 30 I know that this job is one of a set of clones, being from Stream 4.

So you might fairly conclude the 4 is significant.

So we could now decode the job as

PR~stream number~.~step name~.XYZ893.~run date~.T1700

Now, using the Life Of A Data Set (LOADS) technique we can glean a little more. ² I last mentioned LOADS in A Record Of Sorts.

You might’ve guessed that there would be a job called PXYZ8934 – and indeed there is. LOADS tells us that our job (PXYZ8924) creates and writes the data set (SMF 15 record) and PXYZ8934 reads it (SMF 14 record). Unsurprisingly it reads it in a step called XYZ893.

So we reasonably decode the data set further:

PR~stream number~.~step name~.~other job step name~.~run date~.T1700

This pattern is quite common in this batch application – where the data set name contains the writing program’s name and the reading program’s name. And the job names reflect their main program.

I think this takes us a long way further than just seeing the data set name raw.

So what does our code do?

What It Used To Do

It used to just recognise a job name in a data set name. This was most likely to occur in a temporary data set name.

For example

SYS17335.T061340.RA000.PXYZ8924.R0492069

from this same job (PXYZ8924).

This did appear in a fair number of permanent data sets, too. For some customers.

Step Name

Now, this is new. If the code sees a step name in one of its data sets it bolds it.

What it can’t see is a different step from a different job.

Creation Date

If the creation date is like ‘D171130’ and it’s a data set qualifier it’s bolded.

If might be possible to detect other forms of date. For example, DG113017 is a qualifier. But I don’t think it’s reasonable to hunt for arbitrary date formats with arbitrary prefixes. Of course, it’s just a SMOP³. 🙂

In the example the low level qualifier (T1700) might be alluding to a time – 5PM? I haven’t coded for this.

Conclusion

As this example shows, there’s value in decoding data set names. Having code do it, spotting patterns for you, is the smarter way. And, in our case, bolding the detected pieces brings the reporting to life.

There are most certainly further insights that can be gained, such as when the SMFID is part of the name, or the previously-mentioned program name. Perhaps by the time you read this I’ll’ve added them.

This machine is learning. 🙂 But, as with ML, nobody’s quite sure what⁴ . 🙂

(Originally posted 2018-01-30.)

I was fortunate enough to spend last week in IBM’s Silicon Valley Lab (SVL), both in formal sessions and talking to developers.

One topic – which isn’t really all that new – is DB2’s REST support, which is pretty comprehensive and good.

The basic idea with this support is you can call DB2 via a URI¹, with a JSON payload, and get JSON back. It’s possible that only one acronym means anything to you in that past sentence. The one that really might need explaining is “JSON”.

JSON stands for “JavaScript Object Notation”. (As with so many acronyms it’s malformed and should really be “JON”. Just to make all those Jons out there feel special.) 🙂

REST stands for “Representational State Transfer”, another malformed acronym. So it goes. 🙂

This is where having played with JavaScript comes in handy. If you were assigning an object in JavaScript you might do it with syntax like

var a = { "x" : 1,
          "y" : 2 };

There’s nothing mysterious about this: Parameter x has value 1 and y has value 2.

The JSON representation is just

{ "x" : 1,
  "y" : 2 };

Anyway this isn’t really the point of this post, but it gives you some idea of what you can pass into DB2 via REST, using standard HTTP methods such as GET, POST and PUT. (Making “HTTP” a live link to the Wikipedia article wasn’t really gratuitous as it contains descriptions of these methods.)

I’m more interested in how it shows up in DB2, and particularly in the instrumentation.

So what follows are a couple of topics I discussed while in SVL. But first something I should mention. Otherwise the rest² won’t make much sense.

REST Support Sits Atop DDF

It’s best to think of REST as a special kind of DDF work, with all the usual attributes. So it runs under an enclave in the DIST address space. And now we’re on familiar ground:

• We have the WLM controls we expect.
• We have the instrumentation – SMF 30 and SMF 101 – we need.

Two Topics

So let’s dive into the two topics.

Service Names

When a service is invoked it is invoked by an externalised name. It’s a two-part name: Collection Name and Service Name. Unsurprising, but I want to confirm it to you, Collection Name maps onto Package Collection Name and Service Name maps onto Package Name.

These appear in the SMF 101 IFCID 239 Package Accounting record. So you can spot the use of specific services in Accounting Trace. And you can tell where the requests came from. Of course, you might want to identify “high CPU” services, and all that jazz.

Some time ago DB2 added support for 128-character Package Names which, of course, broke my code. At the time it might not have seemed all that significant but can you imagine being constrained to having 18-character REST Service Names? Of course the format of the IFCID 239 record was enhanced to handle this³. So you can see the long names.

CPU

JSON can get quite complex, with the example I gave above being about the simplest one you can get. More realistic JSON is a non-trivial tree structure. So parsing it might be non-trivial, or so I feared.

Unlike System XML, there is no “System JSON”. Furthermore JSON parsing isn’t necessarily zIIP-eligible, precisely because of that.

So, if parsing JSON could be complex, isn’t that going to be expensive?

So I spent a nice hour and a half talking “of cabbages and kings⁴ and REST and DDF” with my favourite DDF developer.

The important point is this: The JSON parsing happens entirely on the Enclave.⁵

Two things flow from this:

• Keeping DIST up with DBM1 and MSTR is not a threat as the Enclave work should be classified lower.
• Because the work is on the Enclave I’d expect it to have the same eligibility for zIIP as other DDF work.

There is one element of REST work that is at the DIST address space’s dispatching priority: reading in the JSON. Imagine a video 64-Bit Encoded. In principle that’s perfectly valid in JSON. It could be gigabytes of payload and quite expensive to read in.

We discussed measuring this – to see if it’s an issue. The best way to do it is to subtract the Enclave CPU Time from the Total CPU Time – in the SMF 30 Interval record for the relevant DIST address spaces. Without REST I see this number as small – in all the customers I’ve ever looked at. But if it’s large it’s worth thinking about.

These Are Early Days

I’ve not seen any real data from a “REST into DB2” situation yet – and I’d love to. But I’m pretty sure some of the above will help me (and should help you) understand it and its effects. Certainly I have code that will do a decent job of analysing both SMF 30 and SMF 101 IFCID 239.

I’ve seen quite a lot of material on this topic, but it really helps if you already understand things like REST, JSON and JavaScript. (At first I wrote “terms” instead of “things” but I think it needs to go deeper than just being familiar with the terms.) I got my understanding of these by messing with lots of web-related technologies – for fun and profit – quite a while ago.

It’s early days for this technology – but already I like how DB2 supports REST. More when I get it.

And sorry folks for the, ahem, “brevity” but I was trapped on yet another long flight with only you for company… 🙂

(Originally posted 2018-01-26.)

“What is this DB2 subsystem for?” is an important question – at least to me. And explaining this topic over lunch to a friend of mine who was a DB2 developer, convinced me it has more general importance.

If you’re a DB2 sysprog or DB2 the answer is probably well known to you ¹.

If you’re not a DB2 person it’s probably not so obvious.

Something worth noting at this stage is that very little of what’s in this post is specific to DB2. Much of it applies to IMS and MQ, for example.

Why Do We Care What A DB2 Is For?

In general, the more you know about what is going on on your systems the better.

Certainly, from my perspective as an outsider, I think I’m more effective with a customer if I know what their infrastructural components are for. It’s a kind of architectural thinking – so it might appeal to architects and infrastructure managers in customers as well.

To take a (generalised) example, if I see a DB2 Datasharing group I want to know what each member (DB2 subsystem) is for. Elements of the answer might include statements like:

• This DB2 is for Batch.
• These DB2s are clones of each other, for resilience.

We’ll return to these presently.

One other point: Subsystems arise over time for different reasons. Two examples:

• To provide virtual storage constraint relief.
• Different applications came into being, each with their own DB2. They might even have come from different companies merging. I’ve heard customers refer to these as “heritages”.

Add to that how Datasharing groups come to exist and the picture gets really interesting.

And the reason for all this is to be able to provide governance and manage DB2 subsystems better.

Detecting The Different Roles

This post really emphasises the use of SMF 30 (Address Space) and RMF data. The reason for using this data (and not DB2-specific data) is quite simple: The approach has to scale. Many customers have more than 50 DB2 subsystems, some more than 100. If you do, you’re not going to be keen on processing Accounting Trace for all of them.²

Of course, having formed a high level view this way, you might well want to dive into Accounting Trace – but only for some subsystems and time ranges.

Detecting DB2 Subsystems

This is the easy bit: From SMF 30, if the job name is fredDBM1 and the program name is DSNYASCP then we have subsystem fred. We can see its DB2 version in the Usage Data Section.

Obviously you see its SMFID, Workload, Service Class, Report Class and Reader Start Time.

Detecting Datasharing Groups

You can, most of the time, obtain a good grouping of subsystems into Datasharing groups. The following technique works if each DB2 subsystem has its own IRLM and if the IRLM address space’s job name matches the subsystem. For example the IRLM address space corresponding to DB2ADBM1 (subsystem DB2A) is DB2AIRLM.

A DB2 Datasharing group has two XCF groups, supporting the group’s LOCK1 Coupling Facility structure. One’s name begins with IXCLO. It’s the other one that interests me: It is a proxy for the group name. For example, it might be called DXRPROD when the actual group’s name is DSNPROD or DB2PROD. But it’s close enough. (CF structure names could be used to patch the name, perhaps.)

In any case the IRLM address spaces – as proxies for subsystems – can be used this way to detect Datasharing groups.

I discussed this in detail in DB2 Data Sharing and XCF Job Name back in 2009.

Working Out What Role A Subsystem Plays

This is the bit that really excites me: My bold claim is I can tell you something about the work coming into a DB2 subsystem without using DB2 instrumentation³. And the use of SMF 30 only is important because most people connect SMF 30 across their entire z/OS estate.

The key to this is the Usage Data Section in SMF 30, which I’ve talked about many times.

As an example, take a CICS region. (You detect these by the program name being DFHSIP). If the CICS region connects to a DB2 subsystem you get at least one Usage Data Section that documents the DB2 connection. In particular it tells you the DB2 subsystem it connects to. I’ve discussed this in a 2012 post: Another Usage Of Usage Information.

By the way a CICS region can directly attach to only one DB2 subsystem (likewise only one MQ queue manager).

So you can identify all the CICS regions connecting to a given DB2. You could, in addition to naming them, add up their CPU.

Similarly, for IMS work you’d see a Usage Data Section mentioning IMS.⁴

And almost everything else you could lamely call “Other” or be bold and call it “Batch”. My code counts the distinct job names and sums up the CPU.

DDF is an interesting case: You would find the DIST address space for the subsystem⁵ and examine the Enclave-related fields. I wrote about this in DB2 DDF Transaction Rates Without Tears But essentially you can pick up the DDF Transaction Rate and DDF CPU.

So, let’s take a real example. Here is a DB2 Datasharing group with 6 members, running on 2 LPARs.

The numbers are from my prototype code. All numbers are multi-hour (shift) averages.

Looking at SYSA first:

• DB1A has a large amount of CICS work (over 7 engines’ worth) from 14 regions. It also has a very small amount of DDF work. “Other” here is mainly Batch and it is about an engine’s worth.
• DB2A also has lots of CICS work – over 7 engines’ worth from 16 regions. DDF work is much more significant than on DB1A – at about half an engine. “Other” is again similar.
• DB3A has very little work to it – just a small amount of DDF.

Turning to SYSB:

• DB1B has 11 CICS regions, using 5 engines’ worth of CPU and a small amount of DDF. Again “Other” is just over an engine’s worth.
• DB2B is about as lightly used as DB3A – with a smattering of DDF and a littl bit of “Other”.
• DB3B has 17 CICS regions, using 6.4 engines’ worth of CPU. DDF here is a significant workload, similar to that on DB2A. Here is uses about 1/3 of an engine. “Other” is about 1 engine’s worth of CPU.

So, these are “broad brush” characterisations but I think they are useful. For example, the 2 DDF-oriented subsystems (DB3A and DB2B) might be easy to fold into other subsystems. I say this because they are light on transaction rates and CPU so should pose no threat to the scalability of other subsystems in the Datasharing group.

Conclusion

Much of this instrumentation you might’ve seen before. The trick here is to invert the question “what does this address space connect to?” to yield “what connects to my DB2?” And that way you get a view of the role of the DB2 subsystem.

Of course, this is not nearly as deep an analysis as you could do with DB2 Accounting Trace – such as what transactions get at a DB2. But it is much lighter and more scalable. It’s great for a first pass – which might be all I get to do.

One other thought: This feels like the beginning of a journey to characterise DB2 subsystems. One could, for example, use WLM Service Classes⁶ for classifying each DB2’s role. This wouldn’t work for DDF but for other work you could sum up the 30’s by Service Class instead of e.g. “CICS”.

(Originally posted 2018-01-19.)

So now I get to sit and relax for umpteen hours, while a 787 speeds me to California. So I can write something. And, yes, it’s been a while.

To tell you the truth, I’ve been working on back-to-back engagements and enjoying vacation for what seems like forever. And after a stint of that I generally I end up with a swirl of “if only I had time to write about that” topics in my brain. So here I am ”Going To California On A Big Jet Plane”[1] with time on my hands.

So I have this customer – and it’s not the first like it – who has a Parallel Sysplex with a pair of Coupling Facility LPARs and a couple of Production z/OS LPARs. Not the first, either, to have offline Coupling Facility links. But I didn’t know they had offline links until they told me. Shame on me! 🙂

If a customer tells me things I guess I should know I go on a hunt for how I could have figured it out from the data. And so it was in this case. And, having just been successful at writing code to detect offline links, I’m writing this to show you what’s possible and how.

Three Sources Of Link Data

SMF 74 Subtype 4 is the main Coupling Facility Activity record, and it has three sets of information about links:

• Local Facility Data Section
• Remote Facility Data Section
• Path Data Section

You get at least one 74–4 for each coupling facility from each z/OS connected to it. Ideally you’d enable 74–4 on all systems in the sysplex.[2]

I say “at least” as modern CFs tend to spill to a second and, occasionally, a third. So here’s another thing I’ve absorbed over the last few customer engagements: Customers nowadays have hundreds of structures in each coupling facility.[3]

Local Facility Data Section

You get one of these in the 74–4 record – whether the base or the overflow ones. It has to be there as it has the Coupling Facility name, as well as the machine it’s on.

It also has some relevant things which I’ll touch on now, and come back to later:

• Path masks
• Channel Path Acronyms
• Channel Path IDs (CHPIDs)
• Index into Path Data Sections and count of the number of sections

As examples of Channel Path Acronyms, I’ll take three:

• CIB – Infiniband
• CS5 – ICA-SR
• CL5 – Coupling Express Long Reach

Remote Facility Data Section

You get one or more of these, depending on how many CFs the one in the Local Facility Data Section has duplexing links to. The path-related information in this section is:

• Count of the number of paths
• Channel Path Acronyms
• CHPIDs
• Index into Path Data Sections and count of the number of sections

Pretty similar to Local Facility Data Section information. But not quite. The information is only for online paths.

Path Data Section

I’ve written about this section many times, most recently in And Latency Once More.

You get one section for each link under certain circumstances: The link has to be physical (no IC links) and modern enough (not ICB, for example). Unfortunately, it has to be online. So this section doesn’t help us in our quest for offline links.

And the reason I keep writing about this section is because it is full of value and surprises. But not this time. 😦

Actually There’s A Fourth

I include this for completeness[4]. SMF 73 gives information on CHPIDs – from a z/OS system.

For CF links this information is very basic. Mainly the CHPID, whether it’s shared, whether it’s online, and Channel Path Acronym. [5] For other types of channel, such as FICON, we get a lot more, useful, information.

There is a potential use for this data, quite apart from corroboration: If a CF-related channel path appears in SMF 73 but not in 74–4 it implies it’s not connected. Of course it would be strange if it managed to be online despite that. 🙂 I should probably code to this possibility.

The Truth Is Masked

In the Local Facility Data Section piece I mentioned path masks. It might not have seemed obvious but they are key to this. There are three 8–bit mask fields:

• Path Installed Mask
• Path Available Mask
• Patch Composite Mask

Each bit represents a different path and each path is described by the same bit position in all the masks.

These terms might be familiar in the context of disk controllers.

Consider the following worked example:

This is from a real customer. Reading from top to bottom.

1. The channel path acronyms speak of 4 CS5 (ICA-SR) paths and 4 CIB (Infiniband) paths. Note the order.
2. The Path Installed Mask (Hex FF) has all the bits turned on – so all 8 paths are installed.[6]
3. The CHPIDs are listed. The order here is again important.
4. The Path Available Mask (Hex B1) has some bits on and some off – 10110001 being the bit string. So only some of these paths are available, the rest being installed (in this case) but offline.
5. The Path Composite Mask (also Hex B1) reflects both the Installed and Available Masks.

Taken together, we conclude:

• All 4 Infiniband paths are installed but offline – CHPIDs 0E, 6A, 6F, and 78.
• All 4 ICA-SR paths are installed and online – CHPIDs 04, 24, 25, and C6.

Conclusion

For links between z/OS LPARs and CF LPARs we can see both the online and offline paths. For the CF to CF links we can only see the online ones.

There is an old joke. It goes like this, updated a little: “I wrote a long blog post as I didn’t have time to write a short one”. Well, I did have time to write a short one… 🙂

By the way “Maskerade” is the name of a Terry Pratchett Discworld book, in case you wondered if there was a cultural reference. High time I read it again.