Normal Service Resumed

I’m sat in a nice coffee shop in Wallingford, relaxing after an interesting few weeks.

Over the past few weeks I’ve migrated my blog over to WordPress. If you’re reading this you’ve followed me over so a big thank you.

I migrated 526 blog posts. This number frankly astonishes me, though it’s been almost 15 years so that’s only 30 something a year.

The topics have varied, which is why I stuck the commas in the title some time ago. In the migration I made it official.

In the meantime a few years ago my dear friend Marna Walle joined me in using the “Mainframe, Performance, Topics” name – for our podcast series.

The focus of the blog has not changed at all. It’s just the “social conditions” that have changed: I’m funding the blog – on WordPress. This gives me very little more latitude but does mean I can take it with me – into (eventual) retirement.

The process of writing still needs sorting out – but my experience of migration tells me that WordPress is going to be a good place to be. I certainly can confect HTML and upload pictures and even PDFs. But the WordPress offers a wider ecosystem. For example, I’m writing this across Mac, iPad and iPhone using the very excellent Drafts app and I know Drafts can publish direct to WordPress. (Drafts’ automation might be handy in helping me write, actually.)

So expect even more

  • Mainframe whatever I want to talk about
  • Performance whatever I want to talk about
  • Topics whatever I want to talk about

You get the picture: It’s whatever I want to talk about. 🙂

So normal service resumed, then. 🙂

More On Native Stored Procedures

(Originally posted 2019-08-25.)

This follows up on Going Native With Stored Procedures?, and it contains a nice illustrative graph.

I could excuse a follow up so soon with the words "imagine how unreadably long the post would be I had written this as all as one piece”.

However, the grubby truth is I got to write some code I didn’t expect to just yet. And this post became possible as a result.

Where Were We?

The gist of the previous post is:

  • Non-native stored procedures aren’t zIIP eligible, regardless of their caller’s eligibility.
  • You can see the level of zIIP eligibility for DDF and note if it looks less than you might expect or desire.

And that’s where we (and my code) left it. A useful place to be but not the best that could be done.

The New Code

So, for the very same study as in Going Native With Stored Procedures?, I began to dig, and to teach my code new tricks.

I wanted to know more about the stored procedures that weren’t zIIP-eligible.

To do that would take processing of package-level Db2 Accounting Trace (SMF 101).

Package-Level SMF Records

There are two types of SMF 101:

  • IFCID 3 – Plan-Level Accounting
  • IFCID 239 – Package-Level Accounting

Each IFCID 239 record contains up to 10 QPAC Package Sections[1], each describing one package name. I emphasise “name” because you don’t get one each time you call a stored procedure with that name. Instead you get one section for all calls that invoke that package.

My Old Package-Level Code

To be honest, I stopped earlier with developing my package-level code than I wanted to. The code takes an IFCID 239 record and “flattens” it, with each Package Section placed in a fixed position in the output record.

So, I have records still with up to 10 package names in them.

If you’d sent me Package-Level Accounting for a study involving DDF my code flattened the records and left it at that. There was no reporting.

My New Package-Level Code

Now I have code that takes a 10-package record and turns it into 10 1-package records[2]. I also have some reporting.

The interesting bit is the reporting.

I create a CSV file, sorted by subsystem and, within that, GCP CPU. Each line is a separate Correlation ID and Package combination. There is a nice field in the QPAC section (QPACAAFG) which says which type of package it is. Here are the values QPACAAFG can take.

CodePackage Type
X’0000’Regular package
C’01’Non-Native Stored Procedure
C’02’User-Defined Function (UDF)
C’03’Trigger
C’04’Natived Stored Procedure
C’05’Inline UDF

Notice anything strange about the above table?

I think the X’0000’ is odd but it’s probably indicating “this field not filled”.

In any case you can readily tell what kind of package we have.

When I import the CSV file into Excel (and hold my nose) 🙂 I get a useful spreadsheet[3]. I define a field which is the concatenation of the package type and its name. That enables me to produce a nice graph. Here’s an example.

It shows CPU seconds – both GCP and zIIP – for the 20 packages with the biggest GCP CPU.

I’ve obfuscated all but one package name. We’ll get to that. But here are some observations:

  1. This package is a non-native stored procedure. There is very little zIIP eligibility. (Why there is any zIIP eligibility is unclear.)
  2. This package is a native stored procedure. It has about 60% zIIP eligibility.
  3. This package is well known – SYSLH200 – so I’ve not obfuscated it. It also shows about 60% zIIP.
  4. This is a second native stored procedure – again showing good zIIP eligibility.

Most of the rest of the packages are non-native stored procedures, with a couple of native ones. The zIIP usage is much as you would expect.

Let’s return to SYSLH200. It’s a Db2-supplied default package. You see it a lot. In a different subsystem it’s – by miles – the biggest package. This would be entirely normal for a subsystem that wasn’t a heavy user of explicit DDF packages, such as stored procedures.

Here’s a table of such package names:

Package NameDescription
SYSSHxyydynamic placeholders – small package WITH HOLD
SYSSNxyydynamic placeholders – small Package NOT WITH HOLD
SYSLHxyydynamic placeholders – large package WITH HOLD
SYSLNxyydynamic placeholders – large package NOT WITH HOLD

Now I can spot them I’ve added this table to my bank of standard slides to use in customer engagements. I’ll add other well-known package names over time.

For this subsystem it’s clear the customer has some familiarity with Native Stored Procedures, whether accidentally or intentionally. (It’s possible the native ones were supplied by a vendor and the install script set up the Db2 Catalog to enable them.) But this customer has some way to go – assuming the remaining major stored procedures can and should be converted to native.

Conclusion

This code is going into Production – Just as soon as I can find the time to convert the JCL to an ISPF File Tailoring skeleton.

But I’m already using it so it is in a sense in Production.

So if I ask you for package-level Db2 Accounting Trace this will be one of the good uses I’ll put it to.

The Journey Continues

I’m sure there’ll be “another thrilling instalment” 🙂 in the DDF saga. I just have no idea right now what what it would be.

Stay tuned. 🙂


  1. Mapped by mapping macro DSNDQPAC in the SDSNMACS library shipped with Db2.  ↩

  2. I’ll admit it’s not the most efficient code, involving two passes over the data; One day I’ll probably rework my Assembler DFSORT E15 edit to emit 10 records and eliminate passes over the data.  ↩

  3. Well, useful when I’ve taken each subsystem and plonked it in its own sheet manually 😦 . If anyone knows how to automate this sort of thing I’d love to know.  ↩

Making The Connection?

(Originally posted 2019-08-11.)

I wrote the last blog post on my way to my first SHARE conference. I’m writing this one on the way back.

So a big thank you to all who made me feel so welcome in Pittsburgh. In many cases it was good to meet up with long standing friends; In some cases people I’ve known a long time but never actually met. I hope I’ll be back1.

Right now the team is working on a customer study, one component of which looking at big batch CPU consumers. And I think I‘ve just solved a long-standing issue of interpretation. So I’m sharing the thought with you.

It concerns batch job steps with a TSO step program name – IKJEFT01, IKJEFT1B, or IKJEFT1A. There are two main cases here:

  1. Those that call Db2 – through the TSO connection.
  2. Those that don’t.

The latter are usually something like REXX, or they could be compiled / assembled programs called from the Terminal Monitor Program (TMP) in Batch.

It would be good to know which is which. So, there’s no use tuning the SQL of a job step that doesn’t go to Db2. You’re better off tuning the REXX or program code. As it happens, I am perfectly capable of tuning REXX programs; I’m not so hot at tuning SQL. In any case I’m looking to advise the customer as to which is indicated for any given heavy CPU step.

A Previous Method

Previously we would’ve used our code which matches SMF 101 (DB2 Accounting Trace) with SMF 30 Step-End records (Subtype 4) to detect whether a job step accesses a Db2 subsystem or not. This is the most comprehensive way but it’s quite likely Accounting Trace isn’t turned on for the subsystem the job step attaches to. Further, we don’t even know which subsystem it is.

So this is OK – and in practice we’ve used this method for a long time.

By the way, for TSO Attach Batch the Db2 Correlation ID is the job name2.

A More Scalable Method

There is a more scalable way of distinguishing between TSO job steps that access Db2 and those that don’t. And it’s more scalable because it doesn’t require Db2 Accounting Trace. Further, it yields the subsystem name.

It uses only SMF 30 records.

I’ve mentioned the SMF 30 Usage Data Section before: If an address space accesses Db2 the record will have at least one Usage Data Section pointing to Db2. The Product Name field will say “DB2” and the Product Qualifier field will contain the name of the subsystem. (The Product Version field will tell you which version of Db2 it is, which is also handy.)

Today I tested the following query:

‘Show me all the SMF 30–4 records with a Program Name field beginning “IKJEFT” and with a Usage Data Section that points to Db2."

This query only uses SMF 30. It ran quickly and produced an interesting list. It would be a small matter to turn this into the following query:

‘Show me all the SMF 30–4 records with a Program Name field beginning “IKJEFT” without a Usage Data Section that points to Db2."

The first query yields the first case above: TMP job steps that attach to Db2. The second query yields the other case: Non-Db2 TMP steps.

So now we can more usefully discuss any step whose Program Name begins “IKJEFT” – by working out which case it is.

I consider this a nice result – which is why I’m sharing it with you.

And, yes, if the step goes to Db2 we can dive into Accounting Trace to figure out:

  • If there is after all any non-Db2 CPU to tune out.
  • Any other buckets of time worth tuning.

  1. Both to Pittsburgh and SHARE, in case you wondered.

  2. The observant will note that Correlation ID is up to 12 characters and job name is at most 8. The Correlation ID pads to the right with blanks up to 12.

Going Native With Stored Procedures?

(Originally posted 2019-08-04.)

I hope the term “Going Native” isn’t considered offensive. If it is my defence is that I didn't coin the term “Native Stored Procedures” and that’s what I’m alluding to.

This post is about realising the benefit of Native Stored Procedures for DDF callers. Most particularly, assessing the potential benefit of converting Stored Procedures to Native.

It’s not been long since l wrote about DDF. (My last DDF post was in 2018: DDF TCB Revisited. ) It’s been rather longer since I last wrote about Db2 Stored Procedures: 2007’s WLM-Managed DB2 Stored Procedure Address Spaces is the latest I can find.1

You might ask “what on earth is a z/OS Performance guy doing talking about Db2?” In “Low Self Esteem Mode” 🙂 I might agree, but I’ve always thought I could pull off “one foot in each camp” when it comes to Db2 and z/OS. “I consider it a challenge…” 🙂

Seriously, z/OS folks have quite a bit to say that can help Db2 folks, if they try. And so we come to a recent example.

We’ll get to Stored Procedures presently.2 But let me build up to it.

And, yes, this post is inspired by a recent customer case. And, no, it didn’t alert me to the topic; It just was a good excuse to talk about it.

zIIP Eligibility And DDF

In principle, a DDF workload should see roughly 60% zIIP Eligibility. As I said in DDF TCB Revisited

“The line in the graph – or rather the round blobs – shows zIIP eligibility hovering just under 60% – across all the DB2 subsystems across both customers. (One of the things I like to do – in my SMF 101 DDF Analysis Code – is to count the records with no zIIP eligibility.”

That last sentence is interesting, and I don’t think I spelt it out sufficiently: For quite a while now a DDF thread is either entirely zIIP-eligible or not at all.

I glossed over something in that post – which was the right thing to do as that set of data didn’t display an important structural feature: If your DDF thread calls a Non-Native Stored Procedure (SP) or User-Defined Function (UDF) it loses zIIP eligibility for the duration of that call.

Native Stored Procedures

You can write SPs and UDFs in pretty much any language you like, and “shell out” to anywhere you like3. For example, you can write them in REXX, Java, COBOL, PL/I. And SPs and UDFs can call each other. Lovely for reuse.

These capabilities have been around for at least 20 years. More recently (but not that recently (but progressively enhanced over recent Db2 versions)) a different set of capabilities have emerged: Native Stored Procedures. You write these in an extension to SQL called PL/SQL. The term “Native” here refers to using the built-in language, rather than a programming language.

Native SPs run in the Db2 DBM1 address space – but with the caller’s WLM attributes. Non-Native SPs run in WLM Stored Procedures Server address spaces (as described by WLM-Managed DB2 Stored Procedure Address Spaces and the Redbook it references.)

For the rest of this post I shall talk of SPs – for brevity. UDFs and Triggers are generally similar.

Why Do Native Stored Procedures Matter?

As a Performance person, Native Stored Procedures matter to me because there’s a fundamental difference in zIIP eligibility:

  • When a DDF thread calls a Native SP the zIIP eligibility is preserved.
  • When a DDF thread calls a Non-Native SP the CPU to run the SP is not zIIP eligible.

So, there’s an economic advantage to SPs being Native. Actually it’s twofold:

  • The obvious one of zIIP CPU having lower software (etc) costs than GCP CPU.
  • When work switches between zIIP eligibility and non-eligibility the work doesn’t get undispatched on one processor and redispatched on another immediately. However, if it does – for any reason – get redispatched its eligibility is checked. That could well cause redispatch on a different type of processor. Of course that’s a different processor – with the potential for cacheing loss.

In any case, we’d want to maximise zIIP eligibility, wouldn’t we?

A Case In Point

I said early on in this point I have a real customer case to discuss.

I did my standard “let’s see how much GCP and how much zIIP this Db2 subsystem uses” analysis. The results were emphatically not 60% zIIP eligible, 40% ineligible.

I examined this at three levels:

  • DDF Service Class, using SMF 72-3 Workload Activity data.
  • DIST Address Space, using SMF 30 Interval Independent Enclave data.
  • Db2 DDF Application, using SMF 101 Db2 Accounting Trace data.

They all agreed on transaction rates, GCP CPU, and zIIP CPU.

The customer has 9 subsystems, across 3 LPARs. These are not cloned subsystems, though the same application “style” – JDBC4 – dominated the DDF traffic in all cases.

Of these 9 subsystems only 1 approached a 60%/40% split. The rest varied up 90% or so GCP.

Db2 Accounting Trace is helpful here – if you examine Class 2 CPU times:

  • You get overall TCB time on a GCP [1]
  • You get overall zIIP CPU time [2]
  • You get Stored Procedures CPU time [3]
  • You get UDF CPU time [4]

So I took away [3] and [4] from [1]. I compared the result to [2]. With this adjusted measure of GCP CPU Time I got extremely close to 60% being zIIP eligible – for all 9 subsystems.

As an aside, Stored Procedures Schedule Wait and UDF Schedule Wait time buckets can tell you if there are delays in starting the relevant server address spaces. In this data it was notable that the Test Db2 subsystems were more prone to this than Production. As the machines ran pretty busy it’s good to see the delays being directed to less important work5.

So, Shall We Go Native?

“Not so fast” applies – both as an exclamation and as an expectation:

  • Because these are generally comprehensive application changes it’s going to take a while to convert.
  • It might not even be worth it – at least not for 100% of Non-Native SPs.

Here’s how I think you should proceed:

  1. Figure out how much of a problem this is (and you’ve seen that above).
  2. Use Db2 Accounting Trace at 2 levels to establish which moving parts might be worth reworking:
    • IFCID 3 would show you which applications, including client computer.
    • IFCID 239 would show you which package burns the most CPU, and whether it’s a SP, UDF, or Trigger.
  3. Assess – for each of the “high potential” SPs – which applications call them. Of course, only some of them will benefit – as SPs can be called from a wide variety of non-DDF applications as well as DDF ones.

If you carry out the above steps you should see whether it’s worth it to convert to Native.

One final word of caution: Many Non-Native SPs can’t be converted to PL/SQL so any discussion with your Db2 and Applications colleagues need to be sensitive to that.


  1. I need a Content Management System (CMS) as you, dear reader, might find a more recent one. 🙂 

  2. To “break the fourth wall” a moment, this post is more or less writing itself (at what I’m told is 10972m) so it might or might not let me get to the point. 🙂 

  3. Including to other services, even ones out on the web. 

  4. In Db2 Accounting Trace you can sum up by e.g. Correlation ID. JDBC shows up as “db2jcc_appli”. 

  5. And Workload Activity Report and WLM bear out it is less important work that is getting hit. 

Engineering – Part Three – Whack A zIIP

(Originally posted 2019-08-02.)

(I’m indebted to Howard Hess for the title. You’ll see why it’s appropriate in a bit.)

Since I wrote Engineering – Part Two – Non-Integer Weights Are A Thing I’ve been on holiday and, refreshed, I’ve set to work on supporting the SMF 99 Subtype 14 record.

Working with this data is part of the original long-term plan for the “Engine-ering” project.

Recall (or, perhaps, note) the idea was to take CPU analysis down to the individual processor level. And this, it was hoped, would provide additional insights.[1]

Everything I’ve talked about so far – in any medium – has been based on one of two record types:

  • SMF 70 Subtype 1 – for engine-level CPU busy numbers, as well as Polarisation and individual weights.
  • SMF 113 Subtype 1 – for cache performance.

SMF 99 Subtype 14 provides information on home locations for an LPAR’s logical processors. It’s important to note that a logical processor can be dispatched on a different physical processor from its home processor, especially probably in the case of a Vertical Low (VL)[2]. I will refer to such things as where a logical processor’s home address is as “processor topology”.

It should be noted that SMF 99–14 is a cheap-to-collect, physically small record. One is cut every 5 minutes for each LPAR it’s enabled on.

Immediate Focus

Over the past two years a number of the “hot” situations my team has been involved in have involved customers reconfiguring their machines or LPARs in some ways. For example:

  • Adding physical capacity
  • Shifting weights between LPARs
  • Varying logical processors on and off
  • Having different LPARs have the dominant load at different times of day

All of these are entirely legitimate things to do but they stand to cause changes in the home chips (or cores) of logical processors.

The first step with 99–14 has been to explore ways of depicting the changing (or, for that matter, unchanging) processor topology.

I’ll admit I’m very much in “the babble phase”[3] with this, experimenting with the data and depictions.

So, here’s the first case where I’ve been able to detect changing topology.

Consider the following graph, which is very much zoomed in from 2 days of data – to just over an hour’s worth.

Each data point is from a separate record. From the timestamps you can see the interval is indeed 5 minutes. This is not the only set of intervals where change happens. But it’s the most active one.

There are 13 logical processors defined for this LPAR. All logical processors are in Drawer 2 (so I’ve elided “Drawer” for simplicity.)

Let me talk you through what I see.[4]

  1. Initially two logical processors are offline. (These are in dark blue.) Cluster 2 Chip 1 has 7 logical processors and Cluster 2 Chip 2 has 2.
  2. A change happens in the fifth interval. One logical processor is brought online. Now Cluster 2 Chip 1 has 6 and Cluster 2 Chip 2 has 4. Bringing one online is not enough to explain why Cluster 2 Chip 2 gained two, so one must’ve moved from Cluster 2 Chip 1.
  3. In interval 7 another logical processor is brought online. The changes this time are more complex:
    • A logical processor is taken away from Cluster 1 Chip 1.
    • A logical processor appears on Cluster 1 Chip 2.
    • Two appear on Cluster 1 Chip 3.
    • One is taken away from Cluster 2 Chip 2.
  4. Towards the end, as each of two logical processors are offlined they are taken away from Cluster 2 Chip 2.

The graph is quite a nice way of summarising the changes that have occurred, but it is insufficient.

It doesn’t tell me which logical processors moved.

What we know – not least from SMF 70–1 – is the LPAR’s processors are defined as:

  • 0 – 8 General Purpose Processors (GCPs)
  • 9 – 12 zIIPs

The Initial State

With 99–14, diagrams such as the following become possible:

This is the “original” chip assignment – for the first 4 intervals.

This is very similar to what the original WLM Topology Report Tool[5] would give you. (I claim no originality.)

I drew this diagram by hand; I can’t believe it would be that difficult for me to automate – so I probably will.[6]

After 1 Set Of Moves

Now let’s see what it looks like in Interval 5 – when a GCP was brought online:

GCP 5 has been brought online in Cluster 2 Chip 2, alongside the 2 (non-VL) zIIPs. But also GCP 0 has moved from Cluster 2 Chip 2 to Cluster 2 Chip 1.

What’s The Damage?[7]

Now, what is the interest here? I see two things worth noting:

  • A processor brought online has empty Level 1 and Level 2 caches.
  • A processor that has actually moved also has empty Level 1 and Level 2 caches.

Within the same node/cluster or drawer probably isn’t too bad. (And within a chip – which we can’t see – even less bad as it’s the same Level 3 cache). Further afield is worse.

Of course the effects are transitory – except in the case of VLs being dispatched all over the place all the time. Hence the desire to keep them parked – with no work running on them.

After The Second Set Of Moves

Finally, let’s look at what happened when the second offline GCP was brought online – in Interval 7:

GCP 8 has been brought online in Cluster 1 Chip 2. But also zIIP 10 has moved from Cluster 2 Chip 2 to Cluster 1 Chip 1. Also zIIPs 11 and 12 have moved from Cluster 1 Chip 1 to Cluster 1 Chip 3.

This information alone (99–14) isn’t enough to tell you if there was any impact from these moves. However, you can see that in neither case was a “simple” varying a GCP online quite so simple. Both led to other logical cores moving. This might be news to you; It certainly is to me – though the possibility was always in the back of my mind.

Note: This isn’t a “war story” but rather using old customer data for testing and research. So there is no “oh dear” here.

Later On

To really understand how a machine is laid out and operating you need to consolidate the view across all the LPARs. This requires collecting SMF 99–14 from them all. This, in fact, is a motivator for collecting data from even the least interesting LPAR. (If its CPU usage is small you might not generally bother.)

But there’s a snag: Unlike SMF 70–1, the machine’s plant and serial number isn’t present in the SMF 99–14 record. So to form a machine-level view I have two choices:

  • Input into the REXX exec a list of machines and their SMFIDs.
  • Have code generate a list of machines and their SMFIDs.

I’ll probably do the former first, then the latter.

What also needs doing is figuring out how to display multiple LPARs in a sensible way. There is already a tool doing that. My point in replicating it would be to add animation – so when logical processors’ home chips change we can see that.

Maybe Never

SMF 99–14 records aren’t cut for non-z/OS LPARs, which is a significant limitation. So I can’t see a complete description of a machine. For that you probably need an LPAR dump which isn’t going to happen on a 5-minute interval.

However, for many customer machines, IFL- and ICF-using LPARs are on separate drawers. It’s a design aim for recent machines but isn’t always possible. For example, a single-drawer machine with IFLs and GCPs and zIIPs will see non-z/OS LPARs sharing the drawer. Most notably, this is what a z14 ZR1 is.

One other ambition I have is to drive down to the physical core level. On z14, for instance, the chip has 10 physical cores, though not all are customer-characterisable. But this won’t be possible unless the 99–14 is extended to include this information. This would be useful for the understanding of Level 1 and Level 2 cache behaviour.

Finally, there is no memory information in 99–14. I would dearly love some, of course.

Conclusion

While 99–14 doesn’t completely describe a machine, it does extend our understanding of its behaviour by relating z/OS logical processors to their home chips. Taken with 70–1 and 113–1, this is a rather nice set of information.

Which prompts lots of unanswerable questions. But isn’t that always the way? 🙂

A question you might have asked yourself is “do I need to know this much about my machine?” Generally the answer is probably “no”. But if you are troubleshooting performance or going deep on LPAR design you might well need to. Which is why people like myself (and the various other performance experts) might well be involved anyway. So – for us – the answer is “yes”.

The other time you might want to see this data “in action” is if you are wondering about the impact of reconfigurations – as the customer whose data I’ve shown ought to be. 99–14 won’t tell you about the impact but it might illuminate the other data (70–1 and 113). And together they enhance the story no end.


  1. Actually it’s a matter of “legitimised curiosity”. 🙂  ↩

  2. I’ll follow the usual convention of “VL” for “Vertical Low”, “VM” for “Vertical Medium ”, and “VH” for “Vertical High”.  ↩

  3. See, perhaps, Exiting The Babble Phase?.  ↩

  4. You might well see something different. And that’s where the fun begins.  ↩

  5. Which I now can’t find. 😦 and couldn’t run even if I could find it.  ↩

  6. Actually, I’ve experimented with creating such diagrams using nested HTML tables. It works fine. The idea would be to write some code (probably Python) to generate such (fiddly) HTML from the data.  ↩

  7. In UK English, at least, “What’s the damage” means “what’s the bill”.  ↩

Buttoned Up

(Originally posted 2019-06-18.)

In Automation On Tap I talked about NFC tags and QR codes as ways of initiating automation.

This post is about a different initiator of automation.

I recently acquired the cheapest (and least functional) Elgato Stream Deck. This is a programmable array of buttons. My version has 6 buttons but you can buy one with 15 and one with 32 buttons.

A Previous Attempt Didn’t Push My Buttons

A while back I bought an external numeric keypad for my MacBook Pro. When I was editing our podcast with Audacity on the Mac I used this keypad to edit. It’s just numeric, with a few subsidiary keys.

What makes this programmable is that the Mac sees external keypad keys as distinct from their main-keyboard counterparts. So the “1” key on the external keypad has a different hardware code from the “1” key on the main keyboard. Keyboard Maestro macros can be triggered when keys are pressed. So I wrote macros to control Audacity, triggered by pressing keys on the external keypad.

But there was a problem: How do you know what each button does? It’s quite a pretty keypad – as you can see:

So I really didn’t want to stick labels on the key tops. I also tried to make my own template but there was an obvious difficulty: It’s hard to design one that tells you what the middle buttons in a cluster do.

There’s another problem: Suppose I set up automation for multiple apps – which Keyboard Maestro lets you do. Then I need a template for each app – as each will probably have its own key pad requirements. These templates are fragile and in any case this is a fiddly solution.

So this was good – up to a point. It certainly made podcast editing easier. But that was all.

And then I moved podcast editing to Ferrite on iOS, discussed here. And this keypad is now in a drawer. 😦

The One That I Want

Cue the Elgato Stream Deck Mini. First I should say there are other programmable keypads but this is the one that people talk about. I’d say with good reason.

As you can see, it’s corded and plugs into a USB A port.[1] Power to it is supplied by this cable.

It’s easy to set up, literally just plugging it in. To make it useful, though, you need their app – which is available for Mac or Windows. It’s a free app so no problem.

With their app you drag icons onto a grid that represents the Stream Deck’s buttons. Here’s what mine generally looks like when powered up and the Mac unlocked.

The key thing[2] to notice is the icons on the key tops. This is already a big advance on paper templates and ruined shiny key pads. I didn’t have to do anything to get these icons to show up[3]. What you can see is:

  • 4 applications – Sublime Text, Drafts, Airmail and Firefox.
  • A folder called “More”. More 🙂 on this in a moment.
  • Omnifocus Today

The applications are straightforward, but the Omnifocus Today one is more complex. Pushing the button simulates a hot key – in this case Ctrl+Alt+Shift+Cmd+T. I’ve set up a Keyboard Maestro script triggered by this – extremely difficult to type – hot key combination. This script brings Omnifocus (my task manager) to the front and – via a couple of key presses – switches to my Today perspective.

So it’s not just launching applications. It can kick off more complex automation.

And someone built a bridge to IFTTT’s Maker Channel – which could open up quite a lot of automation possibilities. But right now I only have a test one – which emails me when I press the button.

One thing I haven’t tried yet is setting up buttons to open URLs in a browser. I don’t think I’ve got the real estate for that, with only six buttons.

Unreal Estate

In reality I probably should’ve plumped for the 15 button one[4], rather than the 6, but this has been an interesting exercise in dealing with limited button “real estate”.

So, if I press the “More” button I get this:

Because the button tops are “soft” they can change dynamically – as this demonstrates. This is much better than sticking labels on physical keys.

Two features of note:

  • The back button – top left
  • The “Still More” button bottom right

Being able to create the latter means I can nest folders within folders. This greatly increases my ability to define buttons.

The Right Profile

There’s one final trick that’s worth sharing.

Everything I’ve shown you so far has been from the Default Profile. You can set up different profiles for different apps.

I happen to have a few Keyboard Maestro macros for Excel – to make it a little less tedious to use. So I created a profile for Excel.

So when Excel is in the foreground the Stream Deck looks like this:

I’m still setting it up – and you can see two undefined (blank) buttons. – but you get the general idea. The two Excel-specific buttons – using an image I got off the web[5] – kick off some really quite complex AppleScript:

  • Size Graph – which resizes the current Excel graph to a standard-for-me 20cm high and 30cm wide. This is fiddly so having a single button press to do it is rather nice.
  • Save Graph – which fiddles with menus to get to the ability to save a graph as a .PNG file with a single button press.

(In retrospect I should probably have changed the text to black – which is easy to do.)

So in real estate terms, application-specific profiles are really handy.

One pro tip: Close the Stream Deck editor if you want application-specific profiles to work. I spent ages wondering why they didn’t – until I closed the editor.

Conclusion

This programmable key pad with dynamically-changing keytops is a really nice way to kick off automation from a single key press (or a few if you use cascading menus).

I would go for the most expensive one you can afford – but I’ve shown you ways you can get round the 6-button constraint in the Stream Deck Mini.

Application-specific profiles are a really nice touch.

The whole thing is rather fun for an inveterate tinkerer like me. 🙂

In short, I think I’ll be throwing this in my backpack and taking it with me wherever my MacBook Pro goes.

And one day I’ll upgrade to a model with more buttons.


  1. As they say not to use a USB hub I don’t know if it would plug into USB C via an adapter.  ↩

  2. Pun intended.  ↩

  3. Except the OmniFocus one – where I had to find an icon – but even that was easy.  ↩

  4. At much greater cost, of course. Still more so with the 32-button variant.  ↩

  5. You can create your own icons using a nice editor they have, which contains quite a wide variety of icons. But Excel wasn’t one of them – so I found an image and used that. The preferred dimensions are a 72-pixel square.  ↩

Mainframe Performance Topics Podcast Episode 24 "Our Wurst Episode"

(Originally posted 2019-06-18.)

You’ll have to pardon the pun in our latest podcast episode’s title.

We were also somewhat delayed in getting this out – due to our busy schedules and a few technical gremlins. Hopefully it’s worth the wait.

It’s also quite a long episode so if you listen to it on your commute you’ll have to ask your chauffeur to drive a little more slowly. 🙂

Episode 24 “Our Wurst Episode”

Here are the show notes for Episode 24 “Our Wurst Episode”. The show is called this because we both attended the IBM TechU in Berlin, Germany, and our Topics topic is our trip report.

Feedback

  • We have some feedback (again) based on our use of stereo. We now have glorious mono, based on those comments!

Follow up

What’s New

  • APAR OA55959: NEW FUNCTION – PDUU Support for HTTPS
    • AMAPDUPL: Problem Documentation Upload Utility
    • How you get a dump to IBM, can be compressed, optionally encrypted, and sectioned into smaller data sets
    • HTTPS is important in this because dump can contain sensitive information and FTP is not an acceptable solution for many customers
    • FTPS had issues with e.g. firewalls,
    • Doesn’t look like this option has been incorporated into z/OSMF Incident Log at this time
  • Tailored Fit Pricing for IBM Z
    • Enterprise Capacity Solution
    • Enterprise Consumption Solution
    • Both different from traditional rolling four hour average model
    • For Tailored Fit Pricing, all machines must be IBM z14 Models M01-M05 or ZR1, and at IBM z/OS V2.2, or higher
    • More information here.
    • In Episode 19 Performance topic we talked about Licence-Related Instrumentation.
  • Ask MPT
    • Danny Naicker asks, “In z/OS 2.4 CSA subpool key 8–15, is it usable for user defined applications?”
    • Answer: Prior to z/OS V2.4 User Key Common Storage was available, but it was turned off by default. The downside was no control over who could use it.
      • In base V2.4 that specific capability has gone (the old system-wide switch).
      • Question probably originates from need to still use User-key CSA because of legacy stuff
      • This is where RUCSA (Restricted Use Common Service Area), a new function, comes into play. Allows you to identify applications by using a security definition.
      • Usage of RUCSA prior to V2.4 will need APAR OA56180
      • RUCSA will be offered in V2.4.
    • Thank you to Danny for a good question!

Mainframe Topic: CICS ServerPac in z/OSMF

  • IBM’s first delivery on new installation strategy, will be with CICS and associated SREL products. This is the first of many (really, all).
  • Choice on new installation strategy or old during ShopZ ordering. Choice is:
    • Old is ISPF CustomPac dialogs, or
    • New is z/OSMF Software Management and Workflows.
  • We encourage making the z/OSMF choice, as that is consistent between IBM and other vendors, and is intended to be easier.
  • Infrastructure already available in continuously delivery PTFs, and rolled back to z/OS V2.2. This makes the driving system have the proper infrastructure so anybody can package and deliver that way.
  • More details on the z/OS installation strategy:
    • Software vendors will package similarly, in a z/OSMF Portable Software Instance,
    • Clients will be able to acquire and deploy and configure using z/OSMF.
    • z/OSMF Software Management is used to the acquisiting and deployment. (“Deployment” is the new term for “installation”!)
    • z/OSMF Workflows is used for configuration. You would see the old ServerPac batch jobs as steps in a Workflow.
  • All software that you ordered as a ServerPac, and installed either way, will give you the same (or hopefully better) equivalent installation.
  • There is an IBM Statement Of Direction that this installation choice is coming, but we do not have an exact date yet.
  • For other software ISVs, they can exploit the new z/OS installation strategy whenever they are ready.
  • Prepare now by becoming familiar with z/OSMF Software Management and Workflows

Performance Topic: DB2 And I/O Priority Queuing

  • Follow on from Screencast / Blog post topic: Screencast 12 – Get WLM Set Up Right For DB2.
  • Recent talk has been about whether to turn off I/O Priority Queuing in WLM.
  • Service classes with DB2 subsystems in are heavily I/O Sample oriented, which is unusual among service classes in a system.
  • Means access to CPU is not properly managed, as CPU & zIIP samples few, relative to I/O samples. Reminder: Most of DBM1 is now zIIP-eligible.
  • Can achieve goal even with lots of delay for zIIP or CPU, but that’s definitely not what you want.
  • To see if it is properly managed:
    • See if there are lots of CPU / zIIP Delay samples in RMF Workload Activity.
    • In Db2 might well see Prefetch etc engines exhausted, which could cause unwanted Sync I/Os and bad SQL performance.
      • The effect is just like if there is a real zIIP shortage.
    • Instrumentation for DB2 of relevance is Statistics Trace.
  • You don’t want to just turn off WLM I/O Priority Queuing, as it’s sysplex-wide, it might affect other work that needs it, and Db2 might actually need it.
    • As the name suggests, it gives finer control over I/O priority.
    • So, it’s a case of proceeding with caution.
  • First you need a reasonably achievable goal for the service class. Make sure you’re more or less achieving the existing goal.
  • Second, calculate what the velocity achieved would be without I/O priority queuing .
    • Can take out the Using and Delay for I/O sample counts to do this.
  • If you don’t do the analysis and act on it a shift to not using I/O Priority Queuing could have unpredictable results.
  • You would know that turning off I/O Priority Queuing was helpful by seeing evidence that WLM is managing access to CPU for Db2 better, without hurting other stuff we care about. This evidence would come from RMF Workload Activity Report data.
    • On the Db2 side maybe Statistics Trace says Prefetch etc doesn’t get turned off. Or response times get better.
  • You should evaluate or adjust the goal attainment, but that is BAU. Changing WLM always needs some care.

Topics: Berlin Trip Report May 20–24

  • We both attended IBM Z TechU in Berlin, and got to see each other.
  • Marna had about six sessions.
    • The SMP/E Rookies session had fabulous attendance – 44. Some were more experienced, but most were not.
    • z/OSMF had good attendance too, about 82. More are interested in this topic, especially if you compare to just a couple of years ago.
    • Best attended was the z/OS V2.4 Preview, with about 150 people. There was excellent interest in what is coming in the new release.
  • Marna got to do a couple of things outside the conference:
    • Visiting the Reichstag was fabulous, but make sure to get a reservation.
    • Der Dom was also educational, with a walk to the top!
  • Marna did her own poster to help with z/OSMF configuration, and several people came by to chat.
  • Both Marna and Martin shared a poster about this podcast. We helped with getting one person a podcast app (on each platform), and a subscription to this podcast.
  • Martin had five sessions.
    • Two were with Anna Shugol, Engine-ering, and zHyperLink.
    • One was co-written with Anna, “2 / 4 LPARs”
    • Two were solo efforts: Parallel Sysplex Performance Topics, and Even More Fun With DDF
  • Martin also took a little time out of the conference
    • Each day took a session out to walk in the city.
    • It was interesting to wander round former East Berlin.
  • The next European IBM Z TechU is in Amsterdam May 25–29, 2020.

Customer requirements

  • RFE 131187
    • zOSMF RESTFILES PUT to remove Windows Carriage Return characters
    • Windows files contain a carriage return and line feed and the carriage return character x’0D’ is not being removed. The resulting zOS datasets therefore have a blank line after every data line that shouldn’t be there.

Future conferences where we’ll be

  • Both Marna and Martin in SHARE, Pittsburgh, August 5–9, 2019

On the blog

Contacting Us

You can reach Marna on Twitter as mwalle and by email.

You can reach Martin on Twitter as martinpacker and by email.

Or you can leave a comment below. So it goes…

Elementary My Dear Sherlock

(Originally posted 2019-06-08.)

Students of English literature won’t be alone in recognising the allusion in this post’s title.But this post isn’t about literature.[1].

Sherlocking, as described here is a phenomenon where a developer ships something – typically an app – but then Apple comes along and announces its own version of it.

In very recent memory, examples might be:

  • Enhancements to Apple’s Reminders apps on iOS and Mac OS versus Omnifocus and other “to do list” apps – in iOS 13 and Mac OS 10.15 Catalina.
  • NFC tags kicking off Apple Shortcuts automation versus LaunchCenter Pro’s support of NFC tags (described in Automation On Tap) – in iOS 13.
  • Apple Watch Calculator and PCalc – in Watch OS 6.

But what of it?

At first sight, having your app Sherlocked must be disheartening. But that’s not the end of the story.

What’s at risk for the vendor that gets Sherlocked is subscriptions and future sales. In other words, revenue. For some apps – particularly those that don’t use a subscription model – the sales pattern might be that most sales happen early in the app’s life. So before Sherlocking happens.

But it’s not that simple. Yes, Sherlocking represents a threat but it also represents an opportunity…

… A platform vendor legitimizes a marketplace by sensitising users to the value of a function. For example the new Watch OS Calculator app will introduce users to the idea of a calculator on their wrist.[2] But if PCalc were a better calculator than the Apple one it could still sell well. Fortunately it is. 😁

A built-in app gives one you have to buy a run for its money and free beats fee for many customers – if the basic function is good enough.

But the built-in (Sherlocking) app is usually relatively basic so a purchased app wins by differentiation from the basic. So the message is Sherlocked apps must up their game.

(For me, while I wouldn’t want to wear the “Power User” badge the basic function is rarely good enough. For example I use the excellent Overcast podcasting app instead of the Apple one and it’s inconceivable I wouldn’t use it and hardly conceivable I wouldn’t pay for the premium version.)

Having said that, first party apps have some additional opportunities for integration so could have some advantages. It’s difficult to compete against that. There are private APIs that only Apple can use – but those tend to open up over time.

Basic infrastructure – for example the Reminders database – done right allows third parties to use the infrastructure. Reminders data is shareable across platforms and with other third party apps designed to take advantage of the database.

A good example of a third party app using the Reminders database is Goodtask. Goodtask illustrates one downside of using the built in database, however: The developers had to use a lot of ingenuity to get round the limitations of the Reminders database: As they articulate here they use the Notes field for a task and had to invent their own metadata format.

Unfortunately Mail on iOS doesn’t have this open database and nor does Music.So email apps have to use their own databases – which is a real shame. It means, for instance, you can’t operate on the same email account with a mix of built-in and third party app functions. With Reminders and Goodtask you can.

Here’s an example of why I would want Mail to be open: My favourite email client – on Mac and iOS – is Airmail. I favour it because it has more automation hooks than the default Mail app. But I can’t use it with my work email account because it doesn’t share the default Mail app’s database.

Automation is one area where Apple – as the platform vendor – has an advantage. Those of us who are into automation are still waiting for better automation than x-callback-url[3] – as Shortcuts doesn’t provide a general automation mechanism for third-party apps. A more general mechanism would certainly help. But I digress.

In summary, Sherlocking does represent a threat but also an opportunity. The way to make it the latter is for Sherlocked app developers to continually innovate – to differentiate their apps from Apple’s. Thankfully the best developers are fleet of foot; I don’t envy their position but the best will survive.

If they do innovate at speed the net is the consumer benefits.


  1. Thankfully, given my background in Science and Technology. 🙂 But I’m not a complete philistine; Bits of me are missing. 🙂  ↩

  2. Not an entirely new idea, of course.  ↩

  3. I like x-callback-url so much I can often be seen sporting the t-shirt. 🙂  ↩

A Slice Of PI

(Originally posted 2019-06-01.)

A rather obscure pun, but I hope it’ll make sense. Not “Pi” by the way but “PI”. Though this post contains arithmetic, it’s not a mathematics post.

To be honest, I never knew how we calculated Performance Index (or PI) for Percentile goals before. But now I do, so I’m sharing it with you. Plus a couple of observations, too.

To be even more honest, when I say “I never knew how we calculated Performance Index” I should say “I never knew how we should calculate Performance Index” – as I’ve just corrected it.

(This post follows on (after 6 years) from WLM Response Time Distribution Reporting With RMF. More on that later.)

Before we go any further I have to give a little background information.

What Are Percentile Goals?

When Workload Manager (WLM) manages transactions explicitly two kinds of goals become available:

  • Average Response Time
  • Percentile

By the way, for WLM to manage transactions at all requires cooperation / exploitation by middleware or at any rate a work manager. Examples include:

  • CICS transactions require CICS support
  • DDF transactions require Db2 support
  • TSO uses the traditional transaction ending mechanism
  • Likewise Batch job steps

Average response time goals say something like “the average response time for this service class period should be 0.5 seconds”.

A Percentile response time goal might be “90% of all transactions in this service class period should finish in 300 milliseconds”.

What Is Performance Index – Or PI?

Performance Index (PI) is a measure of goal attainment.

  • A PI of 1.0 is where the goal is just met.
  • Higher than 1.0 and it is missed, the further away from 1.0 the worse the miss.
  • Lower than 1.0 and it is met, the lower the greater ease in meeting the goal.

The point about PI is that it is a metric for goal attainment that is neutral with regard to workload type.

PI is, of course, used to drive WLM’s algorithms. But I regard it as just the first metric. Others, such as WLM’s ability to help a service class period, are important too.

How Do We Calculate PI For Percentile Goals?

The calculation for goal attainment for Average response time goals is straightforward: Sum up the response times for each transaction and divide by the number of transactions ending.

The calculation for Percentile goals is more complex.

For any kind of transaction-based goal, at transaction ending WLM uses the transaction’s response time to assign it to one of 14 buckets. So WLM is counting transaction endings in these buckets.

The buckets have the following boundaries:

BucketMinimum
% Of Goal
Maximum
% of Goal
PI Value
10500.5
250600.6
360700.7
470800.8
580900.9
6901001.0
71001101.1
81101201.2
91201301.3
101301401.4
111401501.5
121502002.0
132004004.0
144004.0

The bolded values are of special significance, as we shall see.

Suppose we have a goal of “85% to complete within 0.2 seconds”. WLM knows how many transactions completed in each bucket and how many overall.

Suppose 1000 transactions completed. 85% of 1000 is 850 transactions.

Starting with Bucket 1, WLM tallies up the transaction endings until it meets 850 transactions. The upper limit of the bucket in which that happens is what determines the PI.

Suppose Buckets 1 to 3 tally up to 800 transactions and Bucket 4 contains 100 transactions. So Buckets 1 to 3 don’t meet 850 but Buckets 1 to 4 do.

Bucket 4’s upper limit is 80% of goal. So the PI is 80%/100 or 0.8.

Suppose it took Buckets 1 to 8 to reach or exceed 850. Then Bucket 7’s upper limit would be 110% and the PI would be 110%/100 or 1.1.

The code I inherited[1] didn’t do this calculation. But now it does.

Actually the calculation is not quite that simple: If by the time we’ve tallied up buckets 1 to 13 and we still haven’t reached that 850 number we set the PI to 4.0 (which makes sense).

Some Observations

From the above description of how PI is calculated for percentile goals, we can observe a few things:

  • PI can never be greater than 4.0, no matter how widely the goal is missed.

  • PI can never be less than 0.5 – as Bucket 1’s maximum is 50% of goal.

  • A PI of 0.5 is special in that it means enough transactions ended in Bucket 1, and some could be much shorter than 50% of goal. To get further definition we’d have to calculate the average response time.

    Now I calculate PI right I’m seeing 0.5 quite a bit.

  • If there are no transaction endings automatically the percentile goal is reached in Bucket 1 as all zero transactions ended there. So PI is meaningless with no transaction endings. so I force the PI to 0, but attempt to flag why in my reporting.

  • Because we’re using buckets there are only a finite number of values PI can take. Averaging over multiple RMF intervals will, of course, yield more.

Revisiting That Old Blog Post

This seems as good a place as any to follow up on WLM Response Time Distribution Reporting With RMF.

I made some refinements to the graph I showed there:

  • In the post I alluded to “near misses” versus “missed by miles” – and the counterpart on the “hits” side. I did indeed define “near” as +20% (and –20%) so Buckets 7–8 (and Buckets 5–6).
  • I added a green datum line for the % value in the goal.
  • I also added transaction rate. My code attempts to scale so that transaction rate doesn’t look ridiculous on a 0 to 100 scale.
  • I considered changing from a red-through-to-green spectrum but I think that’s less consumable. Besides, I like red/blue better.

Here is a modern case of a CICS transaction service class.

Here are some observations:

  • The datum is 95% because the goal is “95% in 1 second, Importance 1”.
  • A goal with “1 second” in it suggests pretty heavy CICS transactions. I’m not surprised there aren’t that many transaction endings.
  • When the transaction rate is significant the red pokes down below the green datum. Not just the pale red (“Just Outside”) but the darker red (“Well Outside”). You could say the blue shrinks away from it, if you prefer.
  • Not shown here but the PI is typically around 1.3.

By the way, WLM doesn’t have complete control over the response time achieved for a transaction. And that’s particularly relevant here.

This transaction goal service class is served by two region goal service classes. Both of these show almost no “Delay For X” samples. What they do have is lots of “Using I/O” and “Using CPU” samples.

So, to improve transaction response time it’s probably necessary to try:

  • Cutting the transaction CPU path length.
  • Reducing the I/O time by (and this is only an example) buffering the data better.

Neither of these are things WLM can do[2].


  1. And I hope I don’t sound defensive when I say that.  ↩

  2. This installation does not have Db2 so the “WLM-Managed Db2 Buffering” function doesn’t apply.  ↩

What's In A Name? – Revisited Again

(Originally posted 2019-05-27.)

It seems to be in the nature of my code development work that I revisit things over and over again. You could call it “agile”, you could call it “fragile”. 🙂 I prefer to think of it as being inspired by each fresh set of customer data.

And so it is with data set names. In What’s In A Name? – Revisited I talked about gleaning information from a data set name and using bolding to aid comprehension. This is an update to that, based on some new code.

But first a confession: I reopened this code because there was a bug in it. But while I was there I was inspired to capture another hill.[1] The very same dataset that caused my REXX to terminate with an error caused a nice piece of inspiration.

The Bad

The bug was in not recognising a Generation Data Group (GDG) data set could have “19” in its low-level qualifier. That led to misinterpreting the generation number as part of a date. (GDS low level qualifiers are of the form GnnnnVmm where the first variable part is the generation number and the second the version number – so “G1719V00” means “generation 1719, version 0”.)

That was an easy one to fix but on to the nicer part.

The Good

The data set name had “BKUP” as the last but one qualifier. This to my eyes signifies the data set is a backup for something. So I added a test to detect either “BKUP” or “BACK” in a qualifier and bold it if present.

I’ve made the code general enough so I can add further mnemonics – such as “OFFSITE” or “UNLOAD”. (In fact – since writing this post on the plane to Berlin I added full-qualifier matching for “OUT” and “NEW” as one of our job dossiers had data sets with these qualifiers.)

When I examine the data set names for one step in particular, every data set with “BKUP” as the low-level qualifier has a corresponding data set with an identical name, apart from the “BKUP” low-level qualifier being missing. The “BKUP” data set is an output data set (and I know about it because of its SMF Type 15 record). The matching data set is an input data set (and I know about it because of its SMF Type 14 record).

So I think we know what’s going on here[2]. 🙂

As I concluded in What’s In A Name? – Revisited there’s value in decoding data set names. And the more common gleanings I can do the better. This is just a nice little further step in that direction.

But I remain conscious of the possibility one could go too far, or get it wrong:

  • Not every word in the English language that appears inside a data set name is significant.
  • Not every significant word in a data set name is even English.

But we roll on. And no doubt the next study will bring fresh ideas for code. And that’s just the way I like it.


  1. Isn’t that always the way?  ↩

  2. To spell it out, some sort of backup processing.  ↩