(Originally posted 2013-10-14.)

While working through a scenario in our residency it became (briefly) important to be able to preserve sort order on a field. But this field wasn’t sorted in any recognisable way. So the records couldn’t be sorted alphabetically or numerically. In fact they had to be sorted so that this field was preserved in the following sequence:

red

orange

yellow

green

blue

violet

pink

white

black

hot

This post talks about two methods of maintaining this sequence.

• Using INREC / OUTREC / OUTFIL CHANGE.

• Using ICETOOL JOINKEYS.

Using CHANGE

The trick is to create an additional numerical field on which to sort – and then to throw it away. Use CHANGE to create it with coding like…

MYFIELD,CHANGE=(4, 
  C' red',X'00000001', 
  C' orange',X'00000002',
  C' yellow',X'00000003',
  C' green',X'00000004', 
  C' blue',X'00000005', 
  C' violet',X'00000006',
  C' pink',X'00000007', 
  C' white',X'00000008', 
  C' black',X'00000009', 
  C' hot',X'0000000A',
  C'NOTSEEN',X'FFFFFFFF'), 
  NOMATCH=(X'00000000') 

This can be used in INREC, OUTREC or OUTFIL. For one-pass sorting purposes INREC would be the place to do it. (But the syntax is OK elsewhere.) And you’d probably want to throw away the field in OUTREC or OUTFIL OUTREC.

Obviously you specify this temporary (4-byte numeric) field on the SORT statement.

The disadvantage of this approach is the table is hardcoded into the DFSORT invocation. You might not like that.

Notice the NOMATCH value of 0. This makes unmatched records collate to the front. You might use OUTFIL INCLUDE to move them to a side file you check for emptiness. (Use OUTFIL SAVE for the rest to send them to the normal output file.)

Notice also the “NOTSEEN” value which collates last. Actually it doesn’t matter where it collates and no input record has that value in the field. The purpose of the “NOTSEEN” line is to make sure the closing bracket isn’t on any real lookup lines. So you could code lines up to the CHANGE inline and the lines from NOTSEEN onwards inline. The lines between are the real lookup table and could be in a data set. Something like

//SYSIN DD *

...

MYFIELD,CHANGE=(4,
/*
//      DD DISP=SHR,DSN=HLQ.LOOKUP.TABLE 
//      DD *
  C'NOTSEEN',X'FFFFFFFF'), 
  NOMATCH=(X'00000000') 

...

This, I think, is reasonably maintainable.

(You might be able to think of another way to keep the closing bracket on a separate line. If so please let me know.)

Using ICETOOL JOINKEYS

If you don’t want to maintain the collation table in the DFSORT invocation you can keep it in a file and use ICETOOL JOINKEYS.

To do the sort would require an additional pass over the data – with the SORT statement on the looked up field in. For smallish amounts of data that’s probably fine. But for larger amounts of data you’ll probably want to use the CHANGE method and live with the maintenance of the table in the DFSORT invocation.

---

Maintaining sort order on a non-standard collating key like this looks important for when you are splitting jobs up to run against subsets of the data and want to bring things back together.

Our case creates a report sequenced in part on this nonstandardly sorted field. The first thing we do – to prepare for cloning – is separate the reporting from the data analysis and update. We use a transient file. When we clone we have multiple transient files and we need to merge them somehow. So maintaining sequence on this (actually the third) key is important:

Without forcing ourselves to define the clones as processing ranges of this field’s value we can’t just concatenate these transient files: We have to keep the order of this field preserved.

Extending The Idea

Though this isn’t relevant to the residency’s purpose – teaching people how to clone batch jobs – there is a nice extension to the idea of sorting using a lookup table.

With DFSORT’s arithmetic operators and other capabilities it’s possible to compute a temporary result and sort on that field. Exploring that idea I’ll leave as an exercise to the reader.

If you want to try it create a file with count and total fields in each record. Use these two fields to calculate an average and sort on it, optionally discarding the resulting average field.

---

This is the sort of practical issue we’re thinking through right now. It’s proving challenging but fun!

(Originally posted 2013-10-12.)

In What I’m Looking Forward To In z/OS 2.1 I mentioned processing VBS (Variable Blocked Spanned) data with REXX. This post describes what I learnt when I used it on our residency z/OS 2.1 system.

The most widely-known VBS data is SMF, though the Tivoli Workload Scheduler (TWS) Audit Log is also in this format.

A lot of different types of data are stored as Variable Blocked (VB) data, but this has the restriction that no record can be longer than the block size. Variable Blocked Spanned (VBS) data can, in contrast, contain records longer than the block size. SMF data often contains records which indeed are longer than the block size.

Prior to z/OS Version 2 Release 1 it was possible to process some SMF data by copying it to a VB data set. But this is risky as actually spanning records would be broken this way. (A typical example is SMF Type 30 Address Space records.)

Reading SMF

For my first experiment I extracted SMF 70 Subtype 1 records and printed the SMF ID (from the record header) and the Hardware and Software Model from the CPU Control Section.

To read the records you typically use EXECIO. My very first experiment used

"EXECIO * DISKR RMFIN (STEM RMFIN. FINIS"

but this caused the job to run out of memory. That’s because there was a lot of data – and the “*” means “read all the records”. I could’ve used

"EXECIO 1 DISKR RMFIN (STEM RMFIN."

(note no “FINIS”.) But I prefer to read, say, 100 records at a time.

When you read VB or VBS data REXX returns records without the 4-byte Record Descriptor Word (RDW). But SMF data contains offsets relative to the start of the record (including the RDW). The solution to this is to add three hexadecimal zero bytes to the front of the record. It’s three rather than four to take into account the fact REXX uses 1-based positions rather than 0-based offsets.

Writing SMF

Reading’s fine and allows you to experiment with the data. But writing is pretty useful, and I have a real use case in mind.

For my experiment I took the very same SMF 70–1 records and wrote them out. Again, no problem. (I did remember to take the three byte prefix off that I’d added when reading.)

Both SMF Dump and, more pickily, ERBSCAN and ERBSHOW were happy with the data.

The use case I have in mind is with different data: We’ll be running a fair number of batch jobs and there’ll be a naming convention that covers them. We’re only interested in these “test case” jobs and not e.g. compile jobs.

So I’ll write a REXX EXEC to read in the entire set of SMF 30 data and write out only the records that pertain to these jobs. Then my analysis code will run much faster (and so will getting the data to my home system be).

Certainly I can (and probably will) extend this to extracting the related SMF 101 (DB2 Accounting Trace) records and maybe the data set ones. But the 30’s will tell us how well our cloning efforts are doing.

---

I’m not recommending you replace more efficient SMF-processing tools (such as Tivoli Decision Support or SAS / MXG) with REXX for Production. But VBS support in REXX makes it easy to prototype analysis and to check data in a quick-to-write way. Which is exactly what I’d thought it would do.

And, to show we’re not all work and no play 🙂 the team is off to the Walkway Over The Hudson. It’s a nice day for it. 🙂

(Originally posted 2013-09-16.)

If you’re looking for a considered opinion you came to the wrong place. 🙂 Or so anyone reading Down In The Dumps? shortly after reading Enigma And Variations Of A Memory Kind might conclude.

It’s possibly a fair cop, possibly not, but it got me thinking…

In reality life is a sequence of experiences, many of which we hopefully learn something from. But it’s a journey of understanding and the question is when to “cut and run”:

Take the “dump accommodation” question, as it’s exemplified by the two posts I led with. You might consider it better to have written about both aspects as one post, rather than two. And that would’ve perhaps happened if I’d waited to pursue the “when is DUMPSRV busy?” line of enquiry. But, by that argument, you’re maybe never ready to publish.

There’re two mental models I have that relate to this:

1) The thinking peters out after a while – and that’s when you decide your opinion is a considered one.

(But when is that exactly? It’s rather like microwave popcorn, popping at a decreasing rate until maybe there’re no more pops.) But here you never quite know.

2) The thinking carries on for an arbitrarily long time, perhaps increasing and perhaps just varying.

In some things I think the “peters out until an opinion can be declared considered” is right but for most it isn’t: Because experience builds – if you let it.

One of the benefits of waiting for your opinion to be a considered one is incorporating amplifications and extensions. In normal conversation, though, a "forget everything since ‘good morning"’ situation occurs quite frequently. Fortunately that’s rare in stuff I write (and I’d like to think I recognise such things and handle them appropriately). So there’s a difference between conversation and publishing.

It seems to me sometimes I don’t give people the space to break in. That’s probably true (and not an endearing fault) but the “iterative” publication approach should give people the chance to break in and give their perspective. And for me to acknowledge it and handle it well.

I also think the “publish when you’ve got enough” approach is helpful in keeping post sizes down: Though you might disagree it limits them enough.

In summary, knowing when to publish and how much is a matter of judgment. It’s difficult to get it right and I wouldn’t claim I always do. Or in other words: This explains it all. 🙂

And that’s my considered opinion. Or is it? 🙂

(Originally posted 2013-09-14.)

That’s such a horrible pun I must’ve used it before. If so sorry (but not very). 🙂

This post follows on from Enigma And Variations Of A Memory Kind in a way. In that post I mentioned DUMPSRV, in almost a throwaway fashion: I happened to notice the memory usage in SMF 30 by DUMPSRV grew at just the point free memory took a dip.

This post takes that idea and extends it a little – and I think it might be something you want in your everyday reporting.

---

I ran a query against Data from all the customer’s LPARs in one pled – using SMF 30 data for DUMPSRV: I pulled out hours when the DUMPSRV CPU was more than 0.1% of a processor, printing the memory used, blocks transferred (think “I/O traffic”) and CPU. This highlighted that across the LPARs quite a lot of dumping happened, sometimes simultaneously on the systems. It made me think that “dump containment” is quite a big issue for this customer.

There are some issues with this approach:

• The granularity is 1 hour as I summarised to that level. With an SMF interval of 30 minutes it’s a little better but it’s still hard to correlate the surge in DUMPSRV in Type 30 with the time it actually occurs.

• I can’t tell who was dumped, just that a big dump capture took place at that point.

One thing that is worth working into the reporting is what happened to free memory at that point. If it was driven into the ground that’s a sign you need to take dumping seriously.

---

As with all such things it’s a matter of priority as to whether I write a “RDUMPSRV” REXX EXEC to detect this sort of thing. It wouldn’t take long.

More to the point I worked up this post from the one liner in the other one because I think it’s a technique worth thinking about: If you’re a Performance person it might not be obvious but you really do want to know about dumping prevalence, and substantial dump occurrences in particular. And if dumping does happen you’ll certainly want to be prepared to handle it in ways I’ve mentioned before – such as adequate memory, good paging subsystem design or, notably, zFlash.

(Originally posted 2013-09-11.)

As I mentioned in We Have Residents! we'll be working with a z/OS 2.1 system in October. In fact I've already logged on to it. I might even get to play with it before the residency starts, depending on current workload – but the priority is to hit the ground running by ensuring the environment is set up to our liking and that we have test data. (And then there are those day-to-day customers…) 🙂

Here are the things that've caught my eye that I'm particularly keen to to try out. Of course there's a lot in z/OS 2.1, so don't treat this list as the definitive list: It's just the things that leapt out at me that could change my daily life – as a programmer and regular user. And, yes, there are other things I like about 2.1 which aren't in this category.

• Regular expressions in FIND and REPLACE in the ISPF PDF Editor.

  At first I'll experiment interactively with this but I can see myself building it into REXX EXECs.

• Processing of VBS data in REXX.

  This might seem obscure but it really means SMF to me. (In the official materials I haven't seen SMF mentioned but SMF is VBS data so I'm hopeful.) Assuming my experiments are successful I'll write much more on this. This is the one most likely to be used in my code.

• Symbol processing enhancements in JCL. At this stage I don't quite know what we'll get out of this but it's the enhancement that's most likely to find its way into our Redbook: I hazard it'll be useful.

One of the nice things about spending 4 weeks in Poughkeepsie is the chance to discuss things with developers: They might ask us to try things out. And we might tell them what we think. 🙂

Of course it's not my production system. And I don't know how long I'll have access to the 2.1 system for. Still, I'll use it while I can and report any highlights (and, hopefully non-existent, lowlights).

And you probably will have your own favourite enhancements.

(Originally posted 2013-09-07.)

This post, unlike the previous one, is “on topic” for a Mainframe Performance Topics blog. I hope nobody’s relying on that. In fact I joke about renaming blog this to “Mainframe, Performance, Topics” 🙂 the next time I ask for the header to be updated. In fact I just might.

---

I recently got some data from a customer that I thought showed a bug in my code. Instead it illustrated an important point about averages.

We all know averages have problems – taken on their own – so this post isn’t really about that. It’d be a duller post if it were.

It’s about how free memory (or, if you prefer, memory utilisation) varies

• By time of day
• By workload mix
• From the average to the minimum

You’ll notice that last point didn’t mention the maximum. This is consistent with being more interested in the free memory than the utilisation, in a number of contexts. Let me explain:

As technology has moved on it’s become more feasible to configure systems (really LPARs) so that there is some free, or at any rate so that paging is (practically) zero. I’m concerned with how well installations meet that aspiration.

So the maximum free doesn’t interest me. But the minimum does. (And so does the average.)

---

Consider the following graph, from the customer I mentioned.

Before this week I plotted the blue line only (and that for each day in the data). This is the average of RMF’s average memory free number – by hour. But I had code that printed in a table the minimum free across the whole set of data (from a different RMF field in the same SMF 71 number).

While the blue line suggests just under 7GB reliably free, the minimum of the minima showed about 100MB free. This is where I thought my code was buggy – as these two numbers appear to contradict each other. I’d forgotten the 100MB number came from RMF’s minimum field and wasn’t just the lowest of the averages.

The red line is, as the legend says, the minimum free number from RMF plotted across the day. And the mystery (or enigma, to half explain this post’s title) is resolved: The 100MB number is the low point of the red line.

I’ll be throwing this graph into production shortly, maybe with a tweak or two (depending on how well it “performs”).

---

But the interesting thing – and the point of this post – is how free memory varies. (You’ll’ve guessed that the “Variations” in the title comes from this.)

If you look at the graph you’ll notice that, mainly, the big variability is overnight. Though there is a notable divergence between the two lines at about 11AM, they’re much closer together during the day.

If you’d asked me what I expected to see I’d say this is about right (but I wouldn’t’ve been certain, not having looked at the data this way before).

Overnight, fairly obviously, the customer runs Batch. And I can prove they do from SMF 30, of course. (I actually spent a happy year working on their Batch a while back.)

I would expect a Batch workload to show a higher degree of variation in memory usage, compared to Online (or whatever it’s called nowadays). For at least two reasons:

• Long-running work managers, such as CICS, MQ, Websphere Application Server and DB2, acquire most of their memory when they start up and variation in usage is slight thereafter. For example, buffer pools are likely to be acquired at start-up and populated shortly thereafter. (I’ve seen this in almost every set of data I’ve looked at over the past mumbleteen 🙂 years.)
• Batch comes and goes. (Job steps start, acquire storage, and release it when they end. Furthermore they’re generally highly variable in their memory footprint, one to another.)

Another thing I’m not that surprised about is Batch driving free memory to zero. Assuming large sorts are happening this is often to be expected: Products like DFSORT manage their usage of memory according to system memory conditions. Often they use what’s free if there is sort work data to fill the memory. This can be managed by the installation, if it is so desired. And there often won’t be enough sort work data to fill memory. (It’s not a direct objective to fill memory, but if there’s good use for the memory why not exploit it?)

---

Interestingly, on another system, on another day, the minimum free memory went to near zero (and the average followed it down, albeit in not quite such an extreme way). This time the cause was a surge in usage by DUMPSRV (from SMF 30). Clearly this was a time when a large dump was captured (or maybe several moderate-sized ones).

I often talk about configuring systems – particularly their memory and paging subsystems – to make Dump Capture as non-disruptive as possible. (Several blog posts over the years have talked about this.) This will be a reminder to talk to the customer about dump containment.

---

Now, the above may be obvious to you (and the value of comparing minima to averages certainly will be) but I hope there are some things to think about in it – notably how Batch behaves, catering for Dump Capture, and the fact RMF gives you the minimum (and maximum) and average memory free numbers.

But I also appreciate most of you don’t look at nearly as many installations as I do: If what you see in yours matches the above that’s fine. If it doesn’t it’s worth figuring out why not (but not necessarily thinking it’s a problem).

Translation: That’s a really nice graph you might like to make for yourself. 🙂

(Originally posted 2013-09-06.)

This post is in response to Kelly's post "Hashtags – love 'em or hate 'em? ".

My true response is "well, neither really." 🙂

A slightly more considered response would be to note that the utility of hashtags has decreased markedly over time:

• In the beginning there was e.g. Twitter without any kind of searchability.
• Then it got better.
• The End. 🙂

Seriously, if you wanted to find material on a subject, outside of something like Twitter that had no search, you'd use a standard web search engine. You might've noticed that I'm not motivated to put tags on my blog posts any more: I've seen far too good results with web searches to bother.

I also think the bother of curating tags is too much – and most tag-bearing sites don't make it easy. Two examples of this are this very blog and Evernote. By curation I mean things like merging two tags, most notably because of spelling or capitalisation issues.

And that's just dealing with my own tags. If we're talking about a tag you expect people to agree on it's even worse.

Having said that, I do plan on using the Twitter hashtag #batchres13 for the residency I'm running in October (and encouraging others to use it as well). That's more a "try to generate a little interest in what we're doing thing" than anything. (Though, used consistently, it might help us keep track of what's said about the residency on Twitter. We'll see.)

Kelly says something interesting:

"I don't attempt to measure my social business with hashtags, simply because I don't think they are ever static or consistently applied by the world at large."

I'd agree with that (and part of it goes hand in hand with what I said about curation). I'd also add that I have no pressing need to measure my social media effectiveness, but rather just to "do what I do". (I've said this before.) Others will have different imperatives and might take a different view.

A phenomenon I'm seeing more of is the "joke" use of tags, particularly on Twitter. Now that's something I can buy into, used lightly.

So, no, I'm not wild about tags either.

And, in case you wonder about the title of this post, the "cultural" reference 🙂 is to this. (This might be slightly NSFW.) I'm guessing I don't get many readers who aren't old enough to be allowed to play this. 🙂

(Originally posted 2013-08-02.)

In my experience Coupling Facility Duplexing configuration and performance is something that tends to get neglected – once the initial configuration decisions have been made. After all it’s rare that customers rework their Duplexing design.

Over the past few weeks I’ve been comprehensively reworking my Coupling Facility tabular reporting, as I recently mentioned in Coupling Facility Topology Information – A Continuing Journey .

This post is about the Duplexing part of that. If you agree it’s time to review your Duplexing reporting read on…

In the previously-mentioned post I talked about signalling rates and overall times at the CF level – for Duplexing. I have those now. While those are interesting they are rather macro level and don’t really talk about outcomes that directly affect applications. (Actually nothing does but I think you’ll agree specific middleware-related structures are more interesting than overall CF numbers when it comes to tuning e.g. Data Sharing applications.)

So let’s talk about structures…

User- Versus System-Managed Duplexing

First, there is only one exploiter of User-Managed Duplexing: DB2 Group Buffer Pools.

Second, the two types are very different from the instrumentation (and other) perspectives: Attempting to treat them the same is a bad idea.

Detecting Primary And Secondary Structures

Formally you need RMF SMF from the Sysplex Data Gatherer z/OS system. In several of my sets of data I don’t have that. So I have to improvise.

But first the formal bit: For the Sysplex Data Gatherer one Request Data Section is written for each structure. Bits in field R744QFLG in this section denote whether the structure is the old instance (primary) or the new instance (secondary) or neither. “Old” and “new” might seem strange names but duplexing is built on top of structure rebuilding, so the terms are not so strange.

If you don’t have data from the Sysplex Data Gatherer you can sometimes still get the answer:

For User-Managed structures (DB2 Group Buffer Pools) the traffic to the primary is higher than to the secondary. But if there’s no traffic you’re stuck. So my code performs the traffic test and reports accordingly.

Actually when I say “higher” I really mean “much higher”.

There is no such traffic test for System-Managed. In Performance terms the primary and secondary are generally identical: The traffic is much the same. So it doesn’t really matter which is which.

Similarly, for the zero-traffic case Performance isn’t a hot topic anyway. So again detecting which is primary and which secondary isn’t important.

Duplexing States And Timings

As I mentioned, User- and System-Managed Duplexing are somewhat different.

• With User-Managed, the “user” (DB2) coordinates writing to primary and secondary structures. So none of what I’m about to tell you applies to the User-Managed case.

• With System-Managed, XES and the two CFs coordinate: And operations in both CFs generally have to complete together.

The coordination for System-Managed manifests itself in the data in a series of fields in the Request Data Section for each version of the structure (whether the system is the Sysplex Data Gatherer or not). These fields generally have a count of events, total time for the events and the sum of the square of the times for the events – enough to calculate average and standard deviations.

So these events (and they’re reported in RMF’s Coupling Facility Activity Report) take System-Managed Duplexing down to the structure level. (The numbers are, unsurprisingly, zero for User-Managed.)

One thing they allow you to do is see one aspect of the Service Time cost of Duplexing. I say “one aspect” because, although there are timings in these fields, Duplexing introduces other effects.

For example a non-duplexed LOCK1 lock structure might have service times in the region of 3 to 20 microseconds, depending on link technology. (Here one would expect all the requests to be performed synchronously and these timings reflect that.)

But use System-Managed Duplexing with it and often most of the requests are performed asynchronously and with service times in the tens of microseconds (or hundreds over, say, extended distances).

But at least the structure-level counters and timings can help point out problems.

But there is a role here for the CF-level duplexing statistics: The path-level signal latency times for Duplexing links (if you have them) can also point to why Duplexing performance is what it is. RMF converts them to estimates of distance at 1 kilometer for each 10 microseconds, which is a clue that a lot has to do with distance.

---

One final word of caution: None of the RMF statistics related to Duplexing – of either flavour – say anything about application impacts from Duplexing. Where there is any evidence at all is from something like DB2 Accounting Trace where maybe the Asynchronous Database I/O Write Wait time is extended. But this is scant information at best.

Realistically the best you can do is give the performance-critical structures the best performance you can.

So, as you can tell, I’ve been busy warming over my CF Duplexing code. A few studies more and I’ll probably do it again. 🙂

(Originally posted 2013-07-30.)

Recently I wrote up some initial results of using OA37826 data in Coupling Facility Topology Information – A Continuing Journey .

That post in turn followed on from System zEC12 CFLEVEL 18 RMF Instrumentation Improvements .

Since then an interesting thing happened – and sooner than I thought it would: I got some data with a broken piece of Coupling Facility (CF) link infrastructure. Lest you think I’m insensitive about bad things happening to customer installations I’m going to say very little about the actual incident.

---

A colleague sent me a few hours of data from a customer I’d visited before. This customer has a mixture of generations of processors, including zEC12. I noticed the “Path Is Degraded” flag was set for a pair of CHPIDs between the zEC12 and one of the CFs, but that others between the pair were OK. (I also checked the flag that accompanies it to denote the flag is itself valid).

The first version of the code that detected this only listed the CHPIDs with the flag set. So I passed the CHPID list to the account team.

But I wasn’t satisfied with that: It occurred to me that actual configuration data would be better.

So I extracted the PCHID, Adapter ID and Port ID from the same section of the record (Path Data Section). Although the PCHIDs were, unsurprisingly, different the Adapter ID and Port ID were common to the two CHPIDs that were said to be degraded.

Not being that heavily into how you plug in CF links I’m not sure what that all means – but I’m going to have to learn.

In any case I’ve enhanced my code to give this additional information for degraded links. I’m also thinking I should add this information for non-degraded ones – as it might expose points of vulnerability. But I doubt I’ll get to do it until I get another zEC12 set of data in.

---

Well that’s the way it was when I first drafted this post. In the 36 hours since I’ve had a fit of “that really won’t do at all” 🙂 and a 3 hour train journey with wifi to press on with coding.

And I’m glad I did:

I can now see that the other two paths between this z/OS system and this coupling facility are on a different adapter. So it seems Installation Planning has been done well and the adapter isn’t a single point of failure. That’s something I’d want to check for in future sets of customer data.

---

Because I have only a couple of hours’ data I couldn’t detect the onset of the problem: If you’re a customer running daily reports you might want to create one to check for path degradation. I also didn’t get to see how performance was affected by the degradation: More data would’ve helped with that, too.

---

So the point of this post is to reinforce the view that you should – if it’s available to you – consider tracking the “Path Is Degraded” flag provided by OA37826 which externalises new support in CFLEVEL 18 (on zEC12 and the new zBC12 machines). Otherwise you’re relying on diagnostics and indicators that most of you wouldn’t ordinarily go near. Once again it’s very nice to have it in SMF.