(Originally posted 2012-08-04.)

I’m not a capacity planner but I play one on TV¹ sometimes.

A customer asked me about the subject of zAAP Capacity Planning so I thought I’d post a few thoughts here. (Almost everything I say here is equally true of a zIIP.)

The main point is I don’t think it terribly different from regular CPU Capacity Planning. But there are some quirks:

• While we do have queuing we also have Crossover. But at least we have good numbers for the latter.
• We have numbers for potential use – through the "Project CPU" mechanism.³.

One thing I’ve said consistently since Processor Pools became a meaningful concept (with the advent of ICF⁴ engines in 1998) is "Report and Manage Processor Pools Separately".

I still think that’s right but there are some considerations with that, most notably in when work eligible for one runs on another:

1. Crossover – where work that’s eligible on a zIIP or zAAP runs (partially) on a GCP⁵. These GCPs might well run slower than the zIIPs or zAAPs, of course. And that eventuality is catered for by RMF.
2. zAAP on zIIP – where zAAP-eligible work runs on a zIIP⁶
3. No zIIPs or zAAPs – where work has to run on GCPs instead.

Actually Cases 1 and 3 are quite similar, even if the mechanisms are different.

I’ll confess I took a decision a while back not to become too obsessed by what the various parameters that control crossover actually do. This is because they evolved somewhat over a short period of time. These days I prefer to see what the data says is happening in a given situation and think about how that could be improved. Usually it’s a trade-off between degree of offload to zIIP or zAAP versus responsiveness. (I’ve seen problem cases in both categories.)

At this point I’ll note that I’ve written a lot about zIIPs and zAAPs in the past. Here are the posts that spring to mind…

• When Good Work Doesn’t Go To zIIP / zAAP Heaven (2007)
• zAAP and zIIP Revisited (At Last) (2011)
• zAAP and zIIP Delay (2011)

So, all of the things I’ve talked about are things to bear in mind when doing zIIP and zAAP Capacity Planning, over and above the usual. So let’s talk about “the usual”…

You can readily figure out how much of the zIIP (or zAAP) pool is being used and by which address space. Many exploiters will tell you well below the address space level. For instance DB2 and DDF – using Accounting Trace. And much of that (via PROJECTCPU) is available for currently-running eligible workloads. So current usage is no problem.

Assuming your application isn’t changing its profile of e.g. zAAP vs GCP then forward projection is as it ever was. But, here are two cases where the ratio may well change:

1. Over time DB2 has changed the way its offload to zIIP function has worked – both eligibilitywise and how it actually distributes the benefit.
2. If you were to, say, replace JNI methods with native java then the offload proportion would change. In this case for the better. Maybe a different JDBC driver could cause that.⁷

So the assumption that an application doesn’t change its profile is probably right most of the time, but certainly worth keeping an eye on.

Maybe this post doesn’t answer the original customer’s question directly – but in an informal medium such as this I hope it contributes to the discussion. As I said at the start, this is pretty much “same as it ever was”⁸ but with a few wrinkles.

---

1. Mainly on Channel z². 🙂
2. OK I’ve done this joke before: The B-52’s – Channel Z Lyrics
3. I’m sure I’m stating the obvious when I say that this only works for workloads already running. Other estimation techniques are necessary for fresh ones. The most notable example of this is IPSec which most customers didn’t use prior to the advent of zIIP support.
4. Internal Coupling Facility, introduced with G5 processors.
5. General purpose processor.
6. But only if there are zIIPs but no zAAPs in the processor complex.
7. I think this happened with Type-2 to Type-4 driver conversions. Someone correct me if I’m wrong, please.
8. Second song reference in the one post. 🙂

(Originally posted 2012-07-10.)

I can tell when a CICS region came up – without looking at CICS-specific instrumentation. What’s more I can repeat the "trick" for any of MQ, DB2, IMS, etc – and so can you.

I’ve just started work on a new piece of reporting. I’ll call it "raddrspc" as that’s the name of the REXX EXEC that I’m writing. It’s about address spaces – most notably long-running ones.

In the mid-1990s when we were building our Batch Tuning offering PMIO, we came up with the term "Job Dossier". I think it was my term, but others in the team reading this are welcome to disagree. 🙂




The idea of a job dossier was to have reporting code put together – in a structured fashion – all the information about a particular batch job we could glean. Essentially from SMF but not necessarily so. I still use job dossiers in my Batch work. But the focus was on timings and things which ended i.e. batch jobs. What I’m starting on is analogous:  a "dossier" of stuff about an address space. Call it an "Address Space Dossier".

The key difference is that I’m interested in interval-level information, rather than events such as data set CLOSEs and steps ending. But I AM interested in events as well. "Interval-level" because I want to apply the Job Dossier idea to things that run for hours, days, weeks or months – such as DB2 subsystems and CICS regions.




It’s been interesting these past few weeks: I’ve been involved in lots of very diverse situations – but they all indicate one thing: It’s time to get nosey about address spaces. And the only way I’m going to do it is if I can write analysis code that makes me quick and aids structured thinking.




SMF 30 Interval records are cut for every address space I’m interested in – so that is a good unifying place to start. I’ll most certainly report on address spaces in an "idealised" way using this data. Then I will detect whether to go down the CICS "leg of the trousers" 🙂 , the DB2 DBM1 one, or whatever. Each of these have their own instrumentation. So if it’s available (for instance DB2 Statistics Trace) I can get specific using it.

There are other sources of information that work for any address space I might be interested in: One is SMF 42-6 Data Set Performance records – also cut on an interval. (Interval proclivities will make this complex, I’m sure.)

But for now, let’s talk about Up Time:




One of the first things I’ve done in raddrspc is extract from SMF 30 Interval records the Reader Start Time (fields SMF30RST and SMF30RSD) for the address space I’m interested in. Because I’m parsing all the interval records for the address space I can see if this changes. In my test case it certainly does: For CICS regions I can see a restart in the middle of my data – which is for a week. I can also see that the previous restart was 24 days before the data starts. And this picture is consistent across all the CICS regions.

So this is not a set of data from a customer who restarts CICS every night. I think that’s a useful nugget of information.

So, in your installation you can conclude similar things – if you don’t know them already.




As I carry on with building raddrspc – which frankly has to be a background activity – I’ll see if there are other pieces of information of similar interest and share them in this blog. I’m sure there will be – because I already know what some of them are.

(Originally posted 2012-07-07.)

It’s said that imitation is the sincerest form of flattery. In this Social Media age I’d say indirection comes a pretty close second. Indeed there’s a nice term for it: "link love".

Standard advice is not to just post links to the content of others. In reality the word "just" should probably be inserted: Don’t just post links to the content of others.

In that vein I want to point you to Page 9 of Xephon’s MVS Update 199 (April 2003). But I want to give you a little perspective – as my small slice of "value add".

(Before I do I would draw your attention to Xephon’s Copyright notice at the bottom of Page 2. And indeed I intend to stick to it myself.)

Many people alight at this blog after a search for something DFSORT-related: The Referer (sic) URLs tell me that. So I’m going to assume it’s enough to state that DFSORT Symbols allow a record’s structure to be mapped as named fields. You specify these in the SYMNAMES DD with lines like:

XYZZY,27,4,CH             A hollow voice says "Plugh".

This line has five parts:

• Symbol name – XYZZY
• Position – 27
• Length – 4
• Type – CH
• Comment –  A hollow voice says "Plugh".

 There are other formats but this post doesn’t require me to describe them.

Contrast that with a DSECT line with similar properties:

XYZZY          DS   CL8        A hollow voice says "Plugh".

One thing that’s missing is the Position (27). It’s implicit from previous statements in the DSECT.

Last week I went through an exercise of converting a DSECT (in fact DFHMNPDA) to DFSORT Symbols – in support of what I described in Another Way Of Looking At CICS Monitor Trace Data. It was a tedious exercise. I did it mostly through mass edits in ISPF. I thought "there has to be a better way", as one does under such circumstances.

My first take was it could be done by having REXX drive HLASM and parsing the output listing. There’s one thing I’ve neglected to mention. It is rather stating the obvious:

HLASM syntax is far more sophisticated than DFSORT Symbols syntax.

So you couldn’t, in general, parse an Assembler deck (in this case a DSECT) and hope to make a good conversion to a DFSORT Symbols deck. A while ago I embarked on this route – after a previous bout of frustration at creating Symbols from a DSECT. It’s the wrong approach.

But getting the Assembler to do the hard work is the right approach.

But friends convinced me – via Facebook 🙂 – that parsing the HLASM output listing is the wrong approach. The information is there but parsing listings with code is just plain miserable – as experiences go. The correct approach is to parse ADATA from HLASM – as it has structured data. Parsing it is a more robust thing to do.

Fortunately, before I had a chance to embark on this (admittedly hobby) project I noticed the Xephon link at the beginning of this post.

I will admit I haven’t even nosed through the listing, yet alone tried the program. I’d assume it’s good – else it wouldn’t’ve been published.

And the reason I’ve not looked at the code is because I may one day want to do something like this and wouldn’t want the copyright issues. Further, I already guess I would want my code to do more. Here’s a for instance:

In the DFHMNPDA DSECT there are numerous XL12 fields. These are STCKE-based fields. As I mentioned in the previous blog post they were widened from 8 bytes to 12 in CICS TS 3.2. My way of processing them is to take bytes 4 to 7 and multiply by 16 to get microseconds. So the DFSORT Symbols that map these 12 bytes will be something like:

SKIP,4
PDRCPUT,4,BI       CPU Time
SKIP,4

I doubt the referenced code would generate that – as it’s pretty specific. If I wrote some code it might well have an option to do that. And there are a number of things I could imagine doing that add value to the basic conversion process – such as generating some diagramming.

Anyhow, give the referenced code a try. I’d like to know how you get on. And if anyone could tell me who wrote the code I’d be glad to acknowledge them here.

(Originally posted 2012-07-02.)

DFSORT’s Arithmetic operators can do many things but the one thing they can’t do is take the square root of a number.

You might think that’s minor but it means you can’t calculate a Standard Deviation. (Variance is fine but dimensionally not so nice when used in conjunction with the Mean.) And I needed Standard Deviation in a real live customer situation.

So I set out to "roll my own". And here’s how I did it…

There is a well known technique for calculating square roots – the Babylonian Method (or Heron’s Method). It’s not terribly advanced maths, frankly: I think I was taught it at about age 14. It goes like this:

1. Call your number whose square root is to be found a, and pick a first guess for the root. We’ll call this quess x₀.
2. Write x₁ = (a + a/x₀)/2. In other words x_n+1 is the average of a and a/x_n.
3. Iterate until done. We’ll talk about what "done" means later.

As you can see it’s a really very simple method and converges reasonably quickly. And it’s not difficult to program, except for one thing: DFSORT doesn’t have a looping capability. (Yes it loops over the input records, but not in the way I mean.)

Before I show you some code let me talk about some considerations:

• Because DFSORT’s arithmetic is integer I describe here taking the integer square root of an integer field value. So, for example, it yields sqrt(5)=2.
• All the arithmetic on the way to finding the root is integer also.
• The code takes square roots of non-negative integers: No imaginary roots here.
• The method returns positive roots: As you probably know roots come in pairs, differing only by a sign.
• The algorithm requires you to provide a first estimate (x₀). My first take was a/2. But in the case of a=1 this leads to a root of 0 straightaway – which is particularly unhelpful. So I had to special-case a=1. So the take I went with in the end is x₀=a. Not generally a brilliant first guess but avoids the special case.
• Because this is integer arithmetic completion can’t be only when sqrt(a)*sqrt(a)=a as in most cases this condition isn’t going to be met. Instead I use x_n+1=x_n as completion: If the two are equal the algorithm isn’t going to produce a different x_n+2.

A reasonable question to ask is whether integer square roots are useful…

A standard technique with DFSORT’s Arithmetic operators is to boost the integer values to represent the number of decimal places required. So, for example, to add together 3.14 and 2.72 you’d be storing them as 314 and 272. Then you’d add these integers and use EDIT=(I.TT) to display them as "4.86". So if you wanted to take the square root of 3.1416 you’d multiply by 10,000 and then apply my technique, using EDIT=(IIII.TT) to move the decimal point back.

Structure Of The Example

This is a standard DFSORT invocation. Rather than give you the usual scaffolding I’ll show you the main components:

• DFSORT Symbols where possible to help keep it clear.
• COPY (though it could’ve been SORT or MERGE).
• INREC to parse the free-form input data.
• OUTREC with multiple IFTHEN clauses to compute the square root.
• OUTFIL to format the output lines.

You could, in fact, structure this program differently. It might (though I haven’t worked through it) combine everything into a single INREC statement (apart from the COPY specification).

Some Sample Data

In my SORTIN I have the following lines:

100 
5 
1 
27 
3 
4 
999 
999999 
99999999

In other words a single field in each record in EBCDIC (rather than binary) form. (You can generalise all that follows with a small amount of thought.)

(In my test case this is instream data – so it’s fixed-length records and hence no RDW).

Parsing EBCDIC Numbers.

In this simple case I created a set of records with binary representations of the numbers using INREC:

OPTION COPY 
INREC FIELDS=(1,10,UFF,TO=BI)

Here the UFF does the parsing and it parses the 10 bytes starting in position 1, with the output being 4-byte BI fields.

A SYMNAMES Deck To Simplify Things

You may know you can define symbols with DFSORT. Here’s the deck I used in this example:

//SYMNAMES DD *
INPUT,1,4,BI 
SQRT,*,4,BI 
PREV,*,4,BI 
/* 

(You’ll want a SYMNOUT DD to go with it.)

• INPUT maps the integer value created by INREC. It’s a, the number whose square root we’re seeking.
• SQRT maps the field where successive estimates of the root are stored.
• PREV maps the field where the previous estimate is stored – so we can tell if an iteration has changed the estimate of the root.

Taking The Square Root

In my example all the work is done in a series of IFTHEN clauses in an OUTREC statement:

1. A single WHEN=INIT clause initialises the estimate (SQRT) and clears the previous estimate (PREV).
2. Repeated WHEN=(SQRT,NE,PREV) clauses compute successive approximations – but only if the last two estimates weren’t the same.

Here’s the code:

   OUTREC IFTHEN=(WHEN=INIT, 
       OVERLAY=(SQRT,INPUT,X'00000000')), 
     IFTHEN=(WHEN=(SQRT,NE,PREV),HIT=NEXT, 
       OVERLAY=(PREV:SQRT,SQRTSQRT,ADD,(INPUT,DIV,SQRT)),DIV,+2,
         TO=BI,LENGTH=4)), 
     IFTHEN=(WHEN=(SQRT,NE,PREV),HIT=NEXT, 
       OVERLAY=(PREV:SQRT,SQRTSQRT,ADD,(INPUT,DIV,SQRT)),DIV,+2,
         TO=BI,LENGTH=4)),
         
         ...
          
     IFTHEN=(WHEN=(SQRT,NE,PREV),HIT=NEXT, 
       OVERLAY=(PREV:SQRT,SQRTSQRT,ADD,(INPUT,DIV,SQRT)),DIV,+2,
         TO=BI,LENGTH=4))

This you’ll recognise as an unrolled loop. The "…" says you can repeat the previous IFTHEN clause as many times as you like.

A good question would be "how many times?" I found good convergence after 15. For example, the 99999999 value went to 10028 in 15 iterations and 10000 in 16. You could pick 20 and be sure of getting an accurate value.

Let’s talk about control flow through the queue of IFTHEN clauses…

• The WHEN=INIT is always executed.
• HIT=NEXT on a WHEN=(logexp) clause means "even if this test succeeds proceed to the next WHEN=(logexp) clause having executed this one’s OVERLAY".
• Even if a WHEN=(logexp)’s logical expression is false all subsequent WHEN=(logexp) clauses are processed – even if, as in this case, they’ll evaluate to false. So this isn’t really a "WHILE(logexp) DO … END" construct (but you can construct the logexp so it works that way).

In the past I’ve likened multiple IFTHEN clauses to a pipeline. This example shows why.

Printing The Results

The following is a quite complicated piece of report formatting. Its aim is to format numbers and then squeeze out the resulting leading blanks. At the same time it has to preserve blanks in text.

OUTFIL IFTHEN=(WHEN=INIT, 
    BUILD=(C'x=',INPUT,EDIT=(IIIIIIIIIIT), 
      C'_sqrt(x)=',SQRT,EDIT=(IIIIIIIIIIIT), 
      C'_sqrt(x)*sqrt(x)=',SQRT,MUL,SQRT,EDIT=(IIIIIIIIIIIT),
      C'_prev_est=',PREV,EDIT=(IIIIIIIIIIIT))), 
  IFTHEN=(WHEN=INIT, 
    BUILD=(1,90,SQZ=(SHIFT=LEFT,PAIR=QUOTE))), 
  IFTHEN=(WHEN=INIT, 
    FINDREP=(IN=C'_',OUT=C' ')) 

Let’s examine it more closely:

• The first WHEN=INIT takes four numbers and formats them:

  • The number we wanted to find the square root of,
  • The square root,
  • What happens when you square that back up again,
  • The previous estimate of the square root.

  These are formatted with leading zeroes turned into blanks and some text is wrapped around them (with underscores standing in for blanks).

• The second WHEN=INIT squeezes all the blanks out. This is why I used underscores in the wrapping text.

• The third WHEN=INIT turns all the underscores into blanks.

Output Using The Sample Data

The output below isn’t particularly pretty. I wanted to use the OUTFIL code I’ve shown you to demonstrate the ability to selectively squeeze blanks out.

x=100 sqrt(x)=10 sqrt(x)*sqrt(x)=100 prev est=10                 
x=5 sqrt(x)=2 sqrt(x)*sqrt(x)=4 prev est=2                       
x=1 sqrt(x)=1 sqrt(x)*sqrt(x)=1 prev est=1                       
x=27 sqrt(x)=5 sqrt(x)*sqrt(x)=25 prev est=5                     
x=3 sqrt(x)=2 sqrt(x)*sqrt(x)=4 prev est=1                       
x=4 sqrt(x)=2 sqrt(x)*sqrt(x)=4 prev est=2                       
x=999 sqrt(x)=31 sqrt(x)*sqrt(x)=961 prev est=31                 
x=999999 sqrt(x)=999 sqrt(x)*sqrt(x)=998001 prev est=1000        
x=99999999 sqrt(x)=10028 sqrt(x)*sqrt(x)=100560784 prev est=10784

Although this has been quite a complex post I hope it’s taught you a few new DFSORT tricks. Actually, the “meat” of it is the Square Root calculation – which I’m using “in anger” (AKA “for real”) right now. And the technique should be readily adaptable.

The question is: Are there other Newton-Raphson-style computations worth doing?

(Originally posted 2012-06-30.)

The CICS Monitor Trace SMF Record (Type 110) has got to be one of the most complicated SMF records in existence – and for good reason.

Which is precisely why I’m not going to attempt to process the raw records in my code. (And why PMCICS doesn’t support releases after a certain point.)

But the "and for good reason" hints at the fact I think this is a tremendously valuable type of instrumentation – because it goes down to the individual transaction instance level.

You’ll have seen from He Picks On CICS I think CICS is a very interesting product to work with, performancewise. (And one day soon I hope to get to play with it from an Application Programming perspective.)

In that post I briefly mentioned CICS Performance Analyzer (CICS PA) – which I’ve spent a lot of time with recently. I like it a lot and the reports (and CSV files) it produces have been very helpful. Not least because it helps sort out the nested components of transaction response time.

In this post, however, I want to outline another approach. If you don’t have (and can’t get) CICS PA this might help you. It might also inspire you if you want to do some custom reporting that CICS PA doesn’t do. (In skilled hands maybe CICS PA will do what my experiment below did. If so it’ll probably do it better.) This approach is to use standard CICS tools and DFSORT to process individual 110 records…

Standard CICS Tools

I’m using two CICS tools. They are both distributed in the SDFHLOAD load library, so if you have CICS you have them. They are:

• DFHMNDUP – which creates CICS Dictionary records, essential to processing Monitor Trace records. This is in SDFHLOAD but not SDFHSAMP but I actually don’t need to know how it works or to modify it. It’s A Kind Of Magic 🙂
• DFH$MOLS – which reads (and decompresses) Monitor Trace records and provides a sample report. It also can create a one-output-record-per-input-record data set. It is in SDFHSAMP – which proved crucial as I needed to know how it works.

DFSORT

We all know (and I hope love 🙂 ) DFSORT and ICETOOL.

---

Putting It All Together

DFH$MOLS needs Dictionary Records for every CICS region – because these records tell it how to process the records. Because you can tailor Monitor Trace records from a region using its Monitor Control Table (MCT) DFH$MOLS can’t assume they’re in a totally standard format. (Actually I see fewer customers tailoring their MCTs, perhaps because there’s less emphasis on space minimisation.) I said at the outset the record is complicated and this is part of the good reason why.

So, prior to running DFH$MOLS I ran DFHMNDUP once for each CICS region. (From Type 30 I already know the jobnames for the regions and the CICS region names happen to be the same – as is often the case.) Though you can feed in the MCT for a region I didn’t need to – because the set of data I’m working with is from a customer who doesn’t tailor their MCTs. So I ended up with five Dictionary Record data sets – each with one record in. (This customer has a single TOR, three AORs and a FOR.)

When running DFH$MOLS I concatenated these five Dictionary Record data sets ahead of the Monitor Trace data set. You have to do it in this order so the Dictionary Record for a region precedes all the Monitor Trace ones for the same region.

While DFH$MOLS can produce formatted reports I wanted to create raw records with the fields in fixed positions. (You can tell where this is headed.) So I specified UNLOAD LOCAL. The "LOCAL" is quite important: It ensures timestamps are in the local timezone. Otherwise timestamps are in GMT. (I actually did go down this "blind alley" first time round.)

Processing was surprisingly swift and the end-result was a data set with records reliably laid out the way I needed them: I’m out of the game of worrying about record formats and record compression etc. Result! 🙂

The goal of  this experiment  was to produce a file with one record per minute – for a single transaction in a single region. The record contains three pieces of information:

1. Time of the minute – e.g. "15:30".
2. Transactions per second – averaged across that minute.
3. Average response time – in milliseconds.

In fact I could’ve formatted these all differently (and summarised differently). So I could’ve used tenths of milliseconds (almost standard – "decimils" or "hectomics"?) and summarised by the second, as two examples.

As I mentioned before I needed to understand some aspects of how DFH$MOLS works – so I read the Assembler source. I’ll admit to not being very familiar with the details of Insert Under Mask (ICM) but it turned out to be important…

The response time for a transaction is given by subtracting the raw record’s start and end times – according to the code. These turn out to be the middle 4 bytes of a STCK (not "E") value. Following the chain of logic further back – and I won’t bore you with the details – it turned out the value is in 16 microsecond units. With DFSORT Arithmetic operators it’s easy to translate that into milliseconds – but best done after summing and piding by the number of transactions.

The timestamp for a transaction is of the form X’02012100′ followed by a value in hundredths of seconds. Again quite easy to turn into something useful in DFSORT. (This time I didn’t need the DFH$MOLS source to tell me this – I browsed the DFH$MOLS output in Hex mode and it was readily apparent.) As I said before my first go was without the "LOCAL" option so the timestamps were from a different time of day and it was obviously GMT. With "LOCAL" I could see it all coming right.

---

There Be Dragons

The above is quite straightforward – even if the bits about ICM and timestamps left you cold. 🙂 But there are some issues worth noting…

I used the CICS 650 (externally TS 3.2) libraries. This was fine as the records are from TS 3.2 CICS regions. This isn’t generally a stable thing to do – as everything in the process could change from release to release. So run the right DFHMNDUP – if you need that step. Also run the right DFH$MOLS.

In CICS TS 3.2 timestamp fields were widened from 8 bytes to 12. As this web page says DFH$MOLS got it right and the UNLOAD format is still the same as before. Critical, therefore that you have the right DFH$MOLS release.

While I don’t expect the timestamps to get widened again any time soon, there might be further similar changes going forwards. So, to repeat, make sure you’re using the right DFH$MOLS (and DFHMNDUP).

I’m sure there are other dragons but they haven’t singed my cardie yet. 🙂

---

What Next?

I started to create a DFSORT Symbols mapping of the output of DFH$MOLS UNLOAD but – for reasons of time – only managed to get the first few dozen fields mapped. Sufficient to do my "day job". I could extend that – and it might make a nice sample.

I could publish my DFSORT experimental case as a sample – possibly building on  the Symbols deck. I could try and keep track of the twists and turns of the UNLOAD records – using (for example) IFTHEN processing. I’m really not missioned to do that. 😦

I could develop some more examples. Again, I’m not missioned to do that. 😦

I hope this post has encouraged some of you to experiment with this approach.

What I’m going to really do next is explain the following:

For the first 45 minutes or so the transaction rate stays more or less constant but the response time is all over the place, including some minutes where it spikes to huge (but realistic) values. Then, suddenly, the response time settles down to a low value – and the transaction rate remains more or less the same as it was before. And stays that way to the end of the data.

I think I have an explanation: From RMF I can see certain things awry until about that point (with the standard 15-minute granularity). To really explain it at the transaction level would take a more detailed analysis. Fortunately I can do this with the DFH$MOLS output. And CICS PA has already told me which fields are likely to be the most useful…

(Originally posted 2012-06-24.)

It’s been a while since I last posted here. And that’s been more a matter of being incredibly busy than anything. Some of you will have seen me "on stage" – there having been several conferences and user groups recently. Some of you were with me at the zChampions meeting and some of you I’m actively working on customer situations with.

(By the way one of the rules of zChampions isn’t "nobody mentions zChampions". 🙂 But it’s true to an extent that "what happens at zChampions stays in zChampions”. 🙂 But only to an extent.)

I won’t signal a let up in activity by suggesting I’ll post more frequently: That’s probably unrealistic.

But in this post I want to talk about z/OS Unix Pipes in JCL vs BatchPipes/MVS. The reason for doing this is that in some circumstances z/OS Unix Pipes in JCL can play a similar role to BatchPipes. But there are caveats. In what follows I’ll generally use the term “Unix Pipes” ¹ as shorthand for “Unix Pipes in JCL” (except for the next heading where I think it’s clearer to write it in full).

Unix Pipes in JCL

Here are two sample pieces of JCL you can play with:

First a writer job. Let’s call it "W":

//WRITE1  EXEC PGM=IEBGENER 
//SYSPRINT DD SYSOUT=K,HOLD=YES 
//SYSIN   DD DUMMY 
//SYSUT1  DD * 
ABCD 
/* 
//SYSUT2  DD PATH='/u/userhfs/pmmpac/pipe1',DSNTYPE=PIPE,
//        LRECL=80,BLKSIZE=3200,RECFM=FB, 
//        PATHOPTS=(OWRONLY,OCREAT), 
//        PATHMODE=(SIWUSR,SIRUSR), 
//        PATHDISP=(DELETE,DELETE) 

And now a reader job ("R"):

//READ1   EXEC PGM=IEBGENER 
//SYSPRINT DD SYSOUT=K,HOLD=YES 
//SYSIN   DD DUMMY 
//SYSUT1  DD PATH='/u/userhfs/pmmpac/pipe1',DSNTYPE=PIPE,
//        LRECL=80,BLKSIZE=3200,RECFM=FB, 
//        PATHOPTS=(ORDONLY,OCREAT), 
//        PATHMODE=(SIWUSR,SIRUSR), 
//        PATHDISP=(DELETE,DELETE) 
/* 
//SYSUT2  DD SYSOUT=K,HOLD=YES 

The pipe here is given by the PATH name “/u/userhfs/pmmpac/pipe1” (and you’ll have to figure out what it should be in your environment).

In this example I’m just copying from instream data (the “ABCD” line) to the pipe and copying from the pipe to the SPOOL. Simple stuff you can get working and build on.

I experimented with PATHOPTS, PATHMODE and PATHDISP and the ones in the sample JCL decks are the ones that worked best. In particular PATHDISP=(DELETE,DELETE) seemed to be necessary to get the pipe deleted after use.

When I run these they appear to perform as a BatchPipes/MVS pipe would: Overlapped and apparently without real I/O² – which are the key benefits.

To those who know BatchPipes/MVS this should all look quite familiar…

Unix Pipes vs BatchPipes/MVS

As I said, they’re quite similar. But there are differences that are worth noting. The most notable ones are:

• While BatchPipes can “add value” with fittings in the pipe-end definitions Unix Pipes can’t – you’d have to have a separate intermediary job (I tend to call “F” for “Filter” / “Fitting”. You can write it in whatever you like. Suggestions?
• You don’t get the equivalent of SMF 91 – which has handy things to enable you to e.g, balance the pipe.
• You don’t have a subsystem – and that’s probably a good thing. Instead you’re using the Unix file system – as the pipe name suggests.
• To convert a non-piping usage to a piping one requires significantly more JCL “surgery”. In particular you can’t use the data set name for the pipe name. And you can see things like DISP are completely different. So it’s more work.
• BatchPipes/MVS has the BatchPipePlex capability to pipe between Parallel Sysplex members using Coupling Facility structures. I don’t think you can do anything like that with Unix Pipes.

There are probably other differences but these are the most noteworthy to me – as someone who’s a lot of experience with BatchPipes/MVS. Some of them definitely can be worked around – such as the lack of Fittings support. Some of them are more difficult. But if you just want a simple inter-job piping capability have a go with Unix Pipes – perhaps starting with the two JCL samples above. And one key advantage of this approach might be that it’s free and available to all (I think) – so long as your jobs are allowed to use Unix System Services³.

And as to posting frequency: I feel under no pressure to publish – whether timing-wise or content-wise. If there’s stuff I think is interesting (and not damaging to a client) you’ll see it here – when I get the chance.

---

¹ Yes, I know Unix shells have their own built-in piping capabilities. Discussing those is beyond the scope of this post.

² I’m told no Unix piping implementation would perform physical I/O – so this isn’t a queue on disk (with the issues that would raise: I/O time and potential physical limits).

³ And I note the perpetual discussion on the IBM-MAIN Listserver group about whether “USS” is an acceptable abbreviation for “Unix System Services”… 🙂

(Originally posted 2012-05-12.)

With just over a week to go I’ve got my presentation materials in for this great conference: IBM System z Technical University 21-25 May in Berlin. I hope to see many friends – old and new (old and young) 🙂 there.

 

For the record my three sessions are:

• zZS08 – I Know What You Did Last Summer
• zZS18 – Optimizing z/OS Batch (repeats)
• zZS21 – Parallel Sysplex Performance Topics

And, as well as seeing me present (which I presume you’d want to else why are you reading this blog?) 🙂 there are lots of great sessions – at all levels of complexity on all kinds of topics.<

See you there

(And if you’re not going all of these are, I think, on Slideshare.)

(Originally posted 2012-04-29.)

If you think this title is obscure bear in mind the original working title was "Send In The Hobgoblins". ¹ 🙂

When I started to write – actually before the "mind mapping" stage – it was going to be all about inconsistency in the way bits of systems are named. You'll see some of that reflected in the finished article (pun intended) but the post has mostly gone in a different direction.

I'd maintain this one is a slightly less obscure title. But I accept it depends on your pronunciation of "CICS". I've heard many nice variants ² but I'm depending heavily on just one. (And, obviously, it's my preferred one.)

I thought it'd be interesting to do a "thought experiment" ³ on what you can glean about CICS from SMF. This is a necessarily brief discussion – though it might be worth working up into a presentation one day – and I've probably touched on some of this before. If I have I hope I don't contradict myself too badly here. (Strike One for consistency.) 🙂

I'm going to do this two different ways: I'll talk about

• Data
• Themes

This isn't meant to be an exhaustive survey but is more intended to get you thinking. And in particular in the Themes section you can probably think of your own themes.

Data

As with every application address space, CICS regions can be looked at using standard SMF 30 Interval records.⁴:

• Most notably, you can identify CICS regions from the program name – DFHSIP – and can establish usage patterns such as CPU and memory.
• From RMF Workload Activity Report data (SMF 72 Subtype 3) you get WLM setup and goal attainment information. The SMF 30 record also contains the WLM workload, service class and report class names so you can easily figure out which CICS regions are in which service class, etc..

Obviously generic address space information can only get you so far. To go further you need more specific information. I'm going to divide it into three categories:

• CICS-Specific
• Other Middleware
• I/O

CICS-Specific

CICS can create SMF 110 records at the subsystem and the transaction level – both of which can be reported on by specialist tools using CICS Performance Analyzer (CICS PA) or more general SMF reporting tools.

Such information contains subsystem performance information, response time components for transactions and virtual storage.

Other Middleware

You can get very good information about when CICS transactions access other middleware:⁵

• For DB2 SMF 101 Accounting Trace gives you lots of information about application performance – as we all know. For CICS transactions the Transaction ID is the middle portion of the Correlation ID (QWHCCV) and the Region is the Connection Name (QWHCCN).⁶
• Similarly, Websphere MQ writes application information in the SMF Type 116 record, which can be related to specific CICS regions and transactions.

I/O

Most performance people know about SMF 42 Subtype 6 Data Set Performance records. For data sets OPENed by the CICS region, these records are cut on an interval basis and when the data set is CLOSEd. (This obviously isn't true, for example, for DB2 data.) These records can be used with the File Control information in CICS 110 to see how, for example, LSR buffering and physical I/O performance interact for a VSAM file.

---

Themes

That was a very brief survey of the most important instrumentation related to CICS. Much of it is not produced by CICS itself. I kept it brief as it's perhaps not the most interesting part of the story: I hope some of the following themes bring it to life.

Naming Convention

(Strike Two for Consistency coming up.) As someone who doesn't know your systems very well it's interesting to me to figure out what your CICS regions are called. And which service classes they're in. etc.

So, to take a recent example, a customer has two major sets of CICS regions cloned across two LPARs. In one case SYSA has CICSAB00 to CICSAB07 and SYSB has clones CICSAB08 to CICSAB15. In the other case SYSA has CICSXY1, 3 and 5 while SYSB has CICSXY2, 4 and 6. Each of these happen to be in their own service class.⁷

You'll've spotted what I like to call "consistency hobgoblins" 🙂 in this:

• One alternates between systems. The other has ranges on each system.
• One starts at zero. The other starts at 1.

The customer took my teasing them about this inconsistency very well – so I don't think they'll mind me mentioning it here (particularly as, apart from them, nobody will recognise the customer).

And actually it doesn't matter – with one minor exception: The application that uses ranges (rather than alternating) would have to perform a naming "shuffle up" if they were ever to add clones. And this is not just a hypothetical scenario.

AOR vs TOR vs QOR vs DOR

You may well be able to tell this from SMF 30 – from the "lightness" of the address space. But it's better to use some of the other instrumentation:

• Certainly there are "footprints in the sand" for things like File Control in SMF 110 so you could detect a File-Owning Region (FOR).
• A CICS region that shows up in DB2 Accounting Trace obviously uses DB2 and looks more like a Data-Owning Region (DOR).
• Likewise for SMF 116 and a Queue-Owning Region (QOR).

Now, regions come in all shapes and sizes and the terms "TOR", "AOR", "FOR" and "DOR" strike me as informal terms – and regions could be playing more than one of these roles so these terms aren't mutually exclusive. But the data is there.

XCF traffic (from SMF 74 Subtype 2) can be interesting:⁸ I noticed one application's CICS regions showed up in the job name field for XCF group DFHIR000, but not for the other application. I was informed there was a VSAM file this application shared – using CICS Function Shipping I guess.

With most topologies there is a unique correlator passed for the life of a transaction through the CICS regions. This correlator (in mangled form) even shows up in DB2. So you can tie together transactions and regions: CICS PA can apparently do this and the next time I get some CICS data in I'm going to learn how to do this. In any case transaction names like "CSMI" (the CICS Mirror transaction) tend to suggest Multi-Region Operation (MRO).

Virtual Storage

I'm reminded of this because in a customer I was able to demonstrate that while both of two applications had Allocated virtual storage of 1500MB the memory backed in one was half that and in the other almost all that. You might deem the former region set moderately loaded and the latter heavily loaded.

The virtual storage numbers – actually both 24-bit and 31-bit – come from Type 30 Interval records. The real storage numbers also from the same records but with some "interpretational help"⁹ from RMF 72-3 records.

But Allocated is a z/OS virtual storage concept: As with DB2 DBM1 address space virtual storage it is generally not the same as used. If it were it'd indicate a subsystem or region in trouble. So we need better information on which to make judgements. Fortunately we have it in the CICS 110 Statistics Trace records: You can do a good job of analysing and managing CICS virtual storage with this (just as you can with IFCID 225 data for DB2).

For one of these two applications virtual storage may well be the thing that determines when the regions need to be split.

Workload Balancing

You can see workload balancing in action at a number of levels:

• At the region level (given a naming convention that lets you identify clones, as above) you can see in Type 30 even CPU numbers, EXCPs etc. If you don't, given supposed clones, you can conclude there isn't some kind of balancing or "round robin" in action – but some other kind of work distribution.
• From CICS SMF 110 (Monitor Trace) you can see transaction volumes and can aggregate by Transaction ID. So an imbalance could be explained – perhaps because the supposed clones run different transactions or some transaction is present in all but at different rates in each clone. Or some other explanation.
• Even without SMF 110 (which a lot of installations don't collect) DB2 Accounting SMF 101 could give you a similar picture (as might MQ's SMF 116).

So the "work distribution and balancing" theme can be addressed readily.

QR TCB vs Others

I mentioned above that virtual storage can sometimes drive the requirement to split CICS regions (whether cloned or not). The Quasi-Reentrant (QR) TCB can be another driver.¹⁰

Traditionally all work in a CICS region ran on the single QR TCB therein. And then File Control was offloaded from it. And the rest, as they say, is history.¹¹

Then as now, if the QR TCB approaches 70% of a processor performance can begin to degrade markedly. For this reason TCB times are documented in the SMF 110 CICs Statistics Trace record. I regularly see CICS regions with more than 70% of an engine (from SMF 30) but to do this an installation needs to understand (using the 110) how much is really the QR TCB.

 

Without the 110's, again you could work with SMF 101 and 116 for DB2 and MQ, respectively. In fact I often do.

---

So, I've tried to give you a flavour of what you can learn about a CICS installation from SMF. i.e. without going near the actual regions themselves. This is indeed just a flavour.

On the "inconsistency" point, consistency isn't vital but good naming conventions have real value. It's an old joke that goes "we like naming conventions so much we have lots of them, some of which contradict each other". 🙂

There are plenty of other examples where there are inconsistencies. A good one is LPAR / z/OS system names. I've seen several customers with the following kind of scenario: "Our systems are called things like A158, SYSC, DSYS, Z001 and MVS1." And it's not just LPAR names and CICS region names, of course.

The inconsistencies in installations often reflect history. And a notable category is Mergers and Acquisitions. (The LPAR names example above is often caused by this.) I'm really impressed at what customers manage to achieve when they do something like this: Getting it to work reliably is the most important thing. Homogenisation of names should be and is secondary.

I really like to see traces of the history in the systems I examine. Some of you reading this have been with me on the journey of your systems' lifetimes for a long time now: I wonder how much history we each remember. 🙂 Next time you see me ask me to pull out some slides from previous engagements: When I do this people are astonished by how much hasn't changed and how much has.

As you possibly spotted that was "Strike Three" for consistency in this post so I guess I'm out. 🙂 This was indeed going to be a post about consistency but took a different direction, as I said. I hope you found the "CICS nosiness" aspect interesting and useful. If you do I might well turn it into a set of slides and add some more material. If you have anything to add I'd be interested in hearing about it – whether you're from Hursley¹² or not.

---

Footnotes

¹ The reference here is, of course, 🙂 to Ralph Waldo Emerson's essay "Self-Reliance" where he wrote "a foolish consistency is the hobgoblin of little minds".

² Such as "kicks", "chicks", "thicks", "six" and "sex" (no, really). 🙂 And my least preferred one is "see eye see ess".

³ If you think I'm self-consciously channelling Einstein here you'd be wrong: It's actually Mao. 🙂 Because the thought experiment is no substitute for experience – according to "On Practice".

⁴ Actually I doubt the utility of SMF 30 Interval records for batch jobs.

⁵ I believe you can get data from IMS relatable to CICS transactions – but I know relatively little about IMS.

⁶ And you can tell a CICS-related 101 record because the value of the QWHCATYP (Connection Type) is QWHCCICS. Further, you can tell things about sign ons from the QWACRINV field value.

⁷ You might not know this but the SMF 72-3 record has the Service Class Description character string – from the WLM policy. I'm slowly evolving my charting to use the description. Time to clean it up, folks. 🙂

⁸ While you get member name in 74-2 (and I'm proud to say I got job name in as a more useful counterpart) you don't get "point to point" information: You just get the messages sent from and to the XCF member. Figuring the actual topology out by matching message rates is fraught. I'd love an algorithm that was effective (or efficient) at this.

⁹ What I mean by this will have to await another post – some time.

¹⁰ 26 years ago I worked on CICS Virtual Storage at a Banking customer. Not a lot has changed. 🙂 20 years ago I was involved in enabling customers to take advantage of multiple processors by splitting regions as described in this section. Again, not a lot has changed. 🙂 But this is unfair because the Virtual Storage and CPU pictures have changed a lot.

¹¹ Or is it hysteria? 🙂

¹² Home of CICS and Websphere MQ Development

(Originally posted 2012-04-12.)

I was pleased when Julian Bridges (who I worked with in IBM Global Services for a number of years) told me he had access to a z/OS Release 13 system. He agreed to write a blog post on the enhancements to the ISPF editor in Release 13 and this is that blog post. Enjoy!

Julian Bridges

It comes as a surprise to many how flexible the ISPF editor can be. Many times sitting with clients typing away with them at your shoulder you hear, “I didn’t know you could do that”. It’s certainly worth hitting F1 in the edit screen or reading “ISPF Edit and Edit Macros” and spending a while trying to understand the power of the commands available.

Whilst much of the power is in the primary commands, in the past few releases of z/OS functionality has been added to the line commands as well.

First is simply the ability to (C)opy or (M)ove data to multiple lines. Previously you could copy or move lines to a single destination but since z/OS 1.10 this has been extended to allow multiple destinations.

For example, I’ve missed a comma from the end of the SYSUT2 and then repeated the line and hence the mistake. I can now use the move overlay line command to add a comma in to each of the lines with the error as follows:

m 0100                                                   , 
000700 //PACK     EXEC PGM=AMATERSE,PARM='PACK'           
000800 //SYSPRINT DD   SYSOUT=*                           
000900 //SYSUT1   DD   DISP=SHR,DSN=JULIAN.TZOSC01.DUMP   
ok 100 //SYSUT2   DD DISP=(,CATLG),DSN=JULIAN.TZOSC01.TRS 
001200 //         SPACE=(CYL,(1000,1000),RLSE),VOL=(,,,3) 
001300 //*                                                 
001400 //PACK     EXEC PGM=AMATERSE,PARM='PACK'           
001500 //SYSPRINT DD   SYSOUT=*                           
001600 //SYSUT1   DD   DISP=SHR,DSN=JULIAN.TZOSC02.DUMP   
ok 700 //SYSUT2   DD DISP=(,CATLG),DSN=JULIAN.TZOSC02.TRS 
001800 //         SPACE=(CYL,(1000,1000),RLSE),VOL=(,,,3) 
001900 //*                                                 
002000 //PACK     EXEC PGM=AMATERSE,PARM='PACK'           
002100 //SYSPRINT DD   SYSOUT=*                           
002200 //SYSUT1   DD   DISP=SHR,DSN=JULIAN.TZOSC03.DUMP   
ok 300 //SYSUT2   DD DISP=(,CATLG),DSN=JULIAN.TZOSC03.TRS 
002400 //         SPACE=(CYL,(1000,1000),RLSE),VOL=(,,,3) 
002500 //*                                                 
002600 //PACK     EXEC PGM=AMATERSE,PARM='PACK'           
002700 //SYSPRINT DD   SYSOUT=*                           
002800 //SYSUT1   DD   DISP=SHR,DSN=JULIAN.TZOSC04.DUMP   
o 2900 //SYSUT2   DD DISP=(,CATLG),DSN=JULIAN.TZOSC04.TRS 
003000 //         SPACE=(CYL,(1000,1000),RLSE),VOL=(,,,3) 
003100 //*                                                 

Note the addition of the “k” on the overlay command to indicate the multiple destinations. The last destination in the file is indicated by missing this “k” and is just the normal overlay “o”. The same is true for “a” after and “b” before destinations as well.

Of course, in this case, it would probably be easier just to type the comma in the correct place but you get the idea.

Secondly, with z/OS 1.13, the ability to write you own line command macros has been made available.

This does involve a few steps but basically now the ability to do pretty much anything you wish is available:

1. Define an ISPF table to associate a line command with a macro.
2. Write your macro.
3. Associate the defined table with your edit session.
4. Run the macro.

Define An ISPF Table To Associate A Line Command With A Macro

Fortunately the ISPF table utility, option 3.16, has been enhanced to make this straightforward. An option at the bottom of the screen now asks if this “Table is an EDIT line command table”

When selected it creates the table in the necessary format and you just have to fill in the blanks. The examples below show what the options mean for existing line commands.

• User command – The line command.
• MACRO – The macro which will run when you run this line command.
• Program Macro – Is this a program macro.
• Block format – Does this macro allow you to select multiple lines by repeating the last char of the command e.g. CC? CC would copy a block of text.
• Multi line – Does this macro allow you to select multiple lines by providing a numeric suffix on the end of the command e.g. C6 will copy the next 6 lines.
• Dest Used – Does this macro allow a destination e.g. C or M must have a destination whereas R doesn’t.

e.g.

User     MACRO    Program  Block    Multi    Dest     
Command           Macro    format   line     Used     
----+--- ----+--- ----+--- ----+--- ----+--- ----+--- 
CL       CLINE    N        Y        Y        Y       

This table must then be saved to a table library allocated to your ISPTLIB concatenation.

Write Your Macro

A few things to bear in mind. You have to use the PROCESS macro instruction to populate the range and destination variables within the macro. This is best illustrated by an example.

/* REXX */                     
Address ISREDIT                 
"macro NOPROCESS"               
"process range CL"             
dw = 72                         
"(srange) = LINENUM .zfrange"   
"(erange) = LINENUM .zlrange"   
do i = srange to erange         
  "(LINE) = LINE " i           
  line = centre(strip(line),dw) 
  "LINE " i " = (LINE)"         
end                             

This macro will centre the lines selected.

Process takes the arguments range, dest or both and the line command being entered. It gives return codes if when called a range or dest is missing.

This macro should then be saved in your SYSEXEC or SYSPROC concatenation.

Associate The Defined Table With Your Edit Session

Select ISPF option 2 and enter the name of the table in the “Line Command Table” field at the bottom of the screen.

This is now remembered whether you edit via option 2 or using “E” from 3.4.

Run The Macro

Single line

****** ***************************** Top of Data ****************************** 
cl 100 I wandered lonely as a cloud                                             
000200 That floats on high o'er vales and hills,                               
000300 When all at once I saw a crowd,                                         
000400 A host, of golden daffodils;                                             
000500 Beside the lake, beneath the trees,                                     
000600 Fluttering and dancing in the breeze.                                   
****** **************************** Bottom of Data **************************** 

Results in

****** ***************************** Top of Data ****************************** 
000100                       I wandered lonely as a cloud                       
000200 That floats on high o'er vales and hills,                               
000300 When all at once I saw a crowd,                                         
000400 A host, of golden daffodils;                                             
000500 Beside the lake, beneath the trees,                                     
000600 Fluttering and dancing in the breeze.                                   
****** **************************** Bottom of Data **************************** 

Block format

****** ***************************** Top of Data ****************************** 
cll 00 I wandered lonely as a cloud                                             
000200 That floats on high o'er vales and hills,                               
000300 When all at once I saw a crowd,                                         
000400 A host, of golden daffodils;                                             
cll 00 Beside the lake, beneath the trees,                                     
000600 Fluttering and dancing in the breeze.                                   
****** **************************** Bottom of Data **************************** 

Results in

****** ***************************** Top of Data ****************************** 
000100                       I wandered lonely as a cloud                       
000200                That floats on high o'er vales and hills,                 
000300                     When all at once I saw a crowd,                     
000400                       A host, of golden daffodils;                       
000500                   Beside the lake, beneath the trees,                   
000600 Fluttering and dancing in the breeze.                                   
****** **************************** Bottom of Data **************************** 

Multi line

****** ***************************** Top of Data ****************************** 
000100 I wandered lonely as a cloud                                             
000200 That floats on high o'er vales and hills,                               
cl99 0 When all at once I saw a crowd,                                         
000400 A host, of golden daffodils;                                             
000500 Beside the lake, beneath the trees,                                     
000600 Fluttering and dancing in the breeze.                                   
****** **************************** Bottom of Data **************************** 

Results in

****** ***************************** Top of Data ****************************** 
000100 I wandered lonely as a cloud                                             
000200 That floats on high o'er vales and hills,                               
000300                     When all at once I saw a crowd,                     
000400                       A host, of golden daffodils;                       
000500                   Beside the lake, beneath the trees,                   
000600                  Fluttering and dancing in the breeze.                   
****** **************************** Bottom of Data **************************** 

Have a play and see how you get on.

(Originally posted 2012-03-25.)

As previously discussed I’m often in a situation of trying to make sense of a set of job-related SMF data. Even though it may be your own installation’s data, you’re probably confronted with what I like to call “a journey of discovery” occasionally, too.

I’m always looking for what I can discern from the data.¹ And, when confronted with a set of data about batch jobs, I go into overdrive. 🙂

This post is about how to tell if a set of batch jobs really are clones of each other. It’s an exercise in pattern definition, albeit loosely.

But first, why would you want to know what’s a clone set of jobs? Remember these are near-identical jobs that run in parallel against subsets of the data. Firstly, if something’s cloned you might be able to clone it further.² Second, if it isn’t cloned you need to recognise that and think about the effort involved to even start with cloning.³

The process of detecting clones is easy to describe but not so easy to do. Here are the steps:

1. Look for similarities in SMF 30 Step- and Job-End records.
2. Likewise in SMF 101 DB2 Accounting Trace.
3. And similarly for data access.

Steps 2 and 3 could be done in either order. And indeed Step 2 would be only be relevant for DB2.

Let’s think about these in a bit more detail…

Step-End And Job-End Evidence

I would expect cloned jobs to run more-or-less alongside each other – though they might be set off in groups. Of course imbalance between the clones would mean they wouldn’t end at the same time.

Additionally the jobs would have the same “step profile”. By this I mean the number of steps is consistent, the same steps in each job are the big ones. The program names are the same. And the performance profile of each step is similar across the clones, so the CPU intensiveness and the EXCP counts are similar.

I would expect also to see a sensible job-naming convention. For example “all the jobs beginning PLCD50 are clones and the suffix is 00, 01, 02 and so on”. From this you get job names like PLCD5000, PLCD5001 etc.

Generally I spot groups of jobs meeting these criteria pretty easily – using SMF Type 30 subtypes 4 (Step) and 5 (Job).

DB2 Invocation Evidence

For DB2 jobs I’d expect corroboration from DB2 Accounting Trace (SMF Type 101):

• Plan names and package names⁴ should be the same.

  In many cases I’ve seen a single DB2 plan name for an entire application, and sometimes crossing application boundaries. Similarly packages are sometimes widely used – for example in the “I/O module” or Stored Procedure cases. Taken together this is a necessary but not sufficient condition.

• DB2 Accounting Trace, as you probably know, can give a very detailed breakdown of where a step’s time goes⁵ – down to the package level. Again, you’d expect to see a similar profile across all the clones.

For any serious DB2 Batch analysis I’d be looking at this data anyway. I’ve written extensively about DB2 Batch, most recently here.

Data Access Evidence

This is where consistency is slightly less to be expected: Most probably DD names will be the same across the cloned jobs. But very often the data set names are slightly different. For example the clone stream number might be encoded in the data set – probably in one of the lower level qualifiers.

For DB2 it’s more difficult to assess which tables a job step access – and probably you need to look at the DB2 Catalog for insight. When you do you may well find the cloned jobs accessing partitions of the same table (in some cases).

There is other evidence of interest here:

In many cases clone jobs (or streams) are preceded by a job whose role is to split the data to feed the clones. Similarly there’s often a follow-on job to merge the results. Detecting these – in the non-DB2 case is usually pretty straightforward. (Even in the DB2 case the scheduler should tell you.) My point here is there’s value in seeing how cloning is working, not least from why there might be imbalance between the clones.

As I said at the outset it’s useful to figure out which jobs in a suite or a window are part of a cloning implementation. And as I hinted in a couple of places there’s also value in understanding balance (or imbalance). In this post I’ve given some tips on the kinds of patterns to look for. Some of this could be codified, I’m sure. In any case the human mind is a wonderful instrument for pattern recognition⁶.

---

¹ I’ve talked about this sort of thing before. Most recently in Published on Slideshare: I Know What You Did Last Summer.

² Recall my recommendation to clone 2, 4, 8, 16 … or else 3, 6, 12, 24… – unless you know differently.

³ See this part and this part especially of the ‘I Said "Parallelise" Not "Paralyse"’ series of blog posts for more on this.

⁴ You only get package-level statistics if you specify Accounting Trace classes 7 and 8.

⁵ You only get the detailed break down if you specify Accounting Trace classes 1, 2 and 3. (And see ⁴.)

⁶ This footnote is a wholly gratuitous reference to the excellent Pattern Recognition, a novel by the excellent William Gibson. 🙂