(Originally posted 2009-11-09.)

There’s a pattern I’ve seen over a number of test Parallel Sysplex environments over the past few years, a couple of them in situations this year:

It’s not much use drawing performance inferences from test environments if they’re not set up properly for performance tests.

Sounds obvious, doesn’t it?

There are two problem areas I want to draw your attention to:

1. Shared Coupling Facility Images
   
   If you run a performance test in an environment with shared coupling facility images you stand to get horrendous request response times and the vast majority of requests going async (given a chance). I’ve even seen environments where XCF refuses to use coupling facility structures and routes ALL the traffic over CTCs. (And I’ve seen a couple of environments where there are no CTCs to route it over and XCF traffic is then reduced to a crawl.)
2. "Short Engine" z/OS Coupled Images
   
   In a recent customer situation I saw the effect of this: The customer was testing DB2 Loads where actually it was a bunch of SQL inserts. They were also duplexing the LOCK1 structure for the data sharing group. The Coupling Facility setup was perfect, but still response times became really bad once duplexing was established for the LOCK1 structure. Two salient facts: Because of duplexing all the LOCK1 requests were async. XCF list structure request response times were always awful.
   
   The answer to why this problem occurred lies in understanding how async requests are handled: The coupled z/OS CPU doesn’t spin in the async case. In the "low LPAR weight relative to logical engines online" case the z/OS LPAR’s logical engines were but rarely dispatch on physical engines. This meant there was a substantial delay in z/OS detecting the completion of an async request. Hence the elongated async response times. As I said, the LOCK1 structure went async once it was duplexed.
   
   As it happens the physical machine wasn’t all that busy: Allowing the LPAR to exceed share – using a soaker job – ensured logical engines remained dispatched on physical engines longer. And, perhaps paradoxically, the async request response times went right down. This, I hope, reassured the customer that in Production (with "longer-engine" coupled z/OS LPARs) async coupling facility response times ought to be OK.

Now, this is just Test. But it could unnecessarily freak people out. But, hopefully, it’s easy to see why Test Parallel Sysplex environments might perform much worse tan Production ones.

(I’m guessing you’re going "duh, I knew Test would be worse than Prod". 🙂 But these two cases are specifics of why Test might be even worse compared to Prod than expected.)

Anyhow, I thought they were interesting. And I have seen 1. quite a few times now. 2. not so much, in fact only once so far.

(Originally posted 2009-11-08.)

I’ve just submitted a set of slides to Slideshare. They’re not mine, they’re not new, they’re not even in a modern format. But they are a very good presentation worth preserving…

In 1993 Curt Cotner presented a set of slides on the new DDF Inactive Thread support in Version 3 of DB2. It’s still highly relevant and this support was the base on which the Version 4 WLM classification line item was built.

You can find the slides here.

I’d also recommend you went on to read John Arwe’s paper on Preemptible-Class SRBs.

(Originally posted 2009-11-08.)

Apologies for being away. For a while I couldn’t get the blog to operate – as an author. And then I gave up trying. 😦

Count this as a "testing 1-2-3" post but also as notice I intend to back with more (hopefully useful) content soon. MUCH has happened in the "intermission". 🙂

(Originally posted 2009-05-07.)

I’m reporting what I’m learning (or I think is significant) in conference sessions on Twitter. My Id is “MartinPacker” and I’m using the Hashtag “#zOS09” to tag my posts. Feel free to follow along. In principle you don’t even need to sign up to Twitter to do this.

It seems more immediate than posting here.

Oh, and feel free to comment on Twitter using the same tag.

(Originally posted 2009-04-26.)

Here’s another “For Beginners” post to encourage people to just leap in and try it…

Javascript is a language popularly used for such things as web pages with some programming in them (and that includes frameworks like dojo), building Firefox extensions, and the Adobe AIR (desktop) runtime. As it happens I’m pretty familiar with Javascript anyway – having done all 3 of the above. This post shows how I got it to run on z/OS using Mozilla’s Rhino…

Mozilla’s Rhino project is a javascript interpreter written in java. (Java 6 does in fact have a general-purpose scripting interface – with javascript as a prime target.) It’s important to note that Rhino requires Java 5.

So here’s what I did…

1. Downloaded the Rhino package to my PC from here.
2. Unpacked it on my PC using a Zip tool (in my case 7-Zip).
3. Tested it out on my PC – just to get comfortable with how it worked (and that was well worth the half an hour it took).
4. FTP’ed binary the included js.jar file to an HFS file (in my case /u//rhino/js.jar). The file was under 1MB in size.
5. Adjusted my CLASSPATH to point additionally to js.jar (including js.jar explicitly, not just the directory it was in).
6. Invoked Rhino in interactive mode:

   java org.mozilla.javascript.tools.shell.Main

7. Typed in a few javascript statements such as

   print(1+2)

8. Quit by typing “quit()”

You can pass the name of a javascript file to Rhino by adding eg “test.js” to the command to invoke it.

In my case I also used the “.profile” startup script to adjust my CLASSPATH and to alias “js” to mean “run the Rhino javascript interpreter”.

So, it’s actually VERY straightforward to run Javascript under z/OS – thanks to Rhino. (And I suspect anyone with Java 6 installed would find it even easier.)

As always, if you know better please feel free to comment here. Remember, I’m learning as I go, and I just want to encourage others to try a few things.

(Originally posted 2009-04-25.)

In this post (and any others in a similar vein) I'm going to be displaying a great deal of ignorance – but I think I'm doing it in a good cause.

I know a fair amount about things like Java, XML and C++ – but NOT on z/OS. So I'm determined to learn and present what I learn as "For Beginners" posts here.

The idea is that I'll encourage other people with more traditional z/OS skills to try some simple new things, such as java. So, if you've not done these things before do have a go. Stuff is remarkably straightforward to do.

Here's how I created a simple "Hello World" java application in Unix System Services.

1. Log onto TSO with a region of 64MB (64000 in the logon panel). When I tried this with 32000 I got a "JVMDBG001" message, complaining of not being able to GETMAIN enough memory to start the JVM (but the JVM started anyway).
2. Type OMVS. (If your userid has been set up to use Unix System Services it should start you in a directory (mine being "/u/userid").
3. Create a subdirectory using "mkdir javatest".
4. Change to this subdirectory using "cd javatest".
5. List what's in this subdirectory (actually nothing) using "ls".
6. Create a new java source file using "oedit Hello.java".
7. While in the (ISPF) Editor add the following lines:

   
   public class Hello{
     public static void main(String args[]){
       System.out.println("Welcome!");
     }
   }
   
   

8. Press PF3 to save. (Treat the resulting dialog as any "leaving ISPF" dialog.)
9. Compile the resulting java with "javac Hello.java".
10. Run the compiled java bytecode with "java Hello".

And that's all there is to it. One note – if you're not familiar with java: Case is significant. Mismatches would cause problems.

The java book I used to get started (and then some) was Deitel & Deitel's "Java How To Program" but there are lots of them to choose from.

If you're more experienced at this than me please feel free to comment as you see fit. But remember this is a "For Beginners" post. I'm trying to encourage people to GET STARTED.

(Originally posted 2009-04-25.)

Back in z/OS R.9 RMF Parallel Sysplex New Fields (in 2007) I mentioned a new field: R742MJOB (XCF Member Job Name.)

At the time I had no real customer data so I could only espouse the HOPE that this field would be useful. (When I asked for it to be added to the SMF 74 Subtype 2 record it seemed to me it probably would be.)

Now that z/OS R.9 is “mainstream” I’m seeing lots of data at this level. And so, because it’s mainstream, I think it’s time to talk some more about this field (and to tell you what I’m seeing in customer data). I think you’ll like it.

But first some preliminaries:

XCF Groups and Members

Before there was Parallel Sysplex there was Sysplex. And the main ingredient of (base) Sysplex was XCF signalling. Applications that use XCF consist of groups and members. A group is essentially an application. So the “SYSGRS” group is the GRS application. Members are address spaces participating in the application.

XCF’s job is to pass messages within members of the same group, whether on one system or several within the Sysplex.

Instrumentation

Since the advent of XCF we’ve had group names, member names and messages sent and received in the 74-2 record. But there’s a problem here, best illustrated by the example of DB2 Data Sharing:

A DB2 Data Sharing group comprises, amongst other Coupling Facility structures, a “LOCK1” structure. It’s a Lock structure and it’s called the “LOCK1” structure because its name is always “_LOCK1”. We can easily identify this structure and the group it belongs to. Associated with the LOCK1 structure are 2 XCF groups:

• IXCLOnnn – which is a particularly unhelpful name as ALL Lock structures have such an XCF group associated with them. (There usually being several other Lock structures in a single Parallel Sysplex – such as GRS Star and VSAM RLS’s Lock structure.) The member name, by the way, would be something equally unhelpful like “M102”.

  The purpose of this group is to resolve potential “False Contention” situations in the Lock1 structure (possibly caused by too small a lock table).

• DXRabcd – where the “abcd” is the Data Sharing Group Name. This is slightly better but it doesn’t help that the member name is e.g. “DXRDPG0$$IPB1003”.

  This XCF group is used by the various IRLM address spaces in the group to resolve locking conflicts from DB2’s perspective. (IXCLOnnn group traffic resolves to either a False Contention or an XES contention, the latter having to be resolved to either a lock being granted or IRLM negotiation causing the requester to wait.)

So, it would be really handy to isolate these two types of traffic for a particular DB2 Data Sharing group (and perhaps to work on reducing it). As you can see, it’s rather hard to do that without more useful information. This is where 74-2 XCF Jobname comes in…

It turns out that for BOTH the IXCLOnnn group and the DXRabcd group the job name is the IRLM address space name. So, if you know the IRLM job name (hopefully because you’ve given it a name close to that of the Data Sharing group it supports) you can easily tell which XCF groups relate to that Data Sharing group. And then perhaps you can do something about the traffic.

Perhaps I’m displaying my ignorance of DB2 here but in the test case I’m looking at right now I see 2 IRLM address spaces in one system and one in the other, all sharing the same XCF groups – both IXCLOnnn and DXRabcd. I didn’t know one could run 2 IRLMs in the same DB2 Data Sharing group in the same LPAR. I’ll admit I’m not really sure what to make of that.

But what about other XCF groups? It turns out that they also have good mnemonic address space names. But then for most other groups it’s obvious what the XCF group is anyway.

Glancing over a friend’s shoulder at a modern RMF XCF Activity Report I didn’t see the job name in the report (but rather the member name). But then I, and I expect most other people, don’t often look at RMF Postprocessor reports all that often.

(Originally posted 2009-04-10.)

To whoever (In the USA I think) Googled “what is REMOVECC” and got to this blog let me give you my thoughts on the matter. (And yes I know, it being a search, you might never come back to read this.)

To understand REMOVECC you have to understand that the “CC” refers to ASA (American Standards Association) Carriage Control, described here.

ASA (formerly ANSI) control characters appear in Position 5 for variable-length records (just after the Record Descriptor Word (RDW)). So a EBCDIC “1” in Position 5 would tell a printer to start a new page, and a “0” would tell it to skip a line.

Originally OUTFIL was used to create PRINTABLE reports. So these kinds of control characters were needed. But more modern (perhaps more complex) uses often don’t require ASA CC characters. So coding REMOVECC on the OUTFIL statement suppresses the control characters.

In this blog entry I used OUTFIL REMOVECC because my output is HTML. No HTML reader (such as a web browser) is going to tolerate ASA control characters. So I removed them.

I hope that clears this up. And I know the question was asked because my Firefox Extension has a “Referer URL” reporting element that tells me things like the Google searches that got people to my blog. I’d be flattered to think it was someone who was trying to understand the referenced blog post.

Update 10 April 2009

Perhaps I should also mention the relatively recent BLKCCH1, BLKCCH2 and BLKCCT1 OUTFIL options:

These replace the page eject carriage control character “1” with a blank in HEADER1, HEADER2 and TRAILER1 respectively. (HEADER3, TRAILER2 and TRAILER3 don’t need equivalent options.) Replacing with a blank still shifts the output to the right by 1 character, so REMOVECC may still be the best choice.

The PTFs for this enhancement appeared in April, 2006 (UK90006 / UK90007).

(Originally posted 2009-04-10.)

For whoever got to my blog with this Google search here’s how you sort data without a header:

First you need to be using z/OS DFSORT Release 10 or to have applied the PTFs for UK90013 (July 2008).

Second, the vehicle for doing this is ICETOOL rather than DFSORT itself.

Use the new DATASORT operator. Here’s an example:

If you code

DATASORT FROM(DD1) TO(DD) FIRST(3) USING(CTL1)

and code a CTL1CNTL like

//CTL1CNTL DD *  SORT FIELDS=(1,8,CH,A,9,4,BI,D)

ICETOOL will sort everything after the first 3 records, leaving the first 3 records unchanged. (In this case the sorted lines are, obviously, collated on two distinct fields.)

In the CTL1CNTL set of control statements you can code an OUTFIL statement (or several) if you want to. There are some restrictions on what statements you can code in the CTL1CNTL data set (and these are documented here.)

As well as being able to avoid sorting header records (FIRST,FIRST(n),HEADER,HEADER(n)) you can also avoid sorting the last few records with LAST, LAST(n), TRAILER or TRAILER(n).

(Originally posted 2009-04-09.)

So here’s yet another way of parsing the WLM XML Service Definition. This time it’s on z/OS, using DFSORT. Relatively recent features in DFSORT have made it easier to do useful things with XML.

In this example I’ve made some attempt to make the output pretty – by creating a HTML table.

Below is the SYSIN you need to parse the <SchedulingEnvironment> elements. I’m assuming you know how to code a basic DFSORT invocation.

(To be able to run this example you need the PTFs for APAR UK90013 – WHEN=GROUP – and for UK90006/UK90007 – JFY and PARSE.)

  OPTION COPY,VLSHRT                                     INREC IFTHEN=(WHEN=INIT,BUILD=(1,4,3X,5)),             IFTHEN=(WHEN=GROUP,                                      BEGIN=(8,80,SS,EQ,C'<SchedulingEnvironment>'),         END=(8,80,SS,EQ,C'</SchedulingEnvironment>'),          PUSH=(5:ID=3))                                       OUTFIL INCLUDE=(5,1,CH,EQ,C'0'),REMOVECC,              HEADER1=('<html>',/,                                   '<body>',/,                                              '<h1>Scheduling Environments</h1>',/,                  '<table border="1">',/,                                '<tr>',/,                                              '<th>Name</th>',/,                                     '<th>Description</th>',/,                              '<th>Resources Required</th>',/,                       '</tr>'),                                              IFTHEN=(WHEN=(8,80,SS,EQ,C'<SchedulingEnvironment>'),      BUILD=(1,4,C'<tr>')),                                             IFTHEN=(WHEN=(14,6,CH,EQ,C'<Name>'),                                  PARSE=(%00=(STARTAFT=C'<Name>',ENDBEFR=C'</Name>',                  FIXLEN=64)),                                                        BUILD=(1,4,%00,JFY=(SHIFT=LEFT,LEAD=C'<td>',TRAIL=C'</td>'))),    IFTHEN=(WHEN=(8,80,SS,EQ,C'<Description>'),                           PARSE=(%01=(STARTAFT=C'<Description>',ENDBEFR=C'</Description>',    FIXLEN=64)),                                                        BUILD=(1,4,%01,JFY=(SHIFT=LEFT,LEAD=C'<td>',TRAIL=C'</td>'))),    IFTHEN=(WHEN=(8,80,SS,EQ,C'<ResourceNames>'),                         BUILD=(1,4,C'<td><ul>')),                                         IFTHEN=(WHEN=(8,80,SS,EQ,C'<ResourceName>'),                          BUILD=(1,4,C'<li>')),                                             IFTHEN=(WHEN=(18,6,CH,EQ,C'<Name>'),                                  PARSE=(%02=(STARTAFT=C'<Name>',ENDBEFR=C'</Name>',                  FIXLEN=64)),                                                        BUILD=(1,4,%02)),                                                 IFTHEN=(WHEN=(8,80,SS,EQ,C'<RequiredState>'),                         PARSE=(%03=(STARTAFT=C'<RequiredState>',                            ENDBEFR=C'</RequiredState>',                                        FIXLEN=64)),                                                        BUILD=(1,4,%03)),                                       IFTHEN=(WHEN=(8,80,SS,EQ,C'</ResourceName>'),               BUILD=(1,4,C'</li>')),                                  IFTHEN=(WHEN=(8,80,SS,EQ,C'</ResourceNames>'),              BUILD=(1,4,C'</ul></td>')),                             IFTHEN=(WHEN=(8,80,SS,EQ,C'</SchedulingEnvironment>'),      BUILD=(1,4,C'</tr>')),                                  TRAILER1=('</table>',/,'</body>',/,'</html>')                       

The best way to read this is as a sequence of two steps:

1. INREC
2. OUTFIL

There’s also a COPY as we’re not sorting, just preserving record order.

INREC

This statement uses WHEN=GROUP to label all records not between the <SchedulingEnvironment> and </SchedulingEnvironment> lines in the XML stream. The label is a sequence number:”PUSH=5:ID=3″ says “overlay position 5 (just after the RDW) with 3 bytes of sequence number, the sequence number incrementing for each group of records. (The “BUILD=1,4,3X,5” beforehand says “insert 3 blanks between the 4-byte RDW and the first byte of actual data”.)

So any line in the input file not in a <SchedulingEnvironment> element have no such tag. For the ones that have a tag we have sequence numbers “000”,”001″ etc. As we’ll see this is wide enough for 99 distinct scheduling environments – and can be adapted if you have more.

OUTFIL

This statement throws away records that aren’t part of a group (and so aren’t within a <SchedulingEnvironment> element), reformats the contents of each <SchedulingEnvironment> element and wraps header and trailer HTML around the result.

“INCLUDE=(5,1,CH,EQ,C’0′)” says “keep records that were tagged as part of a <SchedulingEnvironment> element in the INREC statement”.

“REMOVECC” prevents OUTFIL from writing ANSI carriage-control characters.

“HEADER1” and “TRAILER1” write “boilerplate” HTML.

The remainder – the IFTHEN “pipeline” – does the bulk of the work…

There are several IFTHEN “stages”. Note: If “HIT=NEXT” had been coded on any of them records matching the “WHEN=” clauses would be passed onto the next IFTHEN “stage”, In that sense it would be a true pipeline. In this case I didn’t code “HIT=NEXT” – as each matching “WHEN=” is the only one I want to operate on the matched record.

There are several distinct types of IFTHEN “stage”:

• Ones like

  IFTHEN=(WHEN=(8,80,SS,EQ,C'<SchedulingEnvironment>'),      BUILD=(1,4,C'<tr>')), 

  which is a substring search, with a literal replacement string. “Substring search” means “find the match ANYWHERE in the character range”.

• Ones like

  IFTHEN=(WHEN=(14,6,CH,EQ,C'<Name>'),                                  PARSE=(%00=(STARTAFT=C'<Name>',ENDBEFR=C'</Name>',                  FIXLEN=64)),                                                        BUILD=(1,4,%00,JFY=(SHIFT=LEFT,LEAD=C'<td>',TRAIL=C'</td>'))),  

  This one has a simple search – for a specific string in a particular position. But it also has a free-format parsing:

  PARSE fills a variable (%00) with whatever is between “<Name>” and “</Name>”, now matter how wide or how narrow (subject to a limit of 64 characters – the FIXLEN value).This variable is used in the BUILD specification.

  BUILD uses the new JFY (APAR UK00013) function to sandwich the contents of %00 between “<td>” and “</td>”, removing any trailing spaces.

• Ones like

    IFTHEN=(WHEN=(8,80,SS,EQ,C'<RequiredState>'),                         PARSE=(%03=(STARTAFT=C'<RequiredState>',                            ENDBEFR=C'</RequiredState>',                                        FIXLEN=64)),                                                        BUILD=(1,4,%03)),                                     

  which are a hybrid of the two.

Input Data

Disregarding the lines in the XML file that will ultimately be thrown away we have

    <SchedulingEnvironment>      <Name>DB2P</Name>      <Description>Run for DB2P</Description>      <ResourceNames>        <ResourceName>          <Name>DB2P</Name>          <RequiredState>On</RequiredState>        </ResourceName>      </ResourceNames>    </SchedulingEnvironment>

You’ll notice two different uses of the <Name> element. To distinguish between those the IFTHEN “stages” for the two use fixed offsets: e.g “WHEN=(18,6,CH,EQ,C'<Name>’)”.

Output HTML

For that one <SchedulingEnvironment> the code produces:

<html><body><h1>Scheduling Environments</h1><table border="1"><tr><th>Name</th><th>Description</th><th>Resources Required</th></tr><tr><td>DB2P</td><td>Run for DB2P</td>

<td><ul><li>DB2POn</li></ul></td></tr></table></body></html>

which ultimately formats as

Name	Description	Resources Required
DB2P	Run for DB2P	DB2POn

though I’ve taken out the <html>, </html>, <body>, </body>, <h1> and </h1> tags – to make this blog entry format a little better.

So, admittedly this looks complex but if you pick through it should all make sense. If you’re lucky enough to have the PTFs on (and the latest is from July 2008) or are on z/OS Release 10 you can try this out and adapt the techniques for other parts of the WLM XML Service Definition or indeed other pieces of XML.

One other thing: In this example the <ResourceName> elements relate to a different part of the Service Definition, which looks like:

 <Resources>    <Resource>      <Name>DB2P</Name>      <Description>Subsystem DB2P</Description>    </Resource> </Resources>

which I haven’t attempted to look up in the above code sample. To do it I think I’d be tempted to create 2 flat files – one from <SchedulingEnvironment> elements and one from <ResourceName> elements – and then use ICETOOL SPLICE to fold them in. And then emit HTML by reformatting the result.

But then this blog entry is quite long enough as it is. 🙂