(Originally posted 2011-11-20.)

The very first computer game I ever played was called Rhino and it ran on a Commodore PET. The school had been lent one for a fortnight. (I don’t know why as we didn’t go on to buy any, instead getting a single RML 380Z.) Imagine a character-grid screen where the rhino’s are represented by pi symbols that chase you as you try to move from A to B. It was written in BASIC and allowed from 1 to 10 rhino’s. (We modified it so it would allow up to 100: It played much slower but the overwhelming number of pi signs was good for a 5-minute giggle.)

Having got that off my chest normal service is resumed: 🙂

Today I want to talk to you about a different Rhino. It’s a javascript interpreter written in java. And it’s documented here.

Some of you will know that I don’t think java is a particularly modern language. If you’ve kept your ear to the ground you’ll know there are MANY that have a better claim to the mantle of "coolest language (today)". 🙂 One of the nice things about it, though, is that it is zAAP-eligible. And a couple of nice things about javascript is that a lot of people know it, and it’s got some really neat features.

Here’s a sample that shows one of its capabilities – processing XML:

xml=new XML(stdinText)
print(xml.item.(@type=="carrot").@quantity)

These two lines parse a text string (in the variable "stdinText") as XML and print the value of the quantity attribute of each XML node that has "carrot" as the value of its type attribute. If you know XML I think you’ll see the power of that.

For completeness here is the XML and the whole code to read the XML string from stdin and do the XML query:

First the XML:

<sales vendor="John">
  <item price="4" quantity="6" type="peas"/>
  <item price="3" quantity="10" type="carrot"/>
  <item price="5" quantity="3" type="chips"/>
</sales>

and now the javascript:

importPackage(java.io)
importPackage(java.lang) 
stdinText=""
stdin=new BufferedReader(new InputStreamReader(System["in"])) 
while(stdin.ready()){
  stdinText+=stdin.readLine()
}
xml=new XML(stdinText)
print(xml.item.(@type=="carrot").@quantity)

In this version of the code two java packages are imported that are used to get the XML from stdin. I chose to do this so I could indirect from a file – and also from BPXWUNIX (the REXX interface to z/OS Unix I’ve mentioned several times recently here).

The "new XML()" phrasing is part of the javascript’s E4X capabilities – that allow it to process XML so neatly. (Another part is directly assigning literal XML (not a string) to a javascript variable. See ECMAScript for XML for a description of E4X.

One of the nice things about javascript is the proliferation of frameworks for it. My favourite is Dojo though I’ve also used jQuery. Although Dojo (and jQuery) have a lot to offer for web applications much of that isn’t applicable to z/OS Unix System Services. But some very useful things remain. For example some language extensions like "dojo.forEach()" iteration.

So I set out to try two things:

1. Installing and using Rhino. You’ve seen an example of this above – processing XML.
2. Installing and using the Base component of Dojo.

The latter proved interesting: I made a bit of a mess of it by copying the Dojo Base files up to z/OS one at a time. I should’ve done it wholesale. But I got there in the end. And here’s a test script:

load("dojo.js")
dojo.forEach([1,2,3],function(x) {print(x)})
print(dojo.isRhino)

which does the following:

1. It pulls in the Dojo code.
2. It creates a temporary array – [1,2,3] and iterates over it – printing 1 then 2 then 3.
3. It prints "true" because Dojo is indeed running under Rhino.

So, it was quite easy to get the Rhino javascript interpreter up and running – under java – on z/OS. It would’ve been easy to install Dojo Base if I hadn’t made a mess of it. I think if you have people used to javascript they could be productive quite quickly – so long as you’re comfortable with a java-based environment. As previously mentioned, you could readily call javascript code this way from REXX with BPXWUNIX. Indeed you could have REXX pass the actual javascript code in.

On the performance side you need to be aware that Rhino converts the javascript code to java classes on the fly. It’ll probably perform well after the initial conversion – so stuff with lots of iteration. And I would expect it to offload to a zAAP (or zIIP via zAAP-on-zIIP) quite readily. (I’d love to see measurements if anyone tries it.)

As well as Rhino, Mozilla make a C-based javascript interpreter (SpiderMonkey) but I think that would be a much more difficult thing to get running on z/OS. 

And if you want an exemplar of how far games have come from the original Rhino try Uncharted 3. I’ve just completed in and it’s outstanding. (So outstanding I want to go back and play the whole thing again, as well as dipping my toe in the multiplayer water.)

| Corrected to read "BPXWUNIX" instead of "BPXWDYN".

(Originally posted 2011-11-04.)

I don’t think I’ve mentioned this before in this blog but Tivoli Workload Scheduler (TWS) has a nice "WLM Integration" feature. With it TWS can change the Service Class a job runs in – before submission.

The main purpose of this is to elevate Critical Path work.

We wrote about this in the Redbook mentioned in Touring The Upcoming Batch Optimization On z/OS Redbook, and so some of you will have seen me begin to discuss the area in the presentation.

This week Dean Harrison (a friend who is a renowned expert on TWS) and I presented together on "TWS and WLM". It’s all about this linkage. He talked about how you goad TWS into changing the WLM Service Class for a job. Very sensibly he didn’t say "a better Service Class" but rather "a different Service Class". I was watching for him to trip up but he didn’t. 🙂

(He also did a nice job of handling the whole fraught question of what a Critical Path actually is. I think we all know the score here.)

But back to the "different Service Class" thing: We all know that it doesn’t matter what you call a Service Class, it’s what the goal is that counts. In my portion of the presentation I did the "now you’ve got a Service Class now what?" piece. So I talked, for instance, about whether response time goals were usable with Production Batch or whether you should stick to velocity goals. (On that one I think only if the batch looks like a large number of independent really heavy transactions should you use response time goals.)

So, my role was in the vein of the cliche "what you don’t know can hurt you" for Batch Operations folks. My message to them was "understand what the service classes you’re steering work towards actually mean".

But let’s stand it on its head: From the Performance Analyst’s perspective the cliche still applies:  How do you know what the batch is that shows up in PRDBATHI vs PRDBATMD? You don’t unless you take an interest in how the jobs got into either of those classes. So my message to "us" is "understand how work came to be in each Service Class and what expectations the Batch Operations folks have of it".

The "can hurt you" in the cliche is interesting: You could – whether a Batch Operations person or a Performance Analyst hide behind "separation of concerns". I wouldn’t recommend it though: Presumably installations take a dim view of such things. Another cliche on offer is "hang together or hang separately". I encourage both camps to work together. (Just as I encourage eg CICS and Performance people to.)

Best of all is if you can be skilled in both  TWS and WLM. A tall order, I know. But that might earn you a leading role in the (proposed in the Redbook) Batch Design Authority.

(Originally posted 2011-10-25.)

One of the nice enhancements in z/OS Release 13 JES2 was the support for SYSIN in a JCL procedure. (See here for the announcement letter.)

I have a personal example of where it would’ve been handy. You probably have your own.

We used to distribute sample JCL to use DB2 DSNTIAUL to unload the DB2 Catalog, one table at a time. Obviously with such a repetitive use in a single job step you’d want to use a JCL procedure. And so we did. But there’s a snag:

Each invocation of DSNTIAUL needed the same SYSTSIN DD data. The example below is for DB2 Version 8.

RUN PROGRAM(DSNTIAUL) PLAN(DSNTIB81) PARMS('SQL')
END

You can see that for Version 9 you’d need to change it to DSNTIB91 and anyway neither the program nor the plan name are fixed – as DSNTIAUL is a sample you compile etc yourself. So if you change it you either change it in the PROC or you don’t have one and make a global change.

Changing it in the PROC would be preferable, of course. But it couldn’t be done. So we added a step to copy these two lines to a temporary data set and refer to that from the PROC. Cumbersome but a well-known circumvention.

Enter Release 13. The need to do this has gone as you can now have SYSIN instream in a PROC.

Of course, if I were maintaining this JCL still I wouldn’t rush to publish a version that assumed Release 13. 🙂 I hope in your shop you can use it sooner than I can.

(Originally posted 2011-10-20.)

I mentioned APAR OA21140 before – here. It’s quite an old APAR and so it will (in all likelihood) be on your systems.

I’d like to draw your attention to a subtle 1-byte field in the SMF 74 Subtype 4 (Coupling Facility Activity) record: R744FLPN. It’s the partition number for the coupling facility. (If you’ve seen any one of several of my presentations I’ll’ve talked about this field.)

Here’s a problem I have but most of you don’t: I try to build a picture of your systems just from SMF / RMF data. (Actually I’ve run into lots of customers who have almost the same problem. From a "systems should be self-documenting" standpoint, if nothing else.) Let’s review what we have:

• SMF 70 Subtype 1 gives you lots of information about a machine, including the serial number and all the LPARs – all in gory detail.
• SMF 74 Subtype 4 gives you lots of Coupling Facility information – again in gory detail.

It’s been my contention that if you put these two together good stuff can happen:

• You can get a true picture of the CPU performance of coupling facility, especially in the shared ICF engines case.
• You can match the CF links to the CHPIDs from SMF Type 73 (not that the latter will give you much information).
• You can identify what each of those LPARs that show up using resources in the ICF engine pool actually are.
• In principle I can identify free CF memory that could be redeployed to other LPARs as needed.

Maybe the first of these is the most significant but, as you know, I like to show up in a customer having got quite close to their systems – not asking the questions I should’ve got the answers to from the data. So the next two are nice as well. I consider the fourth "unfinished business" as we don’t have a complete picture of machine memory but it’s still valid.

Today I finally exploited R744FLPN to do the matching. So you might like to, too. I could even define a view across SMF 70 and 74-4 with a full set of matching keys now. Whoo hoo! 🙂

A long time ago I encountered a customer with multiple coupling facility LPARs whose names didn’t match the coupling facility names. So I wrote some code that only worked in the "1 coupling facility on a machine" case: Fairly trivial code.

Then RMF added machine serial number, the number of dedicated and shared engines and the LPAR weights to the 74-4 record. This helped with a customer case where every CF LPAR had the same name – across multiple machines. (Why do people do that?) 🙂

But in asking them to do that I forgot one other thing: LPAR Number. So now OA21140 provides that and we can do a direct match (where the code doesn’t have to do lots of special-case detective work).

As I said, this APAR is quite old. Usually I like to take advantage of new data as soon as a customer sends me it. But I’ve been busy these past 2 years like never before. (You might’ve noticed that.) 🙂 So, finally I have the change in my code in to take advantage of that. (And I still have the other code in place as a fallback.) It turned out to be a tiny change – that took me 15 minutes to code and about the same to test. You’d be amazed at what can get done in "interstitial time". 🙂

All in a day’s work for a Principal Systems Investigator (PSI). 🙂 

(Originally posted 2011-10-17.)

I’ve been looking for an excuse to reference the excellent B52’s Channel Z ever since we started talking about "z". And this is, I think, a good one. 🙂

(Notice BTW the URL in the previous paragraph makes it clear SING365.com (should be "SING360" 🙂 ) is using Domino.)

So, back to the matter at hand…

A customer I’m working with is using Intelligent Resource Director (IRD) Weight Management together with HiperDispatch. Note: I said "Weight" not "Logical Processor" because you can’t use IRD Logical Processor Management with HiperDispatch. The way this works is that IRD changes the weights – if it needs to – and HiperDispatch recomputes the vertical weights and (potentially) adjusts the number of Vertical High, Vertical Medium and Vertical Low processors accordingly. (Adjusting these counts is not the same as parking and unparking, which is an important point to bear in mind.)

My personal interest is as much about how to tell the story as it is about the actual performance. Well, not quite. So, in this case the story is that weights shift within an LPAR Cluster between one LPAR and another. This happens, in my analysis, in a credible and smooth way. There’s no "nervous kitten" here. So the HiperDispatch adjustments happen as well. But how do we show this?

Showing the shifting weights is easy: Stack up the weights for the LPARs by time of day. The total stack height remains constant – which we should expect. (IRD only shifts weights within the cluster.) 

The complicated thing to depict is the number of High, Medium and Low logical processors. The easy bit is to do this by time of day. But suppose, as I was, you’re producing an LPAR layout table? Which is a static depiction. Well, let’s rehearse what the instrumentation available to us is…

APAR OA21140 does a number of things. The relevant thing it does is to introduce SMF70POF in the SMF 70 Subtype 1 record. It’s in the Logical Processor Data Section, and you get one for each logical processor in each LPAR. The field tells you whether the logical engine is vertically polarised and whether it’s High, Medium or Low. (As the Low ones can get parked it helps explain the parking and unparking elements of HiperDispatch’s behaviour.) I take this to be the state at the end of the RMF interval and note there’s a bit which says if it changed during the interval.

This is easy to use in a table if you have a static situation. But, of course, IRD Weight Management makes it dynamic. So, when I summarise over a shift (of maybe 8 hours) some logical engines have changed between Medium and Low (and maybe some are only sometimes High). It all depends on the degree of shifting of weights. So some logical engines aren’t totally in one state throughout the 8-hour shift.

Here’s a way around the problem:

I define a fourth state in my reporting: "Vertical Transitioning" . So you might see an LPAR in my table as, for example, "VH:2 VM: 1 V?: 1 VL: 8". In this example 2 logical engines remain Vertical Highs all the time, 1 is always a Vertical Medium, 8 are always Vertical Low engines. Finally the "V?" means one logical engine transitions between states (possibly VM and VL).

"Vertical Transitioning" isn’t a technically recognised term but I think it captures the essence of the behaviour. Of course there’s wouldn’t be any Vertical Transitioning engines if IRD weren’t shifting the weights.

Which brings me back to the B52’s: "Channel Z’s All Static All Day Forever" is anything but true these days. 🙂

(Originally posted 2011-10-14.)

As mentioned here, I recently participated in a residency in Poughkeepsie. Our task was to write the "Batch Optimization on z/OS" redbook that we hope will get out soon (though I think early next year is the most realistic timescale).

These past two weeks I "toured" the redbook in the Nordics: 5 cities in 2 weeks (Helsinki, Copenhagen, Oslo, Stockholm and Aarhus). I really wish I could’ve taken the whole "band" on tour – but that wasn’t to be. 😦

So here’s the abstract for the presentation:

---

Batch performance optimization remains a hot topic for many customers, whether merging workloads, supporting growth, removing cost or extending the online day. This presentation outlines a structured methodology for optimizing the batch window, incorporating techniques written about in a Redbook written by experts from around the world. This methodology is well-structured and draws on information every installation should have access to."

---

I’ve just uploaded the slides to Slideshare. If you read them the first section is a small amount about the residency. The rest of it pretty much follows the book’s structure. (I mention in the first section that other residents (in the other room) were working on batch containers. I don’t describe these.)

So, I hope you enjoy the slides, and I’m pleased so many people were able to be in my audience. I’d like to reach even more of you some time – as I think our messages are important. And I also hope I represented the viewpoints of my fellow residents adequately: Though we all walked in with very different ideas I think we managed to very constructively meld them together into a methodology we can all be proud of.

(Originally posted 2011-10-10.)

I’ve written about Hackday in the past. See My HackDay 6 Project – Mashing Up RMF, My HackDay 5 Project – z/OS System Logger Analysis and Hackday4 and Referer URLs (though I don’t know why I didn’t write about Hackday 7 or Hackday 8).

So, this year I participated in Hackday 9 (last Friday). My entry follows on from this recent blog post: BPXWUNIX – z/OS’ Best Kept Secret?. Graham Harris, in his comment, mentions using java to process XML (albeit with issues with the XML generated by RMF, caused by the 2048 byte record limit of BPWUNIX). I’d been thinking along similar lines, though not RMF output specifically.

So my entry has at its core a java program that uses standard functionality to process XML. In particular I pass a filename and an XPath expression in to the java program and it returns the results of evaluating the expression. I intend it as a sample, rather than a really useful program – but I think it COULD be used.

I call the java program from REXX using BPXWUNIX. There is a wrinkle here. I’d not realised it before but you can pass in environment variables. Here’s the REXX:

/* REXX */ 
                                                                    
/* Set up environment - else java executable can't be found */ 
env.0=1 
env.1="PATH=/bin:.:/usr/lpp/java/J5.0/bin:." 
                                                                    
/* No standard input lines in this case */ 
stdin.0=0 
                                                                    
/* XML file name */ 
ifn="test4.xml" 
                                                                    
/* XPath expression to evaluate */ 
XPathExpression="//book[author='Neal Stephenson']/title/text()" 
                                                                    
/* Execute java program with XML file name and XPath expression */ 
cmd='java XPathEvaluator.class' ifn '"'XPathExpression'"' 
call bpxwunix cmd,stdin.,stdout.,stderr.,env. 
                                                                    
say "stdout:" 
say "=======" 
do i=1 to stdout.0 
  say i stdout.i 
end 
                  
say "stderr:" 
say "=======" 
do i=1 to stderr.0
  say i stderr.i 
end 

Apart from the "env" lines at the top (and the extra parameter to BPXWUNIX) and the java command this is exactly the same as code as in BPXWUNIX – z/OS’ Best Kept Secret? and so needs no explanation.

 

Here’s the XML file we’re processing, so the XPath expression makes sense.
 
 

<?xml version="1.0"?> 
<inventory> 
  <book year="2000"> 
    <title>Snow Crash</title> 
    <author>Neal Stephenson</author> 
    <publisher>Spectra</publisher> 
    <isbn>0553380958</isbn> 
    <price>14.95</price> 
  </book> 
  <book year="2005"> 
    <title>Burning Tower</title> 
    <author>Larry Niven</author> 
    <author>Jerry Pournelle</author> 
    <publisher>Pocket</publisher> 
    <isbn>0743416910</isbn> 
    <price>5.99</price> 
  </book> 
  <book year="1995"> 
    <title>Zodiac</title> 
    <author>Neal Stephenson</author> 
    <publisher>Spectra</publisher> 
    <isbn>0553573862</isbn> 
    <price>7.50</price> 
  </book> 
  <!-- more books... --> 
</inventory> 

In the XPath expression we look for the text in the title element of each of the book elements whose author element’s value is "Neal Stephenson". (I got this XML and much besides, by the way, from The Java XPath API – Querying XML from Java programs. The result of evaluating the expression is:

stdout:    
=======    
1 Snow Crash
2 Zodiac   
stderr:    
=======    

and this is because only two of the book elements match the [author="Neal Stephenson] condition.
 
The java to drive this is:

 

import java.io.*; 
                                                                    
import org.w3c.dom.*; 
import javax.xml.parsers.*; 
import javax.xml.xpath.*; 
                                                                    
public class XPathEvaluator{ 
                                                                    
  public static void main(String args[]) throws Exception{ 
    DocumentBuilderFactory dbf=DocumentBuilderFactory.newInstance();
    dbf.setNamespaceAware(true); 
                                                                    
    DocumentBuilder docBuilder=dbf.newDocumentBuilder(); 
                                                                    
    Document doc=docBuilder.parse(new FileInputStream(args[0])); 
                                                                    
    XPathFactory xpf=XPathFactory.newInstance(); 
    javax.xml.xpath.XPath xp=xpf.newXPath(); 
                                                                    
    XPathExpression expr=xp.compile(args[1]); 
                                                                    
    Object result=expr.evaluate(doc,XPathConstants.NODESET); 
    NodeList nodes = (NodeList) result; 
     for(int i=0;i<nodes.getLength();i++) { 
       System.out.println(nodes.item(i).getNodeValue());
     } 
   } 
 } 

This is also largely described here.

So you have the sample REXX, java and XML to play with. A couple of things I would note:
 

• There is only one pair of "boundary crossings" between REXX and java – which is, I think, goodness.
• Because the XML parsing is done in java (through XALAN-J) I expect this to be very largely zAAP-eligible. Another nice characteristic.

 

 One final thing: If you search in my blog you’ll see I mention XPath in Workload Manager Policy in XML Format – Part I

and Workload Manager Policy in XML Format – Part III.

(Originally posted 2011-10-01.)

I was going to start this post with an apology. But, as any sensible blogger would, I left it a few days to write this. Now I realise that there’s a wider point than the "I was wrong" one. (But I was wrong – in a way that I think many other people might’ve been wrong too.)

So let me talk about two things in this post:

• zAAP and zIIP Delay.
• How I came to be wrong and what we can all learn from it.

zAAP and zIIP Delay

The fields we’re talking about here appear on the RMF Workload Activity Report and are – in the Type 72-3 record – R723IFAD (for zAAP) and R723SUPD (for zIIP). Personally I use the latter as shown in zAAP and zIIP Revisited (At Last).

They are described as "zAAP delay samples" and "zAAP using samples".

(Hereafter I’m going to drop the "or zIIP" bit. And, by the way, zAAP-on-zIIP doesn’t affect the discussion significantly.)

But what does R723IFAD mean? I had assumed (perhaps mentally fuzzily) that it meant "delay samples because zAAP was unavailable" (likewise zIIP). So my recommendation would have been that the way around it was to provision more specialty engine capacity to the service class period.

It turns out that’s not the right interpretation of the field. Here’s a better one:

For a delay sample to be declared for a service class period for zAAP all of the following criteria have to be met:

• We were trying to run zAAP-eligible code. (I think we knew this.)
• No zAAP could run the work. (My basic assumption.)
• No general-purpose engine (GCP, in my parlance) ran the work. (This is the new bit.)

So, seeing significant samples in the "Delay for zAAP" field means we not only didn’t get to run on a zAAP but we also didn’t run on a GCP. And the implication here is that understanding all this requires us to weave in the GCP view. We could be short on both zAAP and GCP capacity.

Now, I would guess the "take home" is still provision more zAAP capacity to the service class period – if you want to increase its velocity and Delay for zAAP is the major issue. (There are some rare cases where that mightn’t be right – given processors come in integer numbers.) But the reasoning is a little different: You’d rather run zAAP-eligible work on a zAAP than on a GCP, I would think.

For completeness, the "CPU Delay" field (R723CCDE) is for non-zAAP-eligible work and "CPU Capping Delay" (R723CCCA) is also for non-zAAP-eligible work. (Helpfully the SMF manual states that R723CCCA is not a subset of R723CCDE.) If R723CCDE / R723CCDE come into play, then, it’s about provisioning GCP capacity – or, distantly, finding a way to make more of the work eligible for zAAP.

The Wider Lesson

We’re all used to reading numbers off reports and taking them at "face value". If some metric is called "splodgeness" and the value is high we say "splodgeness abounds" without necessarily giving it too much thought. But what is this "splodgeness" whereof we speak?

Often all we get is the description "zAAP delay samples." (If you think we get more then do please look at the SMF manual’s description for R723IFAD.) So we tend to:

• Cling to the certainty the existence of a particular metric gives us. I think we’re grateful to have the metric. After all, consider the counterproposition.
  
• Invent for ourselves an interpretation of what the metric means. I say, perhaps rudely, "invent" because who’s to say if we have the right interpretation? We have to gain a foothold somehow. So actually I’m entirely sympathetic.
  

So, in response to a customer question I set off in search of an answer to the question "what does R723IFAD mean?" I have a friend in RMF who mentioned they got the number from WLM and suggested I ask a mutual friend in WLM. He, very usefully, pointed me at Dan Rosa in Systems Software Development in Poughkeepsie. Dan and I chatted for well over an hour and he helped form a very good understanding of what this field means. So many thanks to Dan!

Now, I count myself as very lucky in having friends in such useful places. I realise that’s a privilege. And I don’t tend to bombard them with questions about each and every SMF record’s field.

So, I think there are lots of fields like that. I’ve stumbled across a fair few. It would be nice to revive the old RMF Field Description manual (last updated in the early 1990’s). I don’t think that’s going to happen, unfortunately. And it would take forever to bring it up to date.
 
But I do think it’s legitimate to gain an understanding of where a field came from, why it was invented, and how it behaves. And that’s what I try to do – for fields I think tell a useful story. And that’s part of why I actually like questions about fields, and part of why I feel like a "kid at Christmas" whenever new data arrives: It gives me a chance to see how this stuff behaves and how y’all’re using our hardware and software.

---

So, in conclusion, we learn and grow together. But there’s always room for better understanding. I guess I knew all this. Tacitly, I expect a lot of you will share my experience.

(Originally posted 2011-09-27.)

Maybe you’ve all heard of BPXWUNIX. Maybe it’s just me that hasn’t. Hence the question mark in the title.

Seriously, I doubt this REXX function is as widely known about as it should be. And this post hopes to illustrate a little of its value.

Suppose you’re writing a REXX program with a stem variable in it. Suppose you want to sort the strings in the stem variable:

• You might write your own sort routine. Not terribly difficult but a little fiddly.
• You might write it to a temporary file, do some dynamic allocation and call DFSORT, reading back from another temporary file. (I did that in something I called "NICETOOL" a long time ago.) This also is fiddly.

or you might want to try the BPXWUNIX approach…

BPXWUNIX is a very nice REXX function. It allows you to pass the contents of a stem variable to a z/OS Unix program and to return the results in two more stem variables. In a little more detail:

• The first parameter is a Unix command string.
• The second parameter is a stem variable where .0 has the count of the number of lines. This is stdin.
• The third (optional) parameter is another stem variable where each item is a line of stdout.
• The fourth (optional) parameter is, likewise, stderr.

Anyone familiar with Unix will know about stdin, stdout and stderr.

Now, there is a standard Unix sort program. This bears little resemblarnce to DFSORT. But that’s not the point. Consider the following program:

/* REXX */ 
stdin.0=5 
stdin.1="KIJJ" 
stdin.2="KQWR" 
stdin.3="ADGF" 
stdin.4="OEPE" 
stdin.5="VNVV" 
                                        
cmd="sort" 

call bpxwunix cmd,stdin.,stdout.,stderr.
                                        
say "stdout:" 
say "=======" 
do i=1 to stdout.0 
  say stdout.i 
end 
                                        
say "stderr:" 
say "=======" 
do i=1 to stderr.0 
   say stderr.i 
end 

The highlighted lines are the most interesting ones. Between them they execute the Unix sort command, marshalling stdin, stdout and stderr. The lines before set up the strings to be sorted and the ones after print the results. (You could source the strings and deal with the results any way you wanted.)

So this is quite simple. If you want to know more about sort go here (for example).

The simple command "sort" without parameters sorts the records alphabetically – based on each record’s entire contents. That may not be what you want. So, without regurgitating the manual:

• There are numerous parameters that control the sort – on the Unix side.
• You might be able to help yourself by "packaging up" the data on the REXX side appropriately.
• You might want to pipe the data into or out of the sort with other Unix commands.

Of course, in this post sort is just an example. There are lots of Unix commands that could do things more easily than REXX could. And this might help along programmers who are primarily used to Unix.

I’m not sure, by the way, that continual crossing of the boundary between REXX and Unix commands is the highest performing thing to. But in this case we only do the round trip once. I’ve not done any performance tests but it seemed to be OK. I suspect that calling grep on a single line of data continually would perform well. But that’s an extreme case.

What I fancy doing – when I get the time – is experimenting with the ideas for XML parsing here.

One limitation you should be aware of is that lines can only be up to 2048 characters long. Ordinarily that wouldn’t be a problem – except for processing SMF. (Not that REXX can directly read SMF anyway. The standard dodge is to copy the VBS data to a transient VB data set and then process it with REXX.)

Any other examples of using BPXWUNIX you’d care to share?

(Originally posted 2011-09-25.)

I can’t believe it’s been almost a week since I wrote Batch Capacity Planning, Part 1 – CPU . Where did the time go? 🙂

Re-reading it I’m struck by the overwhelming theme of Batch’s unpredictability and lumpiness. This is true of memory, as well, but to a much lesser degree.
 
Why to a lesser degree? Well, in most systems I look at the memory usage is mostly fairly constant and dominated by big "server" address spaces (such as DB2) or else CICS regions (and the analogue in the IMS world). So the variability of the Batch workload is overshadowed by the constancy of these other address spaces.

I oversimplified there just a tad. for example:

• A few customers alter their DB2 buffer pools  – using the, unsurprising, ALTER BUFFERPOOL command. (They increase the size and sometimes change the various thresholds, particularly if they perceive Sequential processing to dominate overnight.)
• Many customers shut down some, if not all, of their CICS regions overnight. Decreasingly so, though. Likewise IMS MPRs.

But these changes are step changes and conscious acts of will. Do take them into account, but don’t worry about their variability.
 
But there’s an issue anyway with Batch memory analysis:

• At the System level RMF reports good numbers: The various frame queues and Available Frames are all accurate in SMF Type 71
  
• At the Service Class level (SMF Type 72) the numbers are less reliable: We use R723CPRS (Resident Page Frame Seconds) to figure out how much memory a service class (period) is using. This works fine for Non-Swappable work (such as CICS and DB2 and System). It doesn’t work well for Swappable work (such as TSO and Batch): When you’re logically swapped you’re said not to accumulate (memory) service. So Swappable work is under-counted.

This has been such a nuisance that my standard Memory graph has to take it into account: I take the Online frames and subtract all the non-workload frame queues – LPA, CSA, SQA and the like – and the Available Frames. These all come from SMF 71. What’s left ought to be workload-related memory usage. So I then subtract all the memory usage (R723CPRS-driven) for all the workloads – including Batch and TSO. What’s left I call "Other".

For the most part Other is the under-representation of Swappable workloads –  Batch and, distantly, TSO. But I sometimes see a veneer of constancy – which could be a miscalculation or another category of memory entirely. But at least I can add "Other" into the Batch workload to get a reasonable estimate.
 
What I do note is that, generally, there’s lots of memory in the Available Frames category i.e. Free – during the Batch Window. So it’s almost always true that – from the Memory perspective – Data In Memory (DIM) or Parallelism could be increased.

I find the SMF Type 30 data to be almost entirely useless for memory usage purposes – especially for Batch. About the only thing you can do with it is to use the Virtual Memory fields and pretend that each virtual page allocated is backed by real memory. Which we all know to be a (usually) gross overestimate.So I don’t actually do that.

For Batch, though, the biggest user of memory is typically DFSORT. Now there we have good news: The instrumentation does a nice job of summarising peak memory exploitation: Whether Dataspace, Hiperspace or Large Memory Object sorting. Note: I say "Peak" here. You might be able to do something to turn that into an average. But that would be a little fraught.

All the above has been about actual usage. What about projecting forwards? If you know nothing’s going to change the picture is likely to be static. If you think you’re going to exploit DIM or Parallelism it’s much more difficult, for the same reasons as it was for CPU. But there’s an additional reason:
 
If you were to be very successful at buffering something – especially with only a small number of buffers – then you hold onto the memory for less time. By exploiting DIM well the "area under the curve" could actually go down. Such a case might be using VSAM LSR buffering. I’ve seen the kinds of speed up where this is likely: 2 million I/Os to under a thousand, for example.

So, in summary, Memory Capacity Planning for Batch shares the difficulties of CPU. But it also has a few of its own.