(Originally posted 2013-07-18.)

Back in May I wrote about a new batch residency planned for this October and invited good people to apply to join the team. It’s been very pleasing how many people applied to be residents and the high quality of the entrants: It was genuinely difficult to pick the eventual team. We also had to reduce the scope a bit – which was a disappointment to both Frank Kyne and I. So, if you didn’t get selected I’d encourage you to apply for other residencies as they come up.

So here’s the final team:

• Karen Wilkins – DB2 and COBOL.
• Dean Harrison – Tivoli Workload Scheduler, JCL and Operational Considerations in general.
• Myself – orchestrating and doing the Performance piece, and probably playing with some XML or JSON.

You’ll notice we lost PL/I and VSAM. I’m hoping to sneak some VSAM in – and definitely some Sequential Data Set processing. You might disagree but the loss of PL/I is less severe to me than not dealing with VSAM issues.

Though the residency starts 7 October the next phase is putting some materials together for us to work from and finalising our (large) sample data and the environment we’ll run on. I’m also hoping to have access to our (z/OS 2.1) system so I can try out the idea that you can process SMF data with REXX, and maybe a few other things. You’ll appreciate, though, that the “day job” intervenes quite often. 🙂

We hope to make this quite a “social” residency – so some blog posts and tweets will come out. Stay tuned! And you might want to follow us on twitter: Karen, Dean and myself.

We also are pencilled in to present some of the material at GSE Annual Conference just a few days after the residency finishes in early November – which will be a challenging deadline. With luck I’ll get to present a more polished version at the UKCMG One Day Conference in late November. But I’m not on the agenda yet.

Stay tuned!

(Originally posted 2013-07-16.)

You could consider this another in the Small Programming Enhancement (SPE) 🙂 series. You’ll probably also notice I’ve been doing quite a lot of REXX programming recently. Anyway, here’s a tip for refactoring code I like.

Suppose you have a line of code:

norm_squared=(i-j)*(i-j)

that you want to turn into a function.

No biggie:

norm2: procedure 
parse arg x,y 

return (x-y)*(x-y)

and call it with:

norm_squared=norm2(i,j)

But what about the process of developing that function? Most naturally you might scroll to the bottom of the member you’re editing and add the function there. But that’s a pain as your iterative development would require you to scroll down to where it’s defined at the bottom and up to where it’s called. Many times over.

Try the following, though:

/* REXX */ 

do i=1 to 10 
  do j=1 to 10 
    say i j norm2(i,j)

/* do never */
if 0 then do 

norm2: procedure 
parse arg x,y 

return (x-y)*(x-y) 

/* end do never */
end

  say "After procedure definition"
  end 
end 
exit 

It works just fine: The “if 0” condition is never met: It’s effectively a “do never” and you have to have it or else the REXX interpreter will crash into the function definition. (You can’t use a SIGNAL instruction if this is inside a loop or function definition for this purpose as it throws away the context. Early attempts did do this but it immediately failed once I tried it inside a loop: You learn and move on.)

What this enables you to do is to develop the function “inline” and then you can move it later – to another candidate invocation or indeed to the end of the member (or even to a separate member).

It saves a lot of scrolling about and encourages refactoring into separate routines. It’s not the same as an anonymous function but it’s heading in that direction, in terms of utility.

In that vein you might choose not to move the function away from “in line” – particularly if its use is “one time”. If you don’t you could consider an unmnemonic function name, such as “F12345” – which I wouldn’t normally recommend. But sometimes it’s really hard to come up with a meaningful function name. 🙂

(Originally posted 2013-07-15.)

In REXX sorting is a fraught exercise – and you tend to have to resort to other programs to do it:

• DFSORT for the “heavy lifting”
• UNIX Sort

but you might decide your sort is low volume enough to do in REXX. The trouble is it’s difficult to write a routine that isn’t specific on how the data is sorted, or rather how two items are compared – as we shall see.

Following on from Dragging REXX Into The 21st Century? here’s a technique that is quite general. It uses the REXX INTERPRET instruction.

---

The problem of sorting comprises two challenges:

1. Comparing two items.
2. Deciding which two items to compare when, and acting on the results of the comparison. Much has been written about this, perhaps most notably in Donald Knuth’s Sorting & Searching.

This post is mostly about Challenge 1, though the code presented also addresses Challenge 2.

Comparing Two Items

It’s easy to do a straight numeric comparison in REXX, such as

if a<b then do
  ...
end

but more complex comparisons are, ahem, more complex. 🙂 But, more to the point, a generalised comparison is difficult to do, which is where INTERPRET comes in.

Sample Sort Code

In what follows the program works. That isn’t to say the sorting algorithm is the most efficient over all sets of data, but this post isn’t really about the sort’s efficiency.

/* REXX */ 

/* Set up stem variable to sort */ 
sortin.0=5 
sortin.1="AXXX XS" 
sortin.2="HXXXX BSS" 
sortin.3="FXXX IS" 
sortin.4="XSSSSSSSS ZS" 
sortin.5="A JSS" 

/* Sort the stem variable */ 
call sortStem "comp1" 

/* Print the results */ 
do s=1 to sortin.0 
  say sortin.s 
end 

exit 

/* Sort comparison routine based solely on length of second word */
comp1: procedure 
parse arg x,y 

/* Calculate lengths of second word of x and y */ 
lx=length(word(x,2)) 
ly=length(word(y,2)) 

select 
when lx=ly then return 0 
when lx>ly then return 1 
otherwise return -1 
end 

sortStem: 
parse arg comparison_routine 
flip_flop=0 
do i=sortin.0 to 1 by -1 
  flip_flop=1 
  do j=1 to i-1 
    /* Decide whether to swap */ 
    interpret "compRes="comparison_routine"('"sortin.i"','"sortin.j"')"
    if compRes<0 then do 
      should_swap=1 
    end 
    else do 
      should_swap=0 
    end 

    if should_swap=1 then do
      /* Swap values */ 
      xchg=sortin.j 
      sortin.j=sortin.i 
      sortin.i=xchg 

      flip_flop=0 
    end 
  end 
end 
return

The item comparison routine in this example is, of course, somewhat artificial – and not one I’m likely to use. It compares the length of the second token in each stem variable. I chose it as an example of something that’s not easy to do in, say, DFSORT. The real point is you can compare items using arbitrarily complex criteria. If your collection of stem variables was more complex than this – essentially a list of objects – you’d want the sophistication this approach allows.

---

Incidentally when it comes to INTERPRET performance I think it depends on which statements are actually interpreted and which have already been tokenised:

• If your “heavy lifting” is the code passed to the INTERPRET routine that would be expensive.
• If you pass a call to a routine to INTERPRET that isn’t – so long as the actual routine is “static”, meaning appears in your source code where it can be tokenised.

In the above example code you’ll notice I’m using a hard-coded set of stem variable names. In my actual production code I’ve generalised to allow any stem variable names – so I’m actually much heavier in my use of INTERPRET.

I’m leaning towards INTERPRET in more cases – provided it doesn’t obscure the meaning.

At the start of this post I mentioned DFSORT and Unix sort as alternatives. I think you should consider them, though their limitations and the need to work hard at “marshalling” to use them will limit their appeal. So, I hope you’ll find the technique presented in this post useful for the cases where they’re not appealing.

The most important piece of this post – and the main reason for writing it – is the notion of using a single sorting routine and handing it the name of an item comparison routine. This should lead to code that’s more maintainable. Its long been the staple of other programming languages. You can do it with REXX, too!

(Originally posted 2013-07-14.)

A couple of days ago I had a need to take a REXX string comprising space-separated numbers and find their minimum and maximum values. Here’s the technique I used.

(When I say “space-separated” there can be one or more spaces between the numbers, but there has to be at least one.)

The solution has three components:

1. The REXX SPACE function – to turn the list into a comma-separated string of numbers. (The second parameter is the number of so-called spaces to separate tokens with. The third is the actual character to use – in my case a comma.)
2. The REXX MIN (or MAX) function to compute the minimum (or maximum) value from this comma-separated string. These functions take a set of parameters of arbitrary length and do the maths on them. Parameters are separated by commas, hence the need to use SPACE to make it so.
3. INTERPRET to glue 1 and 2 together.

My need is relatively low volume, so the “health warning” about INTERPRET’s performance is hardly relevant for my use case.

Here’s the code:

/* Return min and max value of string of space-separated numbers */
minAndMax: procedure 
parse arg list 
comma_list=space(list,,",") 
interpret "minimum=min("comma_list")" 
interpret "maximum=max("comma_list")" 
return minimum maximum 

It’s relatively straightforward, taking a list of numbers and returning the minimum and maximum. You’ll notice it doesn’t check that the tokens really are numbers. If I were to extend it I’d probably check for two SLR conditions: Overflow (“*” or similar) and Missing Value (“—” or similar). I’d probably take some of the “List Comprehension” stuff I talked about in Dragging REXX Into The 21st Century? and apply it to the list.

And my code uses this to decide if I have a range of values or just a single one. In the former case it turns the pair of numbers into e.g. “1-5” and the latter just e.g. “4”.

Of course there are other ways to do minimum and maximum for a list of numbers but this one seems the simplest and most elegant to me. “6 months later me” might take a different view. 🙂

(Originally posted 2013-07-07.)

9 months on from this post and I finally get some time to enhance my Coupling Facility reporting.

If you recall I mentioned OA37826, which provides new SMF 74 Subtype 4 data and fields in the RMF Coupling Facility Activity report – in support of enhancements in CFLEVEL 18.[1]

I also mentioned that my 74–4 code needed reworking to properly support the Remote Coupling Facility Section [2] and that this is a prerequisite to working with the new Channel Path Data Section. And part of the reason why this has taken 9 months to get to is that it’s a big set of changes, with much scope for bustage. The other is not having seen enough data to test with.

I have some test data from a very large customer Parallel Sysplex, containing a mixture of zEC12 (at CFLEVEL 18) and z196 (CFLEVEL 17) and z10 (CFLEVEL 16) machines. There are five coupling facilities in all. This is a perfect set of test data for my purposes.

Handling CF Duplexing Properly

The 5 CFs potentially duplex with each other.[3]

Prior to this week’s programming my code picked the first Remote CF Section where the Remote CF wasn’t the same CF as the Local [4]. It just mapped the whole section. With 5 CFs that clearly isn’t good enough.

After a major restructuring of the code I now have all the CFs a specific CF talks to.

That in itself should make for some interesting discussions – particularly when I throw in the traffic and Service Times the Remote CF Sections give me.

Channel Path Identifiers (CHPIDs)

There was a pleasant surprise here: Looking at the data it appears that with OA37826 you get CHPID numbers in both the Local Coupling Facility Data Section (1 per SMF 74–4 record) and the Remote Coupling Facility Data Sections.

It doesn’t matter whether the CF is at CFLEVEL 18 or not: I still see’em.

Previously I’ve relied on the SMF Type 73 (RMF Channel Path Activity) record to tell me the Coupling Links an LPAR has. But SMF 73 can’t tell me which CF they are to (if any). It also can’t tell me which LPARs they’re shared with.

So my code has had to guess. 🙂 And it guessed conservatively. 😦

Now I can see the CHPIDs and the SMF 73 tells me which LCSS[5] the LPAR is in. So I can tell which CHPIDs go to which CFs and which are shared with whom. I can see this even for Internal Coupling (IC) Links. And I can see it for CF-to-CF links.

This was a pleasant surprise as, scouring all the documentation I have, I didn’t expect OA37826 to have value prior to CFLEVEL 18.

It means you can draw out the topology of the Parallel Sysplex much better. If you want to (and I certainly do). More to the point it gives you the ability from data to check you got what you thought you’d designed.

One point on this: I’ve noticed extraneous CHPIDs in the arrays, so it’s important to know how many CHPIDs there are (from other fields in the sections) and only accept the CHPIDs for those. I don’t know whether it’s worth getting that fixed.

Next Steps

So I haven’t even got to the CFLEVEL 18 – specific stuff in my programming. But I can now start on it. And I expect it to yield some valuable information about signal timings and interesting estimates of distance – from the new Channel Path Data Section.[6] That and some information about adapters – which I hope is more than just “tourist information”.

And when I’m done I’ll tell you what I see in this data too.

(Originally posted 2013-06-30.)

I’ll admit it: I have a dirty little secret. 🙂 All I can say is it was in my youth – and I didn’t know any better. 🙂

And folks I’m exclusively revealing my secret here in this blog… 🙂

I learned to program in BASIC. 🙂

Actually the “I didn’t know any better” bit is a lie: My Dad can attest to the fact he had Elsevier Computer Monographs in the house, which I read avidly. (Actually I added to the collection when at university and these books are among the few paper Computing books I still have.) And his view, which I think was the prevailing industry view, was that BASIC rots the brain and Algol is much better.

I don’t disagree about Algol being a superior language. I do disagree that BASIC teaches bad habits you can’t overcome. But you use the language you have available to express yourself. And, by learning enough languages or gaining experience, you develop programming maturity. (You’ll’ve guessed, I hope, that Algol wasn’t available to me.)

To be fair to the “BASIC is horrid” camp it is horrid 🙂 but I don’t think longitudinal studies had been conducted that proved it to be a stupefiant (great French word) 🙂 And, to my mind, getting people programming is more important: We can work on points of style later.

The next steps in my programming career (verb or noun?) 🙂 were Z80 Assembler, a dialect of OPAL, and CORAL. Again used in the “program in what’s available” mode.

I don’t think descending into Z80 Assembler (and actually, more interesting to me, machine code) did me any harm. And I found structured programming really quite easy to do – in OPAL and CORAL.

University saw FORTRAN, C, a functional programming language called SASL, Pascal, amongst others.

So, even before joining IBM, I’d seen a fair few programming languages.

---

This isn’t going to be a taxonomy of programming languages, you’ll be pleased to know.

Here’s an interesting one: A debugger for the Amstrad CPC–464 (I learnt Z80 Assembler on) allowed you to write macros in Forth. I loved it. “What a weird (reverse-Polish) language” you might say. But, and here’s the reason for mentioning it, it was the language that was available to do the job. And that’s a theme I’ll return to.

Now I’m coming to the point: This post, I think (sometimes I’m not so sure), came from conversations about programming languages on IBM-MAIN and further reflection on this blog post of mine.

I’d like to talk about why we chose the programming languages we do (or at least why I do).

• People talk about programming language design. And in Dragging REXX Into The 21st Century? I’m hinting at that – through my efforts to gain some elegance.
• Lots of people talk about efficiency – for example whether a good Assembler programmer can outstrip a COBOL compiler. (Yes over a short enough distance.) 🙂
• People have to choose from the palette of languages available. For example LISP for EMACS. Or whatever your installation has made available.

I think I’m in the third category with a nod to the first, and sometimes I’m forced to worry about efficiency. (Some of our tools wouldn’t be feasible without the tuning we’ve given them: CPU Time isn’t the factor, completing sometime before two years after hell freezes over is.) 🙂 So, for example:

• My mainframe programming is a mixture of Assembler and REXX – because the SMF-crunching code we use requires it. (And actually, in recent years, REXX driving GDDM has been a factor.)
• My stand-alone programming on the PC tends to be Python, though AppleScript on the Mac is a must – because it can drive the Evernote API, for example.
• My “web serving”, which is actually with the server and the client on the same (Linux) machine is Apache with PHP. This pumps out javascript (mostly Dojo for the UI elements). This is a fairly natural choice – and is much better than trying to build a GUI in some native language. The PHP code also automates a lot of FTP gets from the mainframe – textual reports (actually Bookmaster script) and graphics.

It’s horses for courses, and I won’t swear I’ve necessarily made the right choices. But each of these have magical superpowers because of the environment they run in and libraries/built-ins they come with.

A while back someone “threatened” me with having to learn SAS. My response was along the lines of “bring it on, as it’ll be about the 20th language I’ll’ve become familiar with.” I guess that makes me a geek. 🙂

Or maybe a nerd. 🙂

I guess it also explains why this blog contains as much programming as it does. I’m thinking of renaming it to [“Mainframe”,“Performance”,“Topics”]. 🙂

---

The discussion in IBM-MAIN and ruminating about Dragging REXX Into The 21st Century? got me thinking about how fit languages are for the future, and REXX in particular.

I’d say “don’t expect the REXX language to evolve”, whether IBM ports [http://www.oorexx.org](Open Object Rexx) to z/OS or not. (I seriously doubt it will, by the way.) But that doesn’t mean it’s a bad language to write new stuff in. It’s worth noting that if you have, for example, ISPF applications to write it’s still a good language to do it in.

One of the strengths that I think also helps is REXX’s ability to embed host commands. So, for example, if somebody wrote an XML DOM Tree parser set of commands REXX could script it. (If this sounds like what javascript does with, for example, getElementById(), it should.) So the complaint on IBM-MAIN that REXX doesn’t do XML could be solved – with a few 🙂 lines of code.

Another is BPXWUNIX – which I’ve mentioned many times in this blog. It allows you to use all the Unix functionality z/OS offers.

---

One thing I’d admit we lack on z/OS is support for some thumpingly good modern programming languages – Python, node.js and (recent) PHP being but three. And I mention this here because of why:

• I’m not so worried about the expressiveness etc of the languages we have. We can program just fine with what we have.
• I’d like to see more commercial packages ported. These have their own prerequisites when it comes to environments and languages. For example MediaWiki is written in PHP.

So there you are, some stray thoughts on programming languages – on a gloriously sunny Sunday afternoon. And if the tone got tetchier (not techier) 🙂 as you go through it’s because I sat down to write in stages after pulling tall stinging nettles out in the garden. (In provably pessimal 🙂 shorts and t-shirt.) And it’s a well-known fact (to those who know it well)[1] 🙂 that everyone in the UK thinks their garden’s stinging nettles are the most potent in the world. 🙂

(Originally posted 2013-06-24.)

I’ve been fortunate enough to speak at two conferences in the past month:

• UKCMG Annual Conference – in London in May
• System z Technical University – just outside Munich in June

I presented three different sets of material, with very little overlap:

• Time For Dime
• Life And Times Of An Address Space
• DB2 Data Sharing Performance

The first two were given at both conferences and the third one only in Munich.

If you click on the links you’ll see I’ve uploaded them all to Slideshare. I considered it only fair to wait until after the Munich conference to upload them.

---

I thoroughly enjoyed both conferences. But then I usually do: It’s my prime method of learning (other than by doing, and by data analysis) and I’m always pleased to run into old friends and new. (As a firm believer in “strangers are just friends we haven’t met yet” there were a fair number of those, too.)

It was particularly good to see so many people from Hursley – both CICS and Websphere MQ. As well as the usual other Development labs.

If there were to be only one new feature I spotted it would be the support for Variable Blocked Spanned (VBS) records with REXX’s EXECIO in z/OS 2.1 TSO. Think “SMF processing with REXX”. In my residency this autumn I expect to have a chance to play with this.

---

I’m writing this using Byword on an iPad Mini, using a Logitech external keyboard – on a flight to South Africa. It’s proving to be a good combination, especially as Byword has MultiMarkdown support (a set of extensions to Markdown) and Evernote supported. It’ll probably be my new writing rig for a while.

Of course it helps if you put the iPad Mini in the right way round. 🙂

---

And a week has passed since I wrote this. 😦 It’s been a busy week that’s really been an interesting test of the idea of “treating an address space as a black box”. (And so much more besides.) Perhaps I’ll write about it some time.

(Originally posted 2013-06-07.)

I like REXX but sometimes it leaves a little to be desired. This post is about a technique for dealing with some of the issues. I present it in the hope some of you will find it worth building on, or using directly.

Note: I’m talking about Classic REXX and not Open Object REXX.

List Comprehensions are widespread in modern programming languages – because they express concisely otherwise verbose concepts – such as looping.

Here’s an example from javascript:

var numbers = [1, 4, 9];
var roots = numbers.map(Math.sqrt);
/* roots is now [1, 2, 3], numbers is still [1, 4, 9] */

It’s taken from here which is a good description of javascript’s support for arrays.

Essentially it applies the square root function (Math.sqrt) to each element of the array numbers, using the map method. Even though it processes every element there’s no loop in sight. This, to me, is quite elegant and very maintainable. It gets rid of a lot of looping cruft that adds no value.

My Challenge

I have a lot of REXX code – essential to fetch data from the performance databases I build and turn it into graphs and tabular reports. Much of this code iterates over stem variables (similar to arrays – for the non-REXX reader) or character strings that are tokens separated by spaces (blanks).

An example of a blank-delimited token string is:

address_spaces="CICSIP01 CICSIP02 CICSPA CICXYZ DB1ADBM1 MQ1AMSTR MQ1ACHIN"

It would be really nice when processing such a string – perhaps to pick up all the tokens beginning “CICS” – to be able to do it simply. Perhaps an incantation like:

cics_regions=filter("find","CICS",address_spaces)

In this example the filter routine applies the find routine to each token in the string, with a parameter "CICS" (the search argument).

And not a loop in sight.

My Experiment

I implemented versions of map, filter and reduce. I’ll talk about how but first here’s what they do:

Function	Purpose
map	Applies a routine to each element
filter	Creates a subset of the string with each element being kept or discarded based on the routine’s return value (1 to keep and 0 to throw the item away)
reduce	Produce a result based on an initial value and applying a routine to each element.

Here’s a simple version of filter:

filter: procedure 
parse arg f,p1,p2,p3,list 
if list="" then do 
  parse arg f,p1,p2,list 
  if list="" then do 
    parse arg f,p1,list 
    if list="" then do 
      funstem="keepit="f"(" 
    end 
    else do 
      funstem="keepit="f"("p1"," 
    end 
  end 
  else do 
    funstem="keepit="f"("p1","p2"," 
  end 
end 
else do 
  funstem="keepit="f"("p1","p2","p3","
end 
outlist="" 
do forever 
  parse value list with item list 
  interpret funstem""item")" 
  if keepit=1 then do 
    if outlist="" then do 
      outlist=item 
    end 
    else do 
      outlist=outlist item
    end 
  end 
  if list="" then leave 
end 
return outlist 

Variable “list” is the input space-separated list. “outlist” is the output list that filter builds – in the same space-separated list format.

Much of this is in fact parameter handling: The p1, p2, p3 optional parameters need checking for. But the “heavy lifting” comes in three parts:

• Breaking the string into tokens (or items, if you prefer).

• Using interpret to invoke the filter function (named in variable f) against each token.

• Checking the value of the keepit variable on return from the filter function:

  If it’s 1 then keep the item. If not then remove it from the list.

I also wrote a filter called “grepFilter” (amongst others). Recall the example above where I wanted to find the string “CICS” at the beginning of a token. That could’ve been done with a filter that checked for pos("CICS",item)=1. That’s obviously a very simple case. grepFilter, as the name suggests, uses grep against each token. It worked nicely (though I suggest it fails my long-standing “minimise the transitions between REXX and Unix through BPXWUNIX” test).

And then I got playing with examples, including “pipelining” – from, say, map to filter to reduce – such as:

say reduce("sum",0,filter("gt",8,map("timesit",2,"1 2 3 4 5 6")))

Issues

There are a number of issues with this approach:

• You’ll notice the function name (first parameter in the filter example above) is in fact a character string.

  It’s not a function reference as other languages would see it. REXX doesn’t have a first class function data type. Suppose you didn’t have a procedure of that name in your code: You’d get some weird error messages at run time. And while you can pass around character strings all you want the semantics are different from passing around function references.

• The vital piece of REXX that makes this technique possible is the interpret instruction.

  It’s very powerful but comes at a bit of a cost: When the REXX interpreter starts it tokenises the REXX exec – for performance reasons. It can’t tokenise the string passed to interpret. So performance could suffer. For my use cases most of the time (and CPU time) is spent in commands (scripted by REXX) rather than in the REXX code itself. (I also think the process of mapping a function to a list suffers less than the average REXX instruction if run through interpret.

• The requirement to write, for example

  say reduce("sum",0,filter("gt",8,map("timesit",2,"1 2 3 4 5 6")))

  rather than

  say "1 2 3 4 5 6".map("timesit",2).filter("gt",8).reduce("sum",0)

  is inelegant. Fixing this would require subverting a major portion of what REXX is. And that’s not what I’m trying to do.

• The need to apply a function to each item – particularly in the filter case – can be overkill.

  In my Production code I can write

  filter("item>8","1 2 4 8 16 32")

  as I check the first parameter for characters such as “>” and “=”. So no filtering function required.

• REXX doesn’t have anonymous functions and I can’t think of a way to simulate them. Can you? If you look at the linked Wikipedia entry it shows how expressive they can be.

These are worth thinking about but not – I would submit – show stoppers. They just require care in using these techniques and sensible expectations.

Conclusions

It’s perfectly possible to do some modern things in REXX – if you work at it. And this post has been the result of experimentation. Experimentation which I’m going to use directly in some of my programs. (In fact I’ve taken the prototype code and extended it for Production. I’ve kept it “simple” here.)

I’d note that “CMS Pipelines” would do some of this – but not all. And in any case most people don’t have CMS Pipelines – whether on VM or ported to TSO. (TSO is my case, but mostly in batch.)

I don’t believe “Classic” REXX to be under active development so asking for new features is probably a waste of time. Hence my tack of simulating them, and living with the limitations of the simulation: It still makes for clearer, more maintainable code.

Care to try to simulate other modern language features? Lambda or Currying would be pretty similar.

Of course if I had kept my blinkers on then I wouldn’t know about all these programming concepts and wouldn’t be trying to apply them to REXX. But where’s the fun in that?

(Originally posted 2013-06-01.)

Over the years I’ve written emails with data collection requirements dozens of times, with varying degrees of clarity. It would be better, wouldn’t it, to write it once. I don’t think I can get out of the business of writing such emails entirely but here’s a goodly chunk of it.

Another thing that struck me is that the value of some types of data has increased enormously over time. So some data that might’ve been in the “I don’t mind if you don’t send it” category has been elevated to “things will be much better if you do”.

I’d like to articulate that more clearly.

Bare Minimum

I always want SMF Types 70 through to 79, whatever you have.

I put it this loosely because I have almost 100% success with getting the data I really need, because the customer’s already collecting it. That has the distinct advantage of allowing me to ask for historical data, whether for longitudinal studies or just to capture a “problem” period.

There’s rarely been a study where I didn’t want RMF data. (Occasionally I’ve only wanted DB2 Accounting Trace.)

I like to have this data for a few days, but sometimes that isn’t possible. What really freaks my code out is having just a few intervals to work with.

Another question is “for which systems?” That’s a more difficult question to answer. Certainly I want all the major systems in the relevant sysplex. Ideally *all the systems in the sysplex and even all the systems on the machines involved. But that’s usually not realistic. You’ve probably guessed already that the nearer to that ideal the better the insights (probably).

Strongly Enhancing

As you’ll’ve seen from this blog Type 30 Subtypes 2 and 3 Interval records are of increasing value to me: I recently ran some 15 month old data through the latest level of my code and was gratified at how much more insight I gained into how the customer’s systems worked.

A flavour of this is described in posts like Another Usage Of Usage Information.

So this data has definitely moved from the category of “I can just break down CPU usage a little further” to “it will make a large difference if you send it”.

Here the systems and time ranges I’d prefer to see data from can be much less: I probably don’t need to see this data from the Sysprogs’ sandpit.

Nice To Have

With most studies I can get by without the WLM Service Definition but it helps in certain circumstances (as I mentioned in Playing Spot The Difference With WLM Service Definitions.)

I’m OK with either the WLM ISPF TLIB or the XML version (as mentioned in that post).

If I want to take disk performance down to below the volume level SMF 42–6 Data Set Performance records are a must. It’s also the case you can learn an awful lot about DB2 table space and index space fragments from 42–6, there being a well-known naming convention for DB2 data sets.

Specialist Subjects

The above is common to most studies. The following deals with more common specialist needs.

DB2

Most of the time I’m seeking to explain one of two things about DB2: * Where the CPU is going. * Where the time is going. In both cases I need DB2 Accounting Trace (SMF 101). The quality of this is variable. For example, to get CPU down to the Package (Program) level I need Trace Classes 7 and 8, in addition to the usual 1, 2 and 3. (Sometimes even 2 and 3 aren’t on.)

It’s quite likely this isn’t data that in its full glory is being collected all the time. So it’s a 30% chance of getting this retrospectively.

Sometimes I’m keen to understand the DB2 subsystem, which is where Statistics Trace comes in. The default statistics interval (STATIME) used to be a horrendous 30 minutes. Now it’s much lower so I’m pleased that issue has gone away. I ask for Trace Classes 1,3,4,5,6,8,10 which result in SMF 100 and 102 records. (I don’t ask for Performance Trace which also results in 102 records, albeit different ones.)

Again the questions of “for which subsystems?” and “when for?” come into play. That’s where negotiation is important:

• It’s a lot of data to send.
• Some installations deem it too expensive to collect on a continual basis.

I don’t disagree with either of those.

CICS

Here Statistics Trace (SMF 110) is really useful – especially if you have a sensible statistics interval.

For application response time and CPU breakdown to the transaction level Monitor Trace is the thing. Again this is the sort of thing customers don’t keep on a regular basis – or for that many regions. It’s also quite prone to breakage: Some customers remove fields from the record with a customised Monitor Control Table (MCT).

I try to glean what I can from SMF 30 about CICS – as numerous blog posts have pointed out – because I can get it for many more CICS regions than the CICS-specific data would furnish.

Batch

Batch is the area that takes in a widest range of data sources, the most fundamental of which are SMF 30 Subtypes 4 (Step-End) and 5 (Job-End).

I’ve already mentioned DB2 Accounting Trace and it’s most of what you need for understanding the timings of DB2 jobs.

For VSAM data sets SMF 62 is OPEN and 64 is CLOSE. For non-VSAM SMF 14 is Reads and 15 is Writes.

For DFSORT SMF 16 is really handy and even better if SMF=FULL is in effect. (Often it isn’t but I generally wouldn’t stop data collection to fix that.)

MQ

I only occasionally look at Websphere MQ, though I’d like to do much more with it. I don’t think many people are familiar with the data. Statistics Trace (analogous to DB2’s but different) is SMF 115. Accounting Trace – which deals with applications – is SMF 116.

If I want to see what connects to MQ the Usage Information in SMF 30 is generally enough – though it doesn’t tell me much about work coming in through the CHIN address space. For that I really do need Accounting Trace. (An analogous information can be made about remote access to DB2.)

Getting Data To Me

This is usually OK but it’s worth reminding people of a few simple rules:

1. Always use IFASMFDP (and / or IFASMFDL) to move SMF records around. (Using IEBGENER will probably break them.)
2. TERSE the data using AMATERSE (or perhaps TRSMAIN). This works fine for both SMF data and ISPF TLIBs (and I suppose partitioned data sets in general).
3. FTP the data BINARY to ECUREP.
4. Make sure you send the data to the right directory in ECUREP’s file store. The standard encoding of the PMR number helps a number of IBM systems (such as RETAIN) to work swiftly and effectively. I’ll give you a PMR number on my queue (UZPMOS) or we can use another one.

That seems like a lot of rules but most of it should be familiar to anyone who’s ever sent in documentation in support of a PMR. (Only Rule 1 is new.) If you have access to the PMR text – and quite a few customers do – it should also enable you to track the data inbound.

In Conclusion

Realistically people might not have all the data I want and so there’s a process of negotiation, mainly between timeliness and retrospectiveness versus quality. Clarifying that trade off would be helpful, which is why I like to run a Data Collection Kick-Off call. Ideally that call would be face to face, but I’m less insistent on that if distance makes it difficult.

At the end of the day I’m quite flexible and do what I can with whatever data I get. Of course you can’t magic missing data out of thin air, and can only occasionally repair it.

What I hope is that data collection is not overly burdensome and doesn’t cause stress to the customer. I also like to think that when they’ve sent the data in they can relax and more or less forget about it until “showtime”. 🙂

I also hope customers understand why they’ve been asked for the data they have. And that’s part of the point of this post, the rest being articulating what I need.

(Originally posted 2013-05-26.)

I have REXX code that can be called directly by TSO (in DD SYSTSIN data) or else by another REXX function. I want it to behave differently in each case:

• If called directly from TSO I want it to print something.
• If called from another function I want it to return some values to the calling function.

So I thought about how to do this. The answer’s quite simple: Use the parse source command and examine the second word returned.

Here’s a simple example, the function myfunc.

/* REXX myfunc */
interpret "x=1;y=2" 
parse source . envt . 
if envt="COMMAND" then do 
  /* Called from top level command */
    say x y 
end 
else do 
  /* Called from procedure or function */
  return x y 
end 

The interpret command is a fancy way of assigning two variables (and really a leftover from another test). It works but you would normally code

x=1
y=2

The parse source command returns a number of words but it’s the second one that is of interest – and is saved in the variable envt.

The following lines test whether envt has the value “COMMAND” or not. If so the function’s been called directly from TSO. If not it hasn’t. In the one case the variable values are printed. In the other they’re returned to the calling routine.

The calling routine might call have a line similar to

parse value myfunc() with x y

Which would unpack the two variables.

(The original interpret "x=1;y=2" might look silly but interpret "x='1 first';y='2 second'" doesn’t – as a way of passing strings with arbitrary spaces in them back from a routine.)

This is a simplified version of something I want to do in my own code. There might be a few people who will find this useful so why keep it to myself? 🙂 (It’s probably a standard technique nobody taught me.) 🙂