Where My Searches Come From

(Originally posted 2007-12-06.)

This is really just a test of some Firefox Extension code of mine… but it’s kind of interesting in its own right…

The following is a table containing the country-level summary of Search URLs that somehow get a reader to my blog on any given day. (I have another table ranking search terms.)

Most days the USA and India are at the top of the list. And there’s usually a smattering of other countries represented, mainly in Europe.

What’s interesting is that readers from India seem to be mainly interested in JCL, utilities and DFSORT usage. Whereas those from the USA, Europe, China and Japan appear to be mainly interested in Performance- and Architecture-related topics.

I can’t prove this to you but the picture outlined above stays pretty much static.

And then there’re the searches explicitly on names of women. 🙂

Country Searches % Country Searches %
USA 8 33 DE 1 4
IN 6 25 IT 1 4
Unknown 4 16 LT 1 4
FR 2 8 UK 1 4

A Great Song For A Great Cause

(Originally posted 2007-11-30.)

This is a great song for a great cause…

I saw Queen + Paul Rodgers play this in 2005 in Hyde Park. Actually Paul did nothing – Roger Taylor sang it. 🙂

But now they’ve worked on it as part of their current studio sessions and have rushed it out – for free.

So download, play it, donate and tell your friends….

Get it from here and read all about it.

Now I’ve Downloaded And Listened To It

I downloaded it and listened to it a few times. As with all Queen-related things it grew on me after a few listenings. I would say it’s slower than the original 2005 version, which I wish it wasn’t. But on the other hand there’s some excellent guitar work from Brian May. No surprise there. 🙂

All three of them – Paul Rodgers, Brian May and Roger Taylor – shared the vocals. There’s some dispute in our household because I prefer Brian’s and especially Roger’s voices to Paul’s. The rest of the family don’t think much of their voices.

Roll on the album. 🙂

(And here‘s the song at its original speed.)

Memories of DFSORT OUTFIL

(Originally posted 2007-11-25.)

In September 1997 DFSORT Release 13 was shipped (to coincide with the release of OS/390 Release 4). It took a nice idea from Syncsort and extended it.

In case you didn’t know OUTFIL allows you to read an input data set (and perhaps sort it) and write to multiple output files from the resulting records – perhaps selecting subsets of the records and reformatting them (and differently to each output file). All in a single pass over the input data.

Perhaps people really still don’t know about OUTFIL as, while I get many searches that hit my blog for DFSORT topics, OUTFIL is rarely one of the search terms.

There are three features that were then unique to DFSORT:

  • SPLIT

    This is a “card dealer” function. The most obvious (to me) use was in BatchPipes/MVS “Pipe Balancing”. This is where multiple readers absorb the output of a single writer. (Or the other way around.) In this case DFSORT would write to multiple pipes.

  • FNAMES

    This allows you to use any output DD name you like. Consider the following…

    OUTFIL FILES=(01,02),...

    and

    OUTFIL FILES=(GERMANY,FRANCE),...

    The latter is much preferable to the former. (The former generates DD names of “SORTOF01” and “SORTOF02” while the latter uses DD names of “GERMANY” and “FRANCE”.)

  • SAVE

    (This one is, if I remember correctly, my one contribution to the release. I like to make “helpful suggestions” 🙂 to the Development team and Frank (Yaeger) kindly puts some of my ideas into the product.)

    Consider the case where you’ve written multiple subsets of the input records to different output destinations. Suppose you want to write the records that haven’t been written to any destination to one. It can be very complicated (and error-prone) to figure out the right INCLUDE/OMIT parameter string to make this happen. SAVE automatically does that for you:

    OUTFIL FNAMES=SPECIALS,INCLUDE=(SMFID,EQ,C'PRD1',AND,...)OUTFIL FNAMES=THEREST,SAVE

    So I really like this one. 🙂

So those are the bare-bones additions to OUTFIL for Release 13. (DFSORT has since added a lot more function to OUTFIL and I would have to assume Syncsort had done the same.)

I had the chance to run a residency that Summer in Poughkeepsie – and it was a lot of fun. A small team of highly skilled people I’m pleased to call friends and I took a version of our batch-driven SMF analysis code and made a mini batch window out of it. (We do have our own mini scheduler and a batch window topology to play with.) So we got to play with DFSORT Release 13 and the (also new) DFSMS System-Managed Buffering and WLM-Managed JES2 initiators and BatchPipes/MVS and had a fine old time of it – running and measuring and tweaking and doing it all again. We got to do a few presentations based on the results of our playing but never got to turn the foils into a Redbook. (I think we had too much fun “playing sysprog” and the like.) 🙂

One feature that proved really useful was DFSORT’s ability to detect when BSAM or QSAM was needed instead of EXCP. The most common cases are BatchPipes/MVS pipes, Extended Format Sequential (Striped and/or Compressed) and SYSOUT data sets.

Note: I didn’t say “QSAM”. So that ruled Hiperbatch out. To use Hiperbatch you have to write your own E15, E32 or E35 exit to close the data set and to reopen it for QSAM. (There was a sample piece of Assembler code to do this in the HBAID manual.)

From my (largely Performance) perspective, though, OUTFIL is all about avoiding repeated reads of the input data set. Our batch window did a fair amount of that because it read SMF data repeatedly. So OUTFIL fitted nicely into our window. (And if we pretended the output data was VB and not VBS we could get away with piping it as well.)

One thing to be clear about – which I soon realised – was that OUTFIL does not replace multiple sorts with a single one – unless the sort keys etc are identical. But you could feed the same records from a DFSORT OUTFIL job through multiple pipes into the appropriate number of DFSORT SORT jobs. n sorts become n+1 jobs. More balls to keep in the air, of course. And if those sorts are big enough they’ll compete for memory. Which is where another Release 13 feature came in handy: Dynamic Hipersorting. This changed DFSORT Hipersorting from asking MVS (via STGTEST SYSEVENT) how much storage it could have once at the beginning of the sort for sort work to asking the question several times over (as the data was read in). Because of this change Dynamic Hipersorting was much less likely to cause overcommitment of memory – in the multiple concurrent sorts case.

So, to me, DFSORT OUTFIL is yet another of those techniques that you have to “engineer in”. Unless you plan the implementation with the usual diligence nothing will happen. We had a lot of fun finding cases where it would be ideal in customer workloads – when conducting PMIO Batch Window studies. And, like so many other techniques, it’s as valid today as it was 10 years ago.

One final thing: I’ve just remembered that I had another idea that got put into DFSORT Release 13…

The PMIO team worked with DFSORT Development to enhance the SMF 16 record (one per DFSORT invocation) to add a few minor fields – in Release 12. (I even forget what they were.) In Release 13 the record was enhanced to contain input and output data set sections – one per data set. These were very detailed – including the number of records read or written, the access method (including Pipes) and so on. Very nice. To get this additional data you need to run with SMF=FULL. (I’d recommend this anyway.)

Memories of Pipes

(Originally posted 2007-11-25.)

!Somehow I seem to have end up writing a “Memories of…” series of blog posts. That wasn’t the intention but a set of threads on IBM-MAIN Listserver got me to thinking about these nice venerable technologies – VIO, Hiperbatch, Batch LSR and Pipes.

By couching these posts in terms of “memories of” it sounds like they’re perhaps obsolete. With the possible exception of Hiperbatch that probably isn’t true. (And the only thing really wrong with Hiperbatch is its non-support of Extended Format VSAM and Sequential data sets.)

So, back to the topic – BatchPipes/MVS, usually shortened to Pipes…

The concept of record-level interlock really wasn’t new, even at the time… Unix had had pipes for at least 15 years before that, probably 20. In 1990, however, there was a good reason to introduce it to MVS/ESA (as it then was called)… A big customer wanted it. Pipes was born as a result of an exec-level challenge from a specific customer in the USA.

The idea is quite simple… Pipe individual records from a writer job to a reader job – with minimal changes to the application. (In this case “minimal changes” meant changing the DD card to point to your Pipes subsystem.) But I’ve already written about this.

So, when I was in Poughkeepsie for the second burst of writing the “Storage Configuration Guidelines” Redbooks in November 1990 my new-found friend Ted Blank (who was to become a very good family friend a little later on) told me about Pipes. It was more or less the same conversation that led to the “Parallel Sysplex Batch Performance” Redbook (SG24-2557) because it ranged widely onto more general aspects of batch performance.

Pipes was readied for market through until 1993 or so. At that time in IBM a “new entrepreneurial spirit” was abroad in IBM. Rather than make Pipes a MVS/ESA component it was decided to market it as a specific product. (Personally I think this was a mistake – in terms of aceptance and ultimately customers’ batch windows). So the idea was to offer a service to analyse customer data and then lead them into implementing this chargeable product. The data analysis though was more or less restricted to finding “one writer one reader” patterns in the SMF data. True “engineering in” (which is what is called for) happened after this initial screening – if it happened at all.

In parallel (if you’ll pardon the pun) we were starting to build the PMIO Batch Window offering. Our whole premise was to engineer whatever it took into workloads, acknowledging the value of rescheduling to run in parallel, breaking up jobs, and “pacing”. All things you’d need to really get value out of Pipes. So we, in the PMIO team, were fellow travellers seeking to make Pipes successful and build a good Batch Window Tuning practice. (In the middle of this my manager was offered the opportunity to act as an agent for Pipes in Europe. He declined that offer. He’s no longer in IBM, either. Life could’ve been different.) 🙂

I had a great time in the 1990’s “chalking and talking” on Pipes. It taught me I could take a complex topic like Pipes and keep it in my head and chalk and talk it. Who needs foils? 🙂 And sorry if you were a victim of my extemporisation on Pipes in that era. 🙂

What was also nice was when DFSORT automated the detection of when EXCP wasn’t appropriate for sequential I/O…

Not only Pipes but also Extended Format sequential. This means striped and compressed data sets.

In the Summer of 1997 DFSORT Release 13 came out with this support. (Perhaps I should do another piece called “Memories of DFSORT”.)

At the same time – Summer of 1997 – the Pipes team teamed up with BMC, incorporating the latter’s Data Accelerator and Batch Accelerator into SmartBatch. Also CMS Pipelines became available (in the main) as a set of “fittings” called BatchPipeWorks. (You specify them on the file DD as a parameter string to the subsystem. Perhaps I should blog on this as well.) And also BatchPipePlex – which routes pipes through the coupling facility.Neither BatchPipes/MVS nor Smartbatch sold well. But I still think the Pipes approach remains valid and valuable.

Now today we have BatchPipes/MVS Version 2 Release 1 – which comprises the original Pipes function, BatchPipeWorks and BatchPipePlex. (BTW folks that’s the only way to use “comprise” in sentence.) 🙂 I also see cases where products consider prereq’ing Pipes – or at least offer support as an option.

And I believe I can STILL extemporise on Pipes. So if you need me to (and preferably if you’re on my patch) please get in touch.

So it sounds like I’ve given myself two more topics to blog on:

  • Memories of DFSORT R.13
  • BatchPipeWorks

The latter would require me to fire up Pipes on some system or other again. I’m looking forward to playing. 🙂

Not Boris Johnson but a Chat Show In Secondlife?

(Originally posted 2007-11-22.)

Thanks to Kevin Aires for pointing this out to me…

“Boris in Wonderland” is a chat show in Secondlife – hosted by one Boris Frampton of IBM. Go here for the first episode.

Andy Stanford-Clark (Ginger Mandelbrot in Secondlife) is interviewed about his applications and creations.

The Monty Python quote “look out there are Llamas” is relevant here – but you’ll just have to roll the video to find out why. 🙂

One thing you might notice is the hand gestures when the host and his guest are talking. These “speech gestures” are standard now that Secondlife has Voice.

Not much mainframe relevance, I guess, but one day we’ll all have to make this stuff work – and perform. 🙂 And hopefully lots of people will be wanting to connect virtual worlds to CICS and DB2 and Websphere and MQ and…

Memories of Batch LSR

(Originally posted 2007-11-18.)

Another trip down memory lane – prompted by the thread on IBM-MAIN of Batch LSR vs Hiperbatch.

Batch LSR started out as a prototype by Paul Dorn of the IBM Washington Systems Center. He was contributing to a book on writing a subsystem but also trying to solve a problem with Batch VSAM tuning. So Batch LSR started out as an example of a subsystem.

(I don’t recall a rash of subsystems written by users or vendors after that book was published. 😦 But BatchPipes/MVS would later be built as one.)

Now to the problem Batch LSR (hereafter referred to as BLSR was designed to solve, back in the 1980s…

If a batch job accessed a VSAM file it would almost always use VSAM NSR, which was (or rather could be) optimised for sequential access. If, however, the access was random[1] NSR would be pretty hopeless. To use the random-oriented VSAM LSR would require the program to create its own VSAM LSR buffer pool(s) using the BLDVRP macro. Yes, that’s right, Assembler. 🙂 Most batch jobs were never going to be written that way.

So Paul wrote (and Poughkeepsie rewrote) the subsystem. Here’s what it does…

The subsystem creates the VSAM LSR buffer pools for you. And it allows you to specify a number of parameters, such as the location of the buffer pools (above or below the 24-bit line) and their size. It’s very easy to set up and easy to code the JCL changes needed to use it. (In a large number of places in our VSAM-based (SLR) analysis code we’ve done this.)

Here’s a case Roger Fowler and I worked on in the mid-1990s:

A UK customer had a batch job that did 2 million I/Os to an 850 CI data set. Roger said “let’s turn on BLSR”. They did and the job went down to 0.5 million I/Os. So I said “let’s turn on the Deferred Write option”. Roger said “the what?” 🙂 Anyhow we turned it on and the job went down to around 1,000 I/Os. From 2 million down to 1 thousand is pretty good, I’d say. 🙂

There was a tool called BLSRAID which was SAS-based. We didn’t have SAS so we wrote our own VSAM analysis code in 1993. Another piece in the jigsaw that is the PMIO toolset. (Roger and I used this report to make that saving.)

Another tool – for when you have enabled BLSR for a data set / job – is the User F61 GTF trace. This fed into a popular tool – VLBPAA – but you wouldn’t want to run the trace for all that long. In the back of the SG24-2557 “Parallel Sysplex Batch Performance” Red Book I wrote an appendix on playing with this trace. You can use this trace (and could use VLBPAA) to model the effects of big buffer pools.

In principle, as my and Roger’s example showed, you could fit the whole data set into memory. In fact we recommended BUFND=1024 to this customer. Slightly wasteful but a nice round number. (1024 4KB buffers is, of course, 4MB.)

There were many cases down the years where BLSR was a good fit. But quite often we made the recommendation to stick with NSR and tune for sequential access instead. The basic VSAM instrumentation – SMF 62 and 64 – would lead us to robust conclusions. My good friend Dave Betten (now Performance Lead for DFSORT) put together the Type 30, 62 and 64 records in one SLR table. And we regularly used the SMF 42-6 records (designed by Jeff Berger and eulogised by our good friend John Burg) to round out the VSAM data set picture.

Of course in 1997, with OS/390 Release 4 DFSMS, System-Managed Buffering (SMB) came along. This made it much easier to optimise for sequential or random access, using DFSMS constructs. I’m not sure how widely this is used – but it’s worth taking a look at.

Of course DB2, suitably managed, has many I/O strategies. But for VSAM you have to do it yourself. And the Batch LSR Subsystem made it possible for “random” VSAM I/O.

[1] When I say “random” I really mean “direct”. I’m not sure I believe in randomness, particularly in data processing. The point is that there is “dotting about” rather than sequential access.