Memories of Pipes

(Originally posted 2007-11-25.)

!Somehow I seem to have end up writing a “Memories of…” series of blog posts. That wasn’t the intention but a set of threads on IBM-MAIN Listserver got me to thinking about these nice venerable technologies – VIO, Hiperbatch, Batch LSR and Pipes.

By couching these posts in terms of “memories of” it sounds like they’re perhaps obsolete. With the possible exception of Hiperbatch that probably isn’t true. (And the only thing really wrong with Hiperbatch is its non-support of Extended Format VSAM and Sequential data sets.)

So, back to the topic – BatchPipes/MVS, usually shortened to Pipes…

The concept of record-level interlock really wasn’t new, even at the time… Unix had had pipes for at least 15 years before that, probably 20. In 1990, however, there was a good reason to introduce it to MVS/ESA (as it then was called)… A big customer wanted it. Pipes was born as a result of an exec-level challenge from a specific customer in the USA.

The idea is quite simple… Pipe individual records from a writer job to a reader job – with minimal changes to the application. (In this case “minimal changes” meant changing the DD card to point to your Pipes subsystem.) But I’ve already written about this.

So, when I was in Poughkeepsie for the second burst of writing the “Storage Configuration Guidelines” Redbooks in November 1990 my new-found friend Ted Blank (who was to become a very good family friend a little later on) told me about Pipes. It was more or less the same conversation that led to the “Parallel Sysplex Batch Performance” Redbook (SG24-2557) because it ranged widely onto more general aspects of batch performance.

Pipes was readied for market through until 1993 or so. At that time in IBM a “new entrepreneurial spirit” was abroad in IBM. Rather than make Pipes a MVS/ESA component it was decided to market it as a specific product. (Personally I think this was a mistake – in terms of aceptance and ultimately customers’ batch windows). So the idea was to offer a service to analyse customer data and then lead them into implementing this chargeable product. The data analysis though was more or less restricted to finding “one writer one reader” patterns in the SMF data. True “engineering in” (which is what is called for) happened after this initial screening – if it happened at all.

In parallel (if you’ll pardon the pun) we were starting to build the PMIO Batch Window offering. Our whole premise was to engineer whatever it took into workloads, acknowledging the value of rescheduling to run in parallel, breaking up jobs, and “pacing”. All things you’d need to really get value out of Pipes. So we, in the PMIO team, were fellow travellers seeking to make Pipes successful and build a good Batch Window Tuning practice. (In the middle of this my manager was offered the opportunity to act as an agent for Pipes in Europe. He declined that offer. He’s no longer in IBM, either. Life could’ve been different.) 🙂

I had a great time in the 1990’s “chalking and talking” on Pipes. It taught me I could take a complex topic like Pipes and keep it in my head and chalk and talk it. Who needs foils? 🙂 And sorry if you were a victim of my extemporisation on Pipes in that era. 🙂

What was also nice was when DFSORT automated the detection of when EXCP wasn’t appropriate for sequential I/O…

Not only Pipes but also Extended Format sequential. This means striped and compressed data sets.

In the Summer of 1997 DFSORT Release 13 came out with this support. (Perhaps I should do another piece called “Memories of DFSORT”.)

At the same time – Summer of 1997 – the Pipes team teamed up with BMC, incorporating the latter’s Data Accelerator and Batch Accelerator into SmartBatch. Also CMS Pipelines became available (in the main) as a set of “fittings” called BatchPipeWorks. (You specify them on the file DD as a parameter string to the subsystem. Perhaps I should blog on this as well.) And also BatchPipePlex – which routes pipes through the coupling facility.Neither BatchPipes/MVS nor Smartbatch sold well. But I still think the Pipes approach remains valid and valuable.

Now today we have BatchPipes/MVS Version 2 Release 1 – which comprises the original Pipes function, BatchPipeWorks and BatchPipePlex. (BTW folks that’s the only way to use “comprise” in sentence.) 🙂 I also see cases where products consider prereq’ing Pipes – or at least offer support as an option.

And I believe I can STILL extemporise on Pipes. So if you need me to (and preferably if you’re on my patch) please get in touch.

So it sounds like I’ve given myself two more topics to blog on:

  • Memories of DFSORT R.13
  • BatchPipeWorks

The latter would require me to fire up Pipes on some system or other again. I’m looking forward to playing. 🙂

Not Boris Johnson but a Chat Show In Secondlife?

(Originally posted 2007-11-22.)

Thanks to Kevin Aires for pointing this out to me…

“Boris in Wonderland” is a chat show in Secondlife – hosted by one Boris Frampton of IBM. Go here for the first episode.

Andy Stanford-Clark (Ginger Mandelbrot in Secondlife) is interviewed about his applications and creations.

The Monty Python quote “look out there are Llamas” is relevant here – but you’ll just have to roll the video to find out why. 🙂

One thing you might notice is the hand gestures when the host and his guest are talking. These “speech gestures” are standard now that Secondlife has Voice.

Not much mainframe relevance, I guess, but one day we’ll all have to make this stuff work – and perform. 🙂 And hopefully lots of people will be wanting to connect virtual worlds to CICS and DB2 and Websphere and MQ and…

Memories of Batch LSR

(Originally posted 2007-11-18.)

Another trip down memory lane – prompted by the thread on IBM-MAIN of Batch LSR vs Hiperbatch.

Batch LSR started out as a prototype by Paul Dorn of the IBM Washington Systems Center. He was contributing to a book on writing a subsystem but also trying to solve a problem with Batch VSAM tuning. So Batch LSR started out as an example of a subsystem.

(I don’t recall a rash of subsystems written by users or vendors after that book was published. 😦 But BatchPipes/MVS would later be built as one.)

Now to the problem Batch LSR (hereafter referred to as BLSR was designed to solve, back in the 1980s…

If a batch job accessed a VSAM file it would almost always use VSAM NSR, which was (or rather could be) optimised for sequential access. If, however, the access was random[1] NSR would be pretty hopeless. To use the random-oriented VSAM LSR would require the program to create its own VSAM LSR buffer pool(s) using the BLDVRP macro. Yes, that’s right, Assembler. 🙂 Most batch jobs were never going to be written that way.

So Paul wrote (and Poughkeepsie rewrote) the subsystem. Here’s what it does…

The subsystem creates the VSAM LSR buffer pools for you. And it allows you to specify a number of parameters, such as the location of the buffer pools (above or below the 24-bit line) and their size. It’s very easy to set up and easy to code the JCL changes needed to use it. (In a large number of places in our VSAM-based (SLR) analysis code we’ve done this.)

Here’s a case Roger Fowler and I worked on in the mid-1990s:

A UK customer had a batch job that did 2 million I/Os to an 850 CI data set. Roger said “let’s turn on BLSR”. They did and the job went down to 0.5 million I/Os. So I said “let’s turn on the Deferred Write option”. Roger said “the what?” 🙂 Anyhow we turned it on and the job went down to around 1,000 I/Os. From 2 million down to 1 thousand is pretty good, I’d say. 🙂

There was a tool called BLSRAID which was SAS-based. We didn’t have SAS so we wrote our own VSAM analysis code in 1993. Another piece in the jigsaw that is the PMIO toolset. (Roger and I used this report to make that saving.)

Another tool – for when you have enabled BLSR for a data set / job – is the User F61 GTF trace. This fed into a popular tool – VLBPAA – but you wouldn’t want to run the trace for all that long. In the back of the SG24-2557 “Parallel Sysplex Batch Performance” Red Book I wrote an appendix on playing with this trace. You can use this trace (and could use VLBPAA) to model the effects of big buffer pools.

In principle, as my and Roger’s example showed, you could fit the whole data set into memory. In fact we recommended BUFND=1024 to this customer. Slightly wasteful but a nice round number. (1024 4KB buffers is, of course, 4MB.)

There were many cases down the years where BLSR was a good fit. But quite often we made the recommendation to stick with NSR and tune for sequential access instead. The basic VSAM instrumentation – SMF 62 and 64 – would lead us to robust conclusions. My good friend Dave Betten (now Performance Lead for DFSORT) put together the Type 30, 62 and 64 records in one SLR table. And we regularly used the SMF 42-6 records (designed by Jeff Berger and eulogised by our good friend John Burg) to round out the VSAM data set picture.

Of course in 1997, with OS/390 Release 4 DFSMS, System-Managed Buffering (SMB) came along. This made it much easier to optimise for sequential or random access, using DFSMS constructs. I’m not sure how widely this is used – but it’s worth taking a look at.

Of course DB2, suitably managed, has many I/O strategies. But for VSAM you have to do it yourself. And the Batch LSR Subsystem made it possible for “random” VSAM I/O.

[1] When I say “random” I really mean “direct”. I’m not sure I believe in randomness, particularly in data processing. The point is that there is “dotting about” rather than sequential access.

DYNDISP and DDF?

(Originally posted 2007-11-15.)

To whoever it is in Denmark that’s repeatedly hitting my blog today with a search for “dyndisp” and “ddf” I’d love to know what it is that is causing you to search for those two terms together. As you’re on my (day job) patch perhaps I should be talking to you directly.

So feel free to contact me by clicking on this link. (I’m hoping the “mailto:” URL form works on your system.)

And, yes, I am paying attention to the search traffic that comes to my blog. Big Brother? I hope you don’t think so.

z/OS R.9 RMF Parallel Sysplex New Fields

(Originally posted 2007-11-12.)

I’ve just re-read the XCF (74-2) and CF (74-4) sections of the z/OS Release 9 SMF Manual. There are some nice things in there…

XCF (74-2)

  • Number of buffer 1KB blocks by path (R742PUSE). This should tell you whether memory waste by over-specifying CLASSLEN is a problem. And some of the other scenarios.
  • Member job name (R742MJOB). I’d like to believe this would tell us which IRLM a given member is, for example.

Coupling Facility (74-4)

  • Whether this CF LPAR is using Dynamic Dispatch (R744FFLG bit 3)
  • How many shared processors (R744FPSN) and how many dedicated processors (R744FPDN) this CF LPAR has.
  • Structure Execution Time (R744SETM) which requires CFLEVEL 15. Useful for Coupling Facility capacity planning.
  • Whether this processor is dedicated (R744PTYP) and what its weight is if it’s shared (R744PWGT).

I haven’t actually looked at any sample R.9 data yet. I’ll have to put that to rights and get back to you when I see some of these numbers in action. But I do think they extend the data model rather nicely, particularly the CF CPU stuff.

Has anyone seen this data in action yet?

On Further Investigation…

I have found a set of 1.9 data. And here’s what I can immediately confirm…

R742MJOB is a very usable job name. Here’s an example…

From ERBSHOW…

#38:   +0000:  E2E8E2C2 40404040 C9E7C3D3 D6F0F0F4  *SYSB    IXCLO004
       +0010:  D4F8F040 40404040 40404040 40404040  *M80             
       +0020:  00030000 0000001A 00000016 00000000  *                
       +0030:  C7D9E240 40404040                    *GRS

Whereas before we had an anonymous member (M80) and an anonymous lock structure (IXCL0004) we now know that M80 is SYSB’s GRS address space and therefore that IXCL0004 is GRS Star. Other Lock Structure exploiters similarly fall into place. I think this will be more interesting for eg IRLM.

Memories of Hiperbatch

(Originally posted 2007-11-11.)

It’s nice to see a flurry of activity in IBM-MAIN about Hiperbatch. And it’s more for the emotional reason of reminiscence than for any stunning insights that I’m blogging about it…

Back in 1988 I ran a technical project in my then customer, Lloyds Bank, to evaluate Data In Memory (DIM). It was a fun project with a wide range of workloads on multiple machines, including the then-new DB2. So we tried out all the analysis tools and even did a bit of Roll Your Own…

Through the IBM internal FORUMs I met another IBM Systems Engineer, John O’Connell, doing a similar thing for his customer, Pratt and Whitney in Connecticut. And he wrote some SAS code to evaluate VIO to Expanded Storage. We shared this code with Lloyds Bank. (I ran into him at several conferences. And later I discovered he’d left IBM and, I think, gone to work for a customer. Are you out there, John?)

And I wrote a presentation on VIO to Expanded Storage (called VIOTOES – which sounds funny if you pronounce it right). 🙂 There were two key elements in this presentation:

  • How to evaluate the opportunities for VIO. And what happened if you fiddled with the VIOMAXSIZE tuning knob (the maximum (primary plus 15 secondary extents) temporary data set size that would end up as VIO (and hence potentially in Expanded Storage)).
  • How to use the then new-fangled DFSMS ACS routines to control which data sets were even considered for VIO – and all sorts of other funky tricks with DFSMS.

This presentation went down well with IBMers and customers alike and could be considered my first conference presentation.

The point of the above is to set the scene for Hiperbatch…

So, we announce Hiperbatch as part of MVS/ESA 3.1.3. (Funny how we went 3.1.0, 3.1.0e but not 3.1.1 or 3.1.2.) And it had a hardware prerequisite of a 3090 S processor (because of the MVPG instruction – even though technically one COULD move pages between Central and Expanded Storage using ordinary instructions if we’d chosen to implement it that way.) The important thing is that we wanted an exclusive by tying this super duper new facility to a brand new processor.

And because of this my fourth line manager at the time decided we all had to run Hiperbatch Aid (HBAID) studies. At this point I learnt I was not a “team player”. Well duh. 🙂 I declared I wasn’t going to do it because we’d already crawled all over Lloyds Bank looking for genuine DIM benefit. And there wasn’t likely to be any from Hiperbatch. With that defence the requirement to run HBAID was waived.

You’d think from that I’d a downer on Hiperbatch and HBAID, wouldn’t you? 🙂 Far from it actually…

I did enjoy running HBAID in one or two other customers and I did get quite creative with Hiperbatch. And that was the trick, in my opinion – getting creative. And that realisation led on to other things…

One of the really nice things about HBAID was that it Gantt’ed out data set lives. And from that you could glimpse where other techniques might be useful (such as VIO and OUTFIL). So I invented[1] a technique called LOADS which stood for (for those of you averse to “flyovers”) 🙂 “Life Of A Data Set”. These signatures were really rather handy. Here’s a slightly later example…

A “standard” BatchPipes/MVS pipe candidate would be a sequential writer job followed by a sequential reader job for the same data set. There are, of course, lots of scenarios where Pipes is useful, each with their own signatures (LOADS).

And it was my good friend Ted Blank who told me about Pipes in late 1990. (And he had been involved in HBAID.)

Ted also encouraged me to start writing a book on Batch Performance. This later became SG24-2557 “Parallel Sysplex Batch Performance” and I believe you can still find it online. (Only this week I referred someone to it as a starter manual for what he wanted to do – but I DO regard it as being somewhat dated.) 😦

And the writing of the book got me into writing batch tools – which generalised what HBAID did and then some. And that’s how I came to be one of the developers of PMIO , the Batch Window analysis toolset / consulting offering. You may have heard of PMIO.

I’m aware of very few installations running Hiperbatch and even fewer running Pipes. 😦 But at least I got something out of it. And we did evolve the “state of the art” as far as batch window was concerned.

As for Hiperbatch’s applicability. I think it’s worth a look at. But there are so many other techniques around that more or less cover the same ground. Though some, like Pipes, cost money. And others are extremely creative to apply. But I don’t think it was ever going to take off in a big way. But that’s OK, given Hiperbatch was built to solve a problem one important customer.

[1]Actually I don’t claim to have invented it, just popularised and generalised it. After all HBAID itself was doing much the same thing. In fact someone once suggested I should patent LOADS but I declined on the “not exactly oriiginal work” basis.