(Originally posted 2007-11-25.)
!Somehow I seem to have end up writing a “Memories of…” series of blog posts. That wasn’t the intention but a set of threads on IBM-MAIN Listserver got me to thinking about these nice venerable technologies – VIO, Hiperbatch, Batch LSR and Pipes.
By couching these posts in terms of “memories of” it sounds like they’re perhaps obsolete. With the possible exception of Hiperbatch that probably isn’t true. (And the only thing really wrong with Hiperbatch is its non-support of Extended Format VSAM and Sequential data sets.)
So, back to the topic – BatchPipes/MVS, usually shortened to Pipes…
The concept of record-level interlock really wasn’t new, even at the time… Unix had had pipes for at least 15 years before that, probably 20. In 1990, however, there was a good reason to introduce it to MVS/ESA (as it then was called)… A big customer wanted it. Pipes was born as a result of an exec-level challenge from a specific customer in the USA.
The idea is quite simple… Pipe individual records from a writer job to a reader job – with minimal changes to the application. (In this case “minimal changes” meant changing the DD card to point to your Pipes subsystem.) But I’ve already written about this.
So, when I was in Poughkeepsie for the second burst of writing the “Storage Configuration Guidelines” Redbooks in November 1990 my new-found friend Ted Blank (who was to become a very good family friend a little later on) told me about Pipes. It was more or less the same conversation that led to the “Parallel Sysplex Batch Performance” Redbook (SG24-2557) because it ranged widely onto more general aspects of batch performance.
Pipes was readied for market through until 1993 or so. At that time in IBM a “new entrepreneurial spirit” was abroad in IBM. Rather than make Pipes a MVS/ESA component it was decided to market it as a specific product. (Personally I think this was a mistake – in terms of aceptance and ultimately customers’ batch windows). So the idea was to offer a service to analyse customer data and then lead them into implementing this chargeable product. The data analysis though was more or less restricted to finding “one writer one reader” patterns in the SMF data. True “engineering in” (which is what is called for) happened after this initial screening – if it happened at all.
In parallel (if you’ll pardon the pun) we were starting to build the PMIO Batch Window offering. Our whole premise was to engineer whatever it took into workloads, acknowledging the value of rescheduling to run in parallel, breaking up jobs, and “pacing”. All things you’d need to really get value out of Pipes. So we, in the PMIO team, were fellow travellers seeking to make Pipes successful and build a good Batch Window Tuning practice. (In the middle of this my manager was offered the opportunity to act as an agent for Pipes in Europe. He declined that offer. He’s no longer in IBM, either. Life could’ve been different.) 🙂
I had a great time in the 1990’s “chalking and talking” on Pipes. It taught me I could take a complex topic like Pipes and keep it in my head and chalk and talk it. Who needs foils? 🙂 And sorry if you were a victim of my extemporisation on Pipes in that era. 🙂
What was also nice was when DFSORT automated the detection of when EXCP wasn’t appropriate for sequential I/O…
Not only Pipes but also Extended Format sequential. This means striped and compressed data sets.
In the Summer of 1997 DFSORT Release 13 came out with this support. (Perhaps I should do another piece called “Memories of DFSORT”.)
At the same time – Summer of 1997 – the Pipes team teamed up with BMC, incorporating the latter’s Data Accelerator and Batch Accelerator into SmartBatch. Also CMS Pipelines became available (in the main) as a set of “fittings” called BatchPipeWorks. (You specify them on the file DD as a parameter string to the subsystem. Perhaps I should blog on this as well.) And also BatchPipePlex – which routes pipes through the coupling facility.Neither BatchPipes/MVS nor Smartbatch sold well. But I still think the Pipes approach remains valid and valuable.
Now today we have BatchPipes/MVS Version 2 Release 1 – which comprises the original Pipes function, BatchPipeWorks and BatchPipePlex. (BTW folks that’s the only way to use “comprise” in sentence.) 🙂 I also see cases where products consider prereq’ing Pipes – or at least offer support as an option.
And I believe I can STILL extemporise on Pipes. So if you need me to (and preferably if you’re on my patch) please get in touch.
So it sounds like I’ve given myself two more topics to blog on:
- Memories of DFSORT R.13
- BatchPipeWorks
The latter would require me to fire up Pipes on some system or other again. I’m looking forward to playing. 🙂
Mainframe, Performance, Topics 