New Support for BatchPipes/MVS

(Originally posted 2007-10-11.)

As many of you know I’ve been very fond of BatchPipes/MVS (aka “Pipes”) down the years (17 to be precise). So I’m pleased to seeAPAR PK34251: ADDING BATCHPIPES SUBSYS SUPPORT TO TEMPLATE UTILITY describes some new support in the DB2 Load Utility (as driven by the Template utility) which makes it much easier to use with BatchPipes/MVS.

(For reference here’s the BatchPipes For OS/390 Version 2 Release 1 announcement letter.)

I can see a number of scenarios where the ability to load from a pipe would be handy, potentially speeding up the load. (Whether it actually does will depend on all the other factors that govern the Load utility’s speed.) The most usual scenario is an unload step – perhaps using a utility or SQL, some transformation, and then a (re)load. This might or might not be into the same DB2 subsystem, but probably won’t be into the same DB2 table. The unload could be piped into the transformation step. If the transformation step doesn’t involve a sort then all three – the unload, the transformation and the load – could conceivably be done in parallel. (If it does involve a sort then the sort’s input phase would be overlapped with the unload and the sort’s output phase with the load, leaving just the (usually small) intermediate sort merge phase not overlapped.) For what it’s worth DFSORT has been able automatically to detect pipes for input and output – for 10 years now. 🙂 When it detects a pipe it switches to BSAM to process it, rather than using EXCP. (BSAM and QSAM are the only supported access methods for Pipes – as well as for Extended Format Sequential data sets – whether striped, compressed or not.)

The referenced APAR description does a good job, I think, of discussing considerations when using a pipe to load the data from.

And now I’m off to edit the BatchPipes Wikipedia entry. 🙂