(Originally posted 2012-03-22.)

Don’t reach for your dictionary to look up the single word in the title: I just made it up. 🙂 But I hope it’ll make sense as a term once you’ve read this post.

Once in a while I like to post on Social Media. Looking back it seems to be about every 6-9 months. Some recent examples are:

• A Comment on "Now Your Embarrassing/Job-Threatening Facebook Photos Could Haunt You For Seven Years"
• Who Are You And What Have You Done With My Readers? 🙂
• Exiting The Babble Phase?

In this post I want to talk about what I’ll call "communities". You might prefer the term "consituencies". So I’ll use them interchangeably. Take a look at the following graphic. It’s a pair of Venn diagrams. It’s a gross oversimplification but I hope it illustrates a point or two.

Let’s continue but from a different standpoint: Why is it that what I say only sometimes resonates with you? (I’m doing well if most of what I say hits home.) 🙂

This question and the Venn diagrams are related:

You’ll see I’m in all the purple sets and you’re in one of them. And conversely with the green ones: You’re in all of them and I’m in only one.

Each set represents a community. So you might be in my "school friends" constituency – and the converse is probably true. When I talk about z/OS and SMF that probably won’t resonate with you. And when you talk about skateboarding it probably won’t resonate very strongly with me – as I’m not a skateboarder. But each of these topics undoubtedly resonates with several people – because they are in the appropriate community. It’s a happy accident if both "z/OS and SMF" and "skateboarding" resonate with the same person.

Which brings me on to some interesting points about the diagram:

• It doesn’t describe all our contacts – otherwise it would be impossibly crowded.
• It doesn’t – as drawn – admit to the possibility of us being in multiple communities we’re both members of. But that’s just because I chose to keep it simple.
  
• As I mentioned before, it’s possible for communities (topologists would perhaps call them "neighbourhoods"¹) of 2 different people to unexpectedly interest a third person. That’s, of course, very common. When it happens it’s magic, I think.
  

So far so "ho hum": We all know this stuff. And it’s really laying the groundwork for what I really want to talk about: Who we communicate with and why. Hence the "whocasting" neologism.

It’s actually my friend Bill Seubert who prompted me (perhaps unwittingly) to write this post: He talks a lot about interactions between communities, often disjoint or worse. At least that’s my "take home" of some of what he says.

I think it’s a fairly common experience to have someone say "I didn’t understand a word of that last post". I think we all get feedback somewhat along those lines. (I also think we get quite a lot of "I liked the way you put that".) I’m sure it’s not just me and (perhaps defensively) I claim it’s not (always) 🙂 just a complaint about obscurity: I think it’s a fact of life when you have so many constituencies.

If I start thinking about all the constituencies I have it’s quite a long list. Some would be: Family, School Friends, College Friends (and a subset The Pi Collective), IBM Training Stream Friends, Mainframers, Social Media Fellow Conspirators :-), and so on. That’s pretty diverse. And, again, I’m sure that’s not just me.

I think sometimes it’s fun to figure out exactly for whom a message is created. Sometimes it’s only one person. Sometimes it’s a well defined group. Sometimes it’s an ill-defined group. And, my favourite, sometimes it’s just tossed into the void to see who bites / giggles / reacts / whatever. While it might be "narrowcasting" or "broadcasting" it often isn’t either of these. The "whocasting" relates to the game of figuring it out.

Let’s talk about constituencies / communities some more:

For a start you don’t see my communities as sharply as I do. And vice versa. (I’ll leave the debate as to whether a community really has sharp boundaries out of this post, for brevity².) If that’s true then that might making the game of spotting which community someone else is communicating with a whole lot trickier.

What’s very interesting to me is the dynamics between communities:

• Groups of communities can "triangulate" on you. If you were now in a community whose position were opposite to that of a community you grew up in that would be interesting, wouldn’t it? 🙂
  
• If those two communities got to fighting with each other – and I’ve seen it happen to other people – that might be stressful and destructive. I’d sell ringside tickets to that one. 🙂 If one were conciliatory one might exclaim "couldn’t we all just get a bong?"³ 🙂
• If two communities came together through their common member(s) that’d be really good to see.

The fact that these inter-community effects are real and happen quite often reinforces my view that there’s nowhere to hide. All you can do is be yourself and speak with your "authentic voice". This is very much like a good party: There are lots of conversations going on and it’s really rather noisy. You can duck in and out of the conversations and generally nobody much minds if you invite yourself into a conversation. If there is a difference it might be in who can overhear the conversations: I’d say it’s easier to overhear in Social Media than in many parties. We’ve even got tools that help us do that.

So, as I said, there’s really nowhere to hide. But that’s, in my opinion, really very nice. Yes, there are a few awkward moments, but generally it’s good. I was at one point going to call this post "Not Afraid"⁴ but I think I’ve said that already. What might I be afraid of? I suppose criticism and looking like a fool. I think I can learn from the former and it’s perhaps too late for the latter: Living openly means looking like a fool is inevitable and learning is usually the result. Oh, and fear itself. 🙂

To sum up, it’s important in Social Media to consider your constituencies and keep track of which communities you’re in, centred on each contact. And to use this information to cultivate what flows from it. But when I say that, like so many things in Social Media, it’s very easy to over-track and analyse. Perhaps, in this post, I’ve done just that (but without the tracking). Oh well. 🙂

---

¹ The allusion here is that in topology neighbourhoods are always neighbourhoods of an element. (I won’t stretch the analogy to consider the other parts of the definition of a neighbourhood as it’s perhaps not useful here.)

² "For brevity" is about as credible as when a mathematician says "clearly". 🙂

³ An old joke which will make some people giggle, some get annoyed, and some just fail to understand it. Which rather makes my point, doesn’t it?

⁴ I realise a reference to Eminem will get up some noses. If it does I’d invite you to look beyond the (potential) offensiveness and see the deftness with which he operates and to consider that much of his bile is directed at himself. I also realise he isn’t cool and doesn’t make me look cool. 🙂 Anyway, the reference is to this song (lyrics here.) In particular I think it’s the line "Holla if you feel you’ve been down the same road" that resonates with this post.

(Originally posted 2012-03-14.)

It’s not often I write a blog post that’s essentially a link to a web page. But on this occasion I will.

Here’s the link: Packer Advocates the Human Side of Social Business.

I hope you also read Willie Favero’s similar piece: Favero Shares His Secrets for Social Media Success.

I don’t know whether it’s the done thing to point to an article about you. Anyhow, I hope you enjoy it – and find it encouraging. I hope you’re not put off by the style that uses the surname rather than the first name. That’s not my personal style but is in keeping with the magazine.

(Originally posted 2012-03-10.)

I’ve just published this presentation on Slideshare. You can get it from here.

Normally I’d give a presentation at least once before publishing it. Unfortunately the event I was going to present it at earlier this week was cancelled. So I’m experimenting a little by publishing it first. As with all presentations it’ll probably evolve. What I’ve not done before is let it begin the evolution before I present it.

I hope, having seen the slides, you’re more inclined to hear me present it: There are quite a few things that you will either scratch your head at in the slides or will know are going to come alive when I present.

Actually the whole thing’s been an experiment in some ways. For a flavour of this see:

• I Know What You Did Last Summer
• I Know What You Did Last Summer – Abstract
• I Know What You Did Last Summer – Some Structure At Last

Anyhow, it’s been interesting to think and write about some slightly different stuff – and at a less detailed level. It’ll be even more interesting than usual to give the presentation because of this. I’m certainly doing it at IBM System z Technical University in Berlin May 21-25.

(Originally posted 2012-03-08.)

Now with free map :-), this is the concluding part of a four part series on batch parallelisation, with especial focus on cloning.

In previous parts I discussed:

1. Motivation
2. Classification
3. Issues

This part wraps up with thoughts on implementation. I’m going to break it down into:

1. Analysis
2. Making Changes
3. Monitoring

While there probably are iterations of this, this is the essential 1-2-3 sequence within the cycle.

I’m repeating the example from Part 3, partly because I raised some issues in relation to this diagram I want to cover here:

Analysis

Finding good places to use cloning is the same as finding good jobs to tune, with one further consideration: Because cloning is riskier and more difficult to do you’d want to be sure it was the right tuning action.

If you can find an easier tuning action that makes the batch speed up enough for now, while being scalable for the future, do it in preference to cloning. If you’re "future proofing" an application to the extent where other tuning methods aren’t going to do enough then consider cloning.

Modifying this advice only slightly, consider that you might be able to postpone cloning for a year or two. In this case keep a list of jobs that might need to be cloned eventually.

A couple of examples of where cloning might be indicated are:

• Single-task high CPU burning steps
• Database I/O intensive steps

Of course, feasibility of cloning comes into it. I’d view this as the last stage in the analysis process. As I like to pun: "the last thing I’m going to do is ask you to change your program code". While there may be some cases where application¹program change can be avoided, the majority of cases will require code surgery. The cases where surgery isn’t required are where the data can be partitioned and the existing program operates just fine on a subset of the data.

Making Changes

(This whole post is "Implementation" but this is the bit where the real implementation happens.)

Let’s divide this into six pieces, with reference to the diagram above:

• Splitting the transaction file
• Changing the program to expect a subset of the data
• Merging the results
• Refactoring JCL
• Changing the Schedule
• Reducing data contention

Splitting

As noted in Part 3, the transaction file drives the loop: Each cycle round it is triggered by reading a single record from this file. Suppose we wanted to clone "4-up" i.e. to create four identical parallel jobs. There are a number of ways we could do this:

1. Use a "card dealer" like DFSORT’s OUTFIL SPLIT to deal four hands.
2. "Chunk" the file, perhaps with DFSORT’s OUTFIL with STARTREC and ENDREC.
3. Split based on criteria, You could use DFSORT OUTFIL with INCLUDE= or OMIT=. Or else you could use an application program.

There are considerations with all of these:

• The card dealer (1) ensures (practically) equal numbers of records in each transaction file, but there is no sense of logical portitioning. So it could provide balance but at the expense of cross-clone contention.
• Neither 2 nor 3 guarantee balance across the clones. For example, Method 3 might divide records into those for North, East, South and West regions – where that division could be decidedly unequal.
• Method 3 might not be scalable to 8-up or 16-up, simply based on the difficulty of finding 8-way or 16-way split criteria.
• Method 2 could allow some clones of the original application program to start earlier than others. In some cases this is a good thing, in others a problem.
• Method 2 would need occasional adjustment to rebalance.
• Method 3 implies non-trivial application coding to effect the split but provides the best chance of minimising contention between streams. (One neat coding shortcut if you’re using DFSORT to do the split is OUTFIL SAVE – which provides a "non of the above" bucket.) Whether you use DFSORT or a home-grown split program depends on the precise split logic – but DFSORT is much simpler and scales slightly more easily to e.g. 8-way and 16-way.

Changing Programs To Expect Subsets Of The Data

In our example the original program processed all the data. It could make the "I am the Alpha and the Omega" assumption. If we split the transaction file we forego this. The most obvious result is that any report the program would have written needs to be rethought: We probably will only be able to write out a file that feeds into a new report writer (Which we’ll talk about below.)

Merging

Batch steps produce, amongst other things, output transaction files and reports. For the sake of (relative) brevity let’s concentrate on these two:

• Output Transaction Files

  Somehow we need to merge these files (though an actual sort is unlikely). It’s important to know what the sensitivity is and cater for it.

• Reports

  Reports usually require some calculations, extractions to form headings, and so on. Sometimes a simple merge of the report output from the cloned program is enough. My expectation, however, is that serious reworking is usually required. Totalling and averaging would be typical examples of where it gets complex (but not impossible).

  I would remove the reporting from the original program and think about where it fits best in the merge. There are advantages to separating the data merge from the "presentation": If today the report is a flat file (1403² format?) you could enhance the report to also³ produce a PDF or HTML version. That might be a nice "modernisation".

Refactoring the JCL

JCL Management is not my forte but it’s obvious to me it’s worth examining the JCL for any job that’s going to be cloned to see how it can best be managed.

It might not be feasible to keep a single piece of JCL in Production libraries for the schedule to submit as multiple parallel jobs. If you can then parameterisation is the way to go. For instance it wouldn’t be helpful to have the job name hardcoded in the JCL. Similarly data set names which differ only by the stream number need care, as would control cards you pass into a program.

Changing The Schedule

You have to change the schedule to accept new job names – for the clones of existing jobs – , insert new jobs (for splitting, merging and reporting), and wire this all up with a reworked set of dependencies.

There are decisions to make, such as whether (in TWS terms) each stream should be its own Application, and what the dependencies should be. For instance, do you keep the streams in lockstep?

One of the key things is planning for recovery: Whereas, in our example, it was all one job step you now have three (or maybe) four phases of execution. Where do you recover from?

Reducing Data Contention

In our example, File A and File B were originally each read by the original application program. If they were keyed VSAM, for example, buffering might’ve been highly effective – particularly with VSAM LSR (Local Shared Resources) buffering. Four clones reading these two data sets will have to do more physical I/O. In the VSAM LSR case some hefty buffer pools could help reduce the contention going 4-up might introduce. In a database manager like DB2 things ought to be better: Data is buffered for the common good.

Dare I mention Hiperbatch? 🙂 For the smallish⁴ Sequential or VSAM NSR (Non-Shared Resources) case this might work well – but it would be a very uncommon approach.

As I hinted above, one of the things that might condition how you split the transaction file is what effect it would have on data contention. If you found a split regime where all the data the clones processed was split the contention could be very low.

If you got to the point where the split regime was "universal" or at least widespread enough some of the contention (and indeed Merging) issues would disappear completely.

Tape is particularly fraught: You can’t have two jobs read the same tape data set at the same time. I’ll indulge myself by mentioning BatchPipes/MVS here 🙂 as it provides a potential solution: A "tape reader" job (probably DFSORT COPY OUTFIL) copying the data to the clones through pipes.

However you do it, the point is you have to manage the contention you could introduce with cloning.

Monitoring

Monitoring isn’t terribly different from any other batch monitoring. You have the usual tools, including:

• Scheduler-based monitoring tools – for how the clones are progressing against the planned schedule.
• SMF – for timings, etc.
• Logs

If you can develop a sensible naming convention for jobs and applications your tools might be easier to use.

One other thing: You need to be able to demonstrate that the application still functions correctly. This is not a new concept, of course, but application testing is going to be challenged by the level of change being introduced.

This concludes the four-part series. If you’ve read all four all the way through thanks for your persistence! The purpose in each was to spur thought, rather then be a complete treatise. My next task is to turn this into a presentation – as the need has arisen to do so. One final thought: If this long (but necessarily very sketchy) post has put you off please re-read Part 1 as I talk there about why this could be necessary.

---

¹ This usage of "application program" most centrally refers to programs written in programming languages such as COBOL. It could also refer to things like DFSORT invocations. The point is these are difficult things to understand and to change.

² You do know about DFSORT’s REMOVECC, don’t you? It tells DFSORT to remove ANSI control characters – such as page breaks. When separating data preparation from presentation you may well find it useful.

³ I bolded "also" here because the original report probably has a consumer – whether human or not – who’d get upset if it didn’t continue to be produced but might like a more modern format. And if it doesn’t… 🙂

⁴ While technically still supported, Hiperbatch has functional limitations, such as not being supported for Extended Format data sets (whether Sequential or VSAM). Further, the only way to process Sequential data with Hiperbatch is QSAM. (For DFSORT you’d have to write an appropriate exit – E15, E32 or E35 – to read or write the data set.)

(Originally posted 2012-03-04.)

Part 1 and Part 2 were, in my opinion a little abstract. But I think they needed to be:

• They set the scene for why parallelising your batch can be important.
• They gave some vocabulary and semantics to help structure our thoughts.

Now we need to go a little deeper.

Let’s start with how to think about the problem of making a job or set of jobs more parallel.

Hetereogeneous Parallelism

Here the trick is to remove dependencies – and that’s where most of the issues are.

I covered a lot of this in Batch Architecture, Part Two.

Homogeneous Parallelism (Cloning)

This is where it can get really tricky. And that’s why the bulk of this post is about the homogeneous case. (Some of the following will also have relevance to the heterogeneous case.)

There are a number of issues to work through when cloning jobs. Here are some of them:

• Converting the serial bulk processing model to something more parallel.
• Handling inter-clone cross-talk
• Resource provisioning
• Scheduling

Parallelising The Bulk Processing Model

The reasons for using batch include the advantage of using a “bulk processing model”: Doing the same thing to lots of data in one job is much more efficient than breaking it up into a huge number of one-datum transactions. But, just because it’s more efficient to process 10 million records in a single batch job doesn’t mean it’s much more efficient than doing it in 10 1 million record jobs.

The trick with cloning is to find a way of breaking up an (e.g.) 10 million record job into ten parallel 1 million record jobs.

Consider the following diagram¹:

A lot of bulk processing looks like this. The salient features are:

• Reading a Master file, one record at a time.

  This could be a sequential file, a concatenation of these, a VSAM file, rows returned by a DB2 query, or any one of a number of other similar “files”. The point is it’s a large number of records or rows – perhaps the 10 million mentioned above. And any serious attempt to parallelise this job is going to have to split this file.

• FIle A is read to provide detail.

  Hopefully this is a keyed (direct and buffer able) read. You don’t want to have to read the whole file to find a match to the record from the Master file.

• Likewise File B.
• The detail that is being filled in – in this case totals – is held in memory by the program.
• When the Master file has been completely processed a report is written – using the summarisation information in memory.

I find this notion of a circuit, driven by a Master file, useful. If you find you can’t draw it that tells you something in itself. I’m sure it’s not the only bulk processing pattern, but it’s a very common one.²

(It would be difficult to attach timings to the activities in the loop. A reasonable stab could be made, under some circumstances, at the data set accesses’ proportions of the overall run time using SMF 42 Subtype 6 records.)³

This is only an example but it illustrates some issues that cloning needs to resolve:

• The Master file needs to somehow be split into 10.
• The ten sub-reports need to be reworked to produce a coherent and correct final report.

We could talk about resolutions of these – and I probably will in Part 4 – but the important thing is to acknowledge these are issues that have to be addressed.

Resources

If you’re going to run more jobs in parallel you could easily “spike up” resource usage, most notably CPU consumption. Memory use might increase also, though some usage patterns (such as DB2) tend to have a noticeable memory impact. I/O bandwidth and initiators are two more things to think about. In the I/O case it could be seen as a case of this. In any case, we know how to monitor resource usage, don’t we?

Handling Inter-Clone Cross Talk

While logically it might be easy to clone a job, things like locking can make it really difficult.

For example, in (a modified version of) the case above, File A and File B might be updated. These updates might be one per record in the Master file, or just at the end. In either case cloning would introduce some locking issues. It might be possible to resolve these issues – perhaps through partitioning.

Even in the unmodified version clones reading from File A and File B might create I/O bottlenecks. In the DB2 case you’d hope this would have a happy ending.

Scheduling

If you’re going to run multiple copies of a job in parallel you need to adjust the schedule.

There are design decisions like whether you keep clones in lockstep:

Consider this example: Suppose you have a pair of cloned streams – consisting of A₀, B₀ and C₀ in Stream 0 and A₁, B₁ and C₁ in Stream 1. Each B_n logically follows its corresponding A_n and each C_n follows the corresponding B_n, based on data flows.

• If you release B₀ and B₁ only when both A_n jobs have completed it’s more controlled but probably takes longer.
• If you release each B_n when the corresponding A_n completes its’ less controlled but probably takes less time.

The term “Recovery Boundary” is probably useful here as recovering from job failures is the thing that makes the complexity introduced by cloning really matter.

I advocate cloning in powers of two: 2-up then 4-up then 8-up, and so on. A modification to this is 3-up, then 6-up, then 12-up – which has a fairly obvious appeal, when you consider typical data sets.

Automating cloning is a useful aim, whether you want to “dynamically” partition the work or just want to be able to move from, say, 4 streams to 8 without too much trouble. I put the word “dynamically” in quotes as realistically the latest you could decide on the number of clone streams is just before you kick them off. In reality it’s probably much earlier than that.

So you need to solve the problem of how you might do this: The first step would be to decide what’s realistic. The second would be to decide what’s needed.







In this post I’ve highlighted some of the issues – mainly for cloning. They’re not terribly different from this you’d encounter if you were pursuing homogeneous parallelism (which you may also need to do). Part 4 will round out the series with some thoughts on implementation.

---

¹ Made with Diagrammix for Mac. I like this particular style, untidy thought it may be. A nice demo is here.

² There probably are formal diagrams of this type, probably with the letters “UML” attached to them. I don’t claim to be the kind of person who would use one. I just think teasing out the circuit like this is helpful for cloning. It’s the circuit itself I’m attached to.

³ Quite apart from the incompleteness of this approach there are issues with overlap between the data sets (and with CPU). A discussion of double buffering, access methods and I/O scheduling is well beyond the scope of this post.

(Originally posted 2012-02-26.)

Given I’m not paid to blog, and given I’ve no real motivation to maximise my blog post count, the frequency of posting is just "what it happens to be".

In that spirit this post isn’t about how to "game" blogging statistics (and it itself isn’t a gratuitous attempt to increment the count by one). 🙂

What I want to convey is my experience with mind-mapping software, in the hope it’s useful to you. It’s also not a review of a specific piece of software, though it’s inevitable my take on one particular piece of software will come into it.

Let’s get the "piece of software" element out of the way…

I use MindNode Pro on both Mac and iPhone. It’s two separate pieces of software that talk to each other. (I think the iPhone software would work well on the iPad but I haven’t tried it so don’t know if it takes advantage of the better "screen real estate".) The "that talk to each other" piece is significant in that you can work on a mind map on the move on the iPhone and then transfer it to a perhaps more powerful environment via Wireless. And back again. Successful "round-tripping" is very important.

So what’s "Mind Mapping" and how am I using it?

To me a mind map is a hierarchical organisation of ideas – in a branching tree format. The term suggests literally mapping your mind which is a rhetorical stretch and a half. But to me it’s just an attempt at organising a set of thoughts. Not being a particularly linear thinker some structure like a tree is reasonable – but a more generalised net is probably better for me.

My experiment was to see if I could create a better-structured blog post: At the time the post I had in mind was, frankly, a mess. It may still be – but it’s a better-organised mess. 🙂

So I started MindNode Pro up on the Mac and proceeded to dump ideas – breaking each idea down into sub-ideas. In the diagrams that follow you’ll see each idea is expressed in very few words.

When I "folded" up sub-trees I ended up with this:

The result was unexpected – a set of four sub-trees, each of which it was clear to me could be its own blog post – but only if I had enough material to make each worthwhile. The whole point here is there’s a natural division into four.

If you "unfold" the second sub-tree you get:

This is a fairly typical sub-tree – although the biggest one is sub-tree one (which creates web real estate issues so I chose the second one instead). So it actually re-inforced my view the material would be too much for a single blog post. Hence the four-parter.¹

Trees are all very well but how do you make a post flow? This is part of what I’m going to term "Mind Map Debugging". That’s because there’s a little more to it than that:

• Flow

  With (at least) the software I’m using you can cross link between tree nodes . I contemplate doing that – perhaps using dash lines – to see if I traverse all the important nodes with a reasonable flow. (This is sounding awfully close to the Seven Bridges of Königsberg Problem but I think that’s taking it too seriously. 🙂

  What I actually did was to eyeball the mind map and see if it flowed. Each sub-tree seems to flow well and the entirety feels right, too.

• Rebalancing

  While doing formal tree balancing is pretty much pointless here, I do think that I might’ve needed to "rebalance" the four sub-trees if they were seriously imbalanced. Otherwise the four blog posts might’ve been one large one and three small ones.

  But this isn’t arbitrary data but rather blog posts fitting into a conceptual whole: The root node needs to stay the root node, in this case.

You can see the way my mind is working here:

• The tree paradigm has some (admittedly) weak contribution to make – in thinking about mind mapping.
• You could apply mind mapping software to other forms of tree depiction: I might well do that for data centres with machines in, which in turn have LPARS, then workloads, then address spaces, then transactions…

So there are lots of possibilities here.

I actually did use the ability to transfer mind maps between Mac and iPhone – adding a small number of nodes on the phone. This post was made possible by the fact you can selectively export parts of the mind map to a bitmap. So, I think the idea of mind maps and this particular implementation worked well. Next stop: FreeMind, which I’ve installed on my Linux work laptop.

---

¹ and hence the "The Sign Of The Four" reference in this post’s title. It’s taken from Sir Arthur Conan Doyle’s second Sherlock Holmes novel. (I’ve tended to call him "SirACD" on Twitter.) 🙂

(Originally posted 2012-02-26.)

I hope you don’t get the idea I’m overly into rigour, talking about Classification. But I think it has to be done – to provide terminology for this series of posts.

This is the second of four posts on Batch Parallelism, following on from Motivation.

If I think about how parallelism works in batch it broadly falls into two camps:

• Heterogeneous
• Homogeneous

(If you look these two terms up in Wikipedia (possibly for the spelling) 🙂 you get to see under a rather tasty 🙂 graphic the words "Clam chowder, a heterogeneous material".) 🙂

Let me explain what I mean by these two, in terms of batch classification.

Heterogeneous

Almost all customers run more than one batch job at a time. Personally, I’ve never seen anyone feeding through a single job at a time.

But a lot of the time it’s separate suites (or applications, if you prefer). Or certainly it’s running dissimilar jobs alongside each other.

You can further divide this case – in a way which actually makes it less abstract: ¹

• Unlinked

  This would be the case with totally separate suites, possibly from different lines of business.

• Weakly Linked

  Again, these are separate suites, but this time the suites feed into each other – at least occasionally. These are less likely to be from separate lines of business – though a thoroughly integrated enterprise might have more cross-suite linkages.

• Strongly Linked

  This would typically be the case of a single suite – where the whole point is to do related things, such that data flows between the jobs (and even steps).

By "linked" I’m mainly talking about data flows, though it could be operational cohesiveness.

Homogeneous

This is the case where work is very strongly related. There are two subcases:

• Cloning

  It’s quite common for applications to be (re-)engineered so that identical jobs run against subsets of the data. This is commonly termed "cloning".

• Within-Step Parallelism

  An example of this is DB2 CP Query Parallelism – where DB2 splits the task up into, effectively, clones – but manages them as a single unit of work.

  Not quite the same, but possibly best fitting here, is substep parallelism.

Which Do YOU Do?

I think most customers do "heterogeneous" to a very considerable degree. That’s because it comes naturally and is the way the business has grown and driven things.

Less common (and I was recently pressed to give a view on how common) is "homogeneous". That’s because it takes real effort.

The answer I gave was something along the lines of "I don’t know for certain but I guess about 30% of customers do homogeneous". ²The reason I gave that answer is because I suspect homogeneous parallelism gets added to applications to make them perform.

It’s my view that applications and going to have to become more homogenously parallel in the future – because of the dynamic I described in Part 1: Over time the speed up required of individual actors (typically batch jobs) is likely to outstrip that delivered by technology.

To become more homogeneously parallel we’re going to have to understand the batch applications much better. (Actually that’s true also of efforts for more heterogeneous parallelism as well. Parts 3 and 4 of this series will address some of this understanding – and provide some guidance on what’s going to need to be understood. And hopefully will make this classification seem less dry and more helpful. 🙂

---

¹ There’s probably a rule that says the leaf nodes of classification schemes yield a higher proportion of concrete examples.

² The "I don’t know for certain" part of it is because I recognise I see a "self-selecting group" or "biased sample" of customer situations: Those that are particularly thorny or exceptionally critical.

(Originally posted 2012-02-19.)

I have enormous trouble pronouncing "parallelise" right – and not saying "paralyse". It’s true, and I bet many of you have the same trouble (sober or not). It’s on a par with "red lorry yellow lorry" or "the Leith Police dismisseth us". 🙂

But it’s a word I think we’re going to have to get used to pronouncing right. And this post will explain why.

This looks to me like a 4-part series of blog posts on increasing Batch Parallelism. (It started off looking like one but the way it turned into four is perhaps material for another post.)

So, why will parallelising batch become increasingly important? There are really three main reasons:

1. Increased "Window Challenge"
2. Resilience
3. Taking Advantage Of Capacity

There is some overlap between these but I think they’re sufficiently distinct to draw out separately – which is what the rest of this blog post does.

Increased "Window Challenge"

From a business perspective this is the big one. I’m seeing a number of business trends that are leading to one inexorable conclusion: The delivered growth in speed of "single actors" (batch jobs) will be outstripped – over time – by the need. In other words, you can’t long-term just buy yourself out of trouble, whether we’re talking about processor speed, disk subsystem or tape speed, transmission line speed, or anything else for that matter.

It’s true this varies by installation, and even between applications (or suites) or business lines in an individual organisation. But this is the general pattern. It’s also a fact that the pressure comes in waves – because of the nature of the underlying business requirements.

Amongst the business drivers I’ve seen:

• Business volume increases.

  Hopefully these are driven by success.

• Mergers and acquisitions.

  Typically I’m seeing the same application having to cope with more data as one or other party’s application is adopted.

  A similar trend is "standardisation of procedures" where existing lines of business come together to use a single application

• More processing.

  In the merger scenario above I’ve seen cases where taking two companies’ data and passing it through the "ongoing" applications means these applications have to be modified (with generally greater pathlength). And decommissioning the "offgoing" applications is another complicating factor.

  External pressures such as regulation often lead to more work per unit of business volume.

  Modern techniques such as Analytics get injected.

  And of course our old friend "just because" i.e. processing grows for all sorts of reasons.

• Shortened Window.

  Much has been said about running batch and online concurrently. But shortening the batch window itself remains important for a number of reasons, amongst which are:

  • Even if you overlap everything there are still only 24 hours in the day.

    In other words the work still has to get done in the cycle, whatever that cycle may be.

  • Running online and batch together increases the aggregate resource requirement.
  • Batch jobs taking locks (or causing database I/O) can still interfere with transactions.
  • Batch and online concurrency is still a difficult feat to achieve.
  • There are often deadlines within the batch and sometimes these get tightened up.

Resilience

With a single-threaded job stream just one broken application data record can hold up the whole thing. Or the loss of an LPAR or DB2 subsystem or VSAM file.

Partitioned data can mean an increase in resilience. For example:

• If the data were processed by geographic region (and you had, say, 5 regions) the damage of a broken record is limited to that region.

  This, of course, depends on region-level separation. And, naturally, any failure is unwanted – but the business impact could be much reduced.

• If the LPAR were to fail in a correctly-set-up multi-image environment, again the impact could be limited.

  There’s a lot to this one. For example, retained locks by a DB2 datasharing member could limit the benefit.

Taking Advantage Of Capacity

Businesses have tended to size machines by online day requirements and there remains the view that generally it is online that’s the peak use of resources. My experience is that about half of installations have batch as the real CPU driver (but probably not the memory driver) and more than half have a bigger I/O bandwidth challenge overnight than during the day.

Where the online day is still the main resource driver an increase in parallelism can usefully absorb the spare capacity overnight.

Where Next?

I contemplate this being a four-part series of blog posts. This part has concentrated on business drivers, almost to the exclusion of technology. The other three posts I expect to be, in order:

1. Classification.
2. Issues.
3. Implementation.

The titles and scope might change a little bit as I flesh them out. I’ll leave you in suspense 🙂 as to what "Classification" might be.

(Originally posted 2012-02-13.)

Way back in April of last year I started to talk about a presentation I hoped to write: "I Know What You Did Last Summer" and I showed a brain dump of ideas. Then in June I blogged the abstract (complete with a revision in a subsequent comment). Despite the occasional comment on Twitter it all went quiet until today.

Now the more cynical among you will be remarking that I forgot all about it. Actually that’s not true. Two things needed to occur before I was going to make much progress:

1. There needed to be a compelling deadline to work to. (Doesn’t there always?) 🙂
2. I needed a narrative framework.

I sort of have 1 – this presentation really will have to be completed before the May timeframe if I’m to present it at a couple of conferences in Europe.

What I want to talk about today is the fact I have 2 – a narrative framework that I think will work.

What I had all along was a message. It goes something like this:

"While we traditionally value the instrumentation on the z/OS mainframe for Performance and Capacity, there are other ways of using what we have – most notably for Inventorying, Gleaning System Understanding, and Talking to IT Architects."

That was the abstract notion I walked in with and, if anything, it’s amplified now rather than attenuated.

The following two graphics from the presentation are the first and last in a layered sequence that provides the narrative framework:

We start with a very high level "Physical Resources" view:

and proceed down until we reach a much more logical "Application Componentry" view:

I won’t spoil your page-loading enjoyment by showing the graphics for the intermediate layers in this blog post. Suffice it to say the colours represent layers. Let’s talk a little more about layers…

Untidiness Of Layering

The layers I present aren’t strictly hierarchical: Without padding out the presentation I’m not going to make them so. But here they are and you’ll see what I mean:

1. Physical – Blue
2. LPAR – Turquoise
3. WLM Constructs – Red
4. Address Space and Coupling Facility Structure and XCF Group / Member – Purple
5. Application – Green

If I really did treat Layer 4 as three separate layers where would it end? It would certainly make the presentation more turgid.

What I can say is that all the elements of Layer 4 belong below Layer 3 and above Layer 5. And that when I look at systems I try to do it in this sequence.

Sparseness Of Style

You’ll notice a lack of words and a lack of connectors. In the real world, of course, there’d be things like CF links and LPARs would have names. But the message isn’t helped by adding any of these. And a certain sparseness of style feels right to me.

Gratuitous Graphics?

You might ask "Why have these graphics at all?" Generally that’s an acid test I apply – possibly to excess. Those of you who’ve seen me present know typically the only graphics in my presentations are graphs. In this case I think a sequence like this helps.

It should be noted I’m under no pressure to "jolly it up" with lots of pretty graphics. In fact this isn’t a commissioned presentation at all: It’s one I think is important. So it gets whatever style I choose to give it, perhaps with advice from others such as you.

Flexibility Of Timing

I joked today on Twitter:

Question: “How long is a piece of string?”

Answer: “Fifty minutes plus questions to One Hour plus questions, depending”.

OK, not a very funny joke but it makes a point:

When I present I generally get 1 hour slots or 1 hour 15 ones. For any presenter it’s tough taking a presentation and shrinking / stretching it appropriately. This structure gives me quite a lot of flexibility, I think. I foresee no difficulty adjusting to any time slot.

Conclusion

This structure enables me to survey the ground in a structured fashion – drawing on instrumentation from a diverse set of sources. And then it provides me a launch pad to make the other points.

For example, the “Inventorying” and “Talking to IT Architects” points flow naturally from this.

So now I’ve got a structure I can get going with the rest of the presentation. I think at last I can say I actually have a show. The rest is just details, inspiration and perspiration. And believe what you will about the proportions of the last two. 🙂

Now if anyone can tell me how in OpenOffice.org to make it honour a PNG file’s transparency I’d be grateful. The original graphs were made using Diagrammix on a Mac and exported as PNG files with a transparent background. When composing this post Firefox was entirely happy to honour that but it seems OOo isn’t. 😦

(Originally posted 2012-02-06.)

I was lucky enough to be in Silicon Valley Lab for DB2 BootCamp last week. There I ran into a DB2 developer I’ve worked very successfully with in the past – John Tobler.

(He’s the guy I look to for questions and issues with DB2 SMF data.)

We had a good discussion about something I’d personally like to see in DB2 Accounting Trace – more WLM information – and this post is as a result of this conversation.

Two salient pieces of information:

1. Accounting Trace already has a field for WLM Service Class (QWACWLME) but it’s only filled in for DDF work.
2. As Willie Favero pointed out in APAR Friday: WLM information is now part of the DISPLAY THREAD command the command now has some WLM information in it.

Putting these two together you come to the conclusion it might technically be possible to get more WLM information into Accounting Trace. That, of course, doesn’t mean it’s going to happen. I have to stress that before going any further. But it’s worthwhile thinking about what’s needed and how useful that would be to customers.

What Should Be Added?

Uncontroversially, I think, QWACWLME should be filled in with Service Class for all work types. I say "uncontroversially" because – if it can be done cheaply – it’s just using space that’s already in the record. I don’t know if it can be done cheaply, though.

More controversially because, taken together, they represent 18 additional bytes in each 101 record are:

• WLM Workload
• WLM Report Class
• WLM Service Class Period

I think I could live without Workload but it seems a shame to exclude it.

As Willie points out Performance Index (PI) is also in the DISPLAY THREAD command but I think we can get that from RMF Workload Activity Report (SMF 72) and that’s probably a better place to get it from.

But the key question is “how useful and important would this extra information be to you?”

Let me outline three areas of use I can immediately see…

Understanding Not Accounted For Time

This time bucket is what you get when you subtract all the time buckets we know about from the headline response time. The two most important causes for this are CPU Queuing and Paging Delay.

If we calculate this time for a record and we know the (behaviour of) the WLM Service Class it’s in we can understand this time better. A bugbear of doing DB2 performance is just this: understanding whether work is subject to queuing or not. (For Paging Delay as a cause of Not Accounted For Time we could do much the same thing.)

Understanding The WLM Aspects Of DB2 Work

It would be useful to be able to break down the work coming into a DB2 subsystem by Service Class, Goal and Importance, wouldn’t it? In particular it would be nice to see the hierarchy of goals and importances, and to be able to relate the works’ WLM attributes to those of address spaces such as DIST and DBM1. (In the former case discovering that the TCB’s in the DIST address space were subject to pre-emption by the DDF work would be a blow.)

Correlating Service Class And Report Class For DDF Work

For non-enclave work I use the Report Class and Service Class in Type 30 to establish how these relate to each other (and what kind of work has which RC and which SC). I can’t do it for DDF work because there’s no usable Type 30 (i.e. with this kind of information in). If the 101 record had these both in you could extend the method.

(In case you wonder what I’m talking about see What’s In A Name?.)

This still doesn’t help us in the non-DDF enclave cases, of course.

Over To You

What do you think? I’ve listed three categories of value that immediately spring to mind (and that’s with the disbenefit of jetlag so maybe not that articulately expressed). But I’d really like to know if this would be of value to you – and to modify the proposal if you think you’d like something slightly different.

There’s no guarantee this will get done – and it’s a bit of an attempt at a “Social Requirements Gathering” process. But it’s worth debating in public, I think.