z/OS Performance Instrumentation Management Techniques wiki

(Originally posted 2007-07-12.)

I’ve just created a wiki to discuss primarily SMF. Mainly from the management perspective, rather than the contents of each individual record.

This follows on from things I’ve mentioned in this blog before.

If you’d like to contribute to it (and it is DESPERATELY in need of contributions right now) get a developerWorks screenname and send it to me here. Then I’ll enable you to edit the wiki.

You don’t need a screenname to be able to view the wiki.

The wiki is here.

Why Mainframe Folk Should Care About Web 2.0

(Originally posted 2007-07-05.)

I presented a set of (someone else’s) foils on Web 2.0 to my team meeting last week. (Interestingly, being 6 months old they were already way out of date – what with Twitter and all.) Remember I’m in a mainframe crowd of effectively “gurus”. πŸ™‚ So why should they be interested in new-fangled webby stuff? So I got to thinking… Dear reader, why should you care about Web 2.0?

The minimal answer is “because it’s going to happen anyway, whether you like it or not, and you and your organisation are going to be left in the dust if you don’t embrace it”. I think that’s a fair answer but really there is positive stuff in there for us.

But rather than quote from The Long Tail or The Wisdom of Crowds (Read it and reading it, respectively) I think it better to point out some examples of Web 2.0 you may well already be using…

  • developerWorks blogs (like this one).
  • Wikipedia wiki (where those two book references above were from)
  • Flickr photo sharing site
  • Twitter microblogging

What these sites have in common is that they get better the more people use them. Both in adding content and also in rating and ranking material (or editing it in the case of wikis). As such they’re marking a shift away from static websites to ones where users have more control and the sites themselves just become enablers. And that leads onto changes in the web that we need to take notice of.

The other element of note is the idea of a “mashup“. This is where content from one site is mashed together with that of another to create (usually) a third site. Good examples of this would be the whole host of mashups built around Google Maps. Now they’ve been smart as they publish an API that web developers and mashup creators can use. The lesson here is that if you build your website so that it can be mashed up with others then your website will be used in such mashups and it will attract many more visitors.

A good analogy might be an insurance company that makes it hard for an insurance quoting website to garner quotes… That insurance company isn’t going to get so many quote requests as one that does.

Now, how does that affect the mainframe? It doesn’t directly but it does lead to a driving up of traffic and an ever higher reliance on good response times. So our old friends scalability and performance come into play. And we play well in those terms. And it does keep the focus on availability as well.

And how does it affect mainframers? My answer would be that we can really use a lot of these new technologies in our day jobs. And if we don’t we risk letting other platforms have all the fun. πŸ™‚

So I’d encourage people to dive into Web 2.0. And that’s what I told my team last week.

Feedback from my UKCMG Mainframe Performance Instrumentation Birds Of A Feather

(Originally posted 2007-07-05.)

It was a very good session, even if it was attended by just a “hard core” of mainframe sites. I think everyone said at least something and several said rather more than that. Here are some things I’d like to note from it…

There was a general feeling that it’d be useful to have a name for a machine that could be entered on the HMC and flow through to SMF records, particularly Type 70 (CPU). So for example a site might like to name it’s machines “North Mainframe” and “South Mainframe” rather than just being identifiable by the hardware serial numbers 5112345 and 8356789. I think this is a really good idea – at least from MY perspective as someone who wanders onsite and would rather use the names YOU use for your machi nes, even if I do remember the serial numbers for at least one customer. πŸ™‚ This idea, though, would require changes in at least three components. So I’m not overly optimistic. But I’ll make some enquiries and see what we can do.

We also think that the serial number should appear in OTHER SMF records (than 70-1 and 74-4) such as the other RMF ones and also Type 30. That would allow much easier matching up.

On the memory front I didn’t meet with that much interest – except that we think it important that the memory numbers become more accurate in Type 30 and Type 72. (Both of these are currently reported based on the notion of Service – which means that swappable workloads are under-reported in Type 72 and those that endure CPU queuing are under-reported in SMF 30.) We also would like to see SMF 70 report how big the machine’s HSA is and how much memory is purchased but not assigned to a partition.

We discussed instrumentation for VLF / LLA and for Catalog – which we think could be improved.

When talking about techniques for managing SMF data the SMFUTIL tool was mentioned. I’ll have to do some research into this, but I still think it worthwhile to post some examples of DFSORT being used to “slice and dice” SMF. One day. The meeting also felt that some kind of “best practices” guidance would be useful. Maybe I should start a wiki on the subject – so that the collective wisdom of the mainframe performance community can be tapped.

All in all I think the session was a success – and I’d like to do it again next year. I’ll work on these items but, as I originally said, there are no guarantees.

So thanks to the participants. Plenty of food for thought.

Abstracts for System z Technical Conference, San Antonio, September 17-21

(Originally posted 2007-06-23.)

Here are my abstracts for the conference:

Session B11: DB2 Data Sharing Performance for Beginners

This presentation provides an introductory-level view of how to look at the DB2 Data Sharing performance numbersfrom both a z/OS / RMF and a DB2h perspective.

Performance topics include: XCF, Coupling Facility, Data Sharing Structures, The application’s perspective, and Structure Duplexing.

Performance topics don’t include: Other forms of Data Sharing eg VSAM RLS, and overly detailed descriptions.

Session P22: Memory Matters in 2008

For z/OS LPARs memory management has changed radically over the years – from both the operating system perspective and that of applications. And the pendulum has swung back and forth between focusing on Real Memory and on Virtual Memory.

This presentation discusses managing both Real and Virtual Memory – from the perspectives of both the operating system and the exploiting products. The products include DB2, DFSORT, CICS, IMS, MQ and Websphere. One topic of particular importance to installations upgrading z/OS is the Release 8 Real Storage Manager rewrite.

Session P23: Much Ado About CPU

zSeries and System z9 processors have in recent years introduced a number of capabilities of real value to mainframe customers. These capabilities have, however, required changes in the way we think about CPU management.

This presentation describes these capabilities and how to evolve your CPU management to take them into account. It is based on the author’s experience of evolving his reporting to support these changes.

And Something Else That Confuses Me

(Originally posted 2007-06-21.)

This customer is using DB2 Hiperpools, despite being on a z9 BC processor and running DB2 Version 7.

What I notice from DB2 Statistics trace is that most of the Hiperpool pages are not backed by memory. In other words they’ve ceased to exist. This despite there being plenty of memory in the LPAR. One thing that’s also significant is that there are very few reads back from the Hiperpools. It’s mostly writes.

Somehow I think the two facts are linked. In any case I would recommend the customer moves to Version 8 and, in the short term, uses Dataspace pools instead of the combination of Hiperpools and Virtual Pools.

Meanwhile I’m still scratching my head. πŸ™‚

It Doesn’t Take Much To Confuse Me

(Originally posted 2007-06-20.)

… with a Performance Consultant? πŸ™‚

Seriously, here’s something that made me go “hmm”…

I started looking at the latest set of data from a customer…

Their biggest-CPU WLM workload is called “BATCH” when they told me they were a CICS shop – and that CICS was their main CPU consumer during the day.

But it turns out that the biggest service class within that workload contains exclusively CICS regions. (And it is called “PRDONL”.)

So that’s all right then. But you can see why I’d be confused. πŸ™‚

Moral: A name is just a label, not a description. At least that’s true with WLM. But funny how we read things into names that generally are there. πŸ™‚

IMS Version 10 Memory Enhancements

(Originally posted 2007-06-13.)

Regular readers would know I’m working on my “Memory Matters in 2008” presentation, which is a re-spin of the ’07 Version. One of the things I talk about is IMS.

Thanks to my team mate Andy Wilkinson for this list of IMS Version 10 enhancements, all of which are virtual storage usage improvements.

Here are the more important items:

  • OTMA message flood protection (prevents LSQA filling). This is retrofitted to IMS V9 by PTF.
  • New IMS log records (x’4511′ and x’4512′) to report on control region memory use.
  • IMS V10 acquires CSA for Fast Path resources in discrete pieces, rather than insisting it all be contiguous, which helps large Fast path customers

And here are some other, less important, IMS V10 items:

  • ACBGEN supports 31-bit addressing for the first time, which helps programs which access thousands of databases.
  • PCBs are passed from CICS to IMS using 31-bit addresses (instead of 24-bit addresses), which also helps programs which access thousands of databases.
  • Some IMS modules move above the 16Mb line, saving about 100KB of virtual storage.
  • Several IMS modules that used to reside in 24-bit common storage now reside in 31-bit common storage

The details of these are in the IMS V10 Release Planning Guide GC18-9717-00 available from here.

UK GSE Conference 30-31 October – Memory Matters in 2008

(Originally posted 2007-06-06.)

One of the problems with having a presentation with a topical title is that you keep having to update the darned thing. πŸ™‚

“Memory Matters in 2007” is a case in point. It’s morphed over the years from two presentations: “Central Storage Performance” and “Expanded Storage Performance” (the latter of which eventually became obsolete) to a presentation that covered OS/390 real memory, DFSORT and DB2 virtual storage, to the current “Memory Matters in 2007” which covers z/OS real memory and a bunch of subsystems (CICS, MQ, DB2, DFSORT, IMS and Websphere Application Server).

Now I’ve been asked twice to present the same topics but fast-forwarded to “Memory Matters in 2008” – when we’re still in 2007.

I suppose I could call it “Memory Will Matter in 2008″ but that rather spoils the pun. (I do wonder how many of my audience in München got the pun.) πŸ™‚

So anyhow, I’m going to update most of the sections to cover eg DB2 Version 9 and CICS TS 3.2. Just a little sooner than I’d anticipated. 😦

The second outing for the ’08 model will be at the UK GSE Conference October 30-31. This is at Chesford Grange, near Leamington. (Haven’t I been here before?) πŸ™‚

What I might try and do is post new foils (or at any rate sketches of them) here and see how people react.

And I suppose my manager will want me to podcast this. πŸ™‚ I wonder where to host such a thing.

Humble Pie Regarding DFSORT and SMF

(Originally posted 2007-05-31.)

(Also posted to MXG-L Listserver (MXG-L@PEACH.EASE.LSOFT.COM) which I highly recommend as a place where mainframe performance people hang out.)

I must confess I feel slightly foolish about this… πŸ™‚

After having asked for input into my UKCMG BOF one of the themes that came out was better ways of doing what IFASMFDP does. (I already have this on my agenda – as you’ll know if you follow my blog.)

So I asked my friend Frank Yaeger (who is the DFSORT Function Development lead) the following question:

“Frank, I thought you told me once that DFSORT had some issues with handling actually spanning records. Did I hear you right?”

to which I got a reply (paraphrased)…

“No, that’s wrong. DFSORT handles spanned records just fine. We can handle records up to 32767 bytes in length”.

So, a healthy slice of humble pie is mine, complete with plastic fork and paper plate. πŸ™‚

Seriously, would anyone care to do a test for me and copy some eg Type 30s using DFSORT? 42-6 and 74-1 are the other cases that bother me. And perhaps 74-5. The test would be to copy and then run the results through (say) MXG. (I suspect using ICEGENER wouldn’t be a fair test as it may well revert to IEBGENER which we’ve always said does break SMF records.)

What I’d like to do, if such tests are successful, is to publish some examples of using DFSORT as a “better IFASMFDP”.

One thing still bothers me a little: Using DFSORT Copy doesn’t produce Dump Header and Trailer records – SMF Types 2 and 3. Does anyone actually care about losing them?

Note: This is emphatically not a DFSORT vs Syncsort question. I’d assume Syncsort also copes with spanned records. If someone is in a position to test that then great. If this were ever to be a DFSORT vs Syncsort thing it’d be on the grounds of what you can do with DFSORT vs Syncsort, not their ability to cope with SMF.

Madness At Passport Control

(Originally posted 2007-05-25.)

I happened to be travelling through Heathrow Terminal 2 yesterday and the day before. Two incidents that made me scratch my head, both related to Passport Control…

  • On the way out there are two Passport Control desks, just past Security. They are set a few feet apart and people can go either side of either of them. For once there was someone on duty, but only one officer. At the head of the queue is a man who asks the Passport guy “Who are you? And can you prove you are a Passport official? You could be anyone.” The debate ensues for a few minutes, with the queue getting longer. Finally the official pulls out his badge and shows the man. The man’s claim is that there were no signs up announcing this as a passport checkpoint.

    Meanwhile some other passengers go for the gaps between the desks. Finally the passport official waves everyone through the gaps.

  • On the way back in there are (as most people know) two queues: EU and non-EU. The EU queue is long and there are 4 people on duty at the head of it. There are more people on duty at the head of the non-existent non-EU queue. Nobody seems to think to let some of the EU travellers come over to the non-EU queue. Next time this happens I think I’ll try the non-EU queue and see what happens.

So this seems like chaos to me. Perhaps it’s just as well that Terminal 2 is being pulled down next year. On the front of the building is a nice sign saying so (tho’ I did already know this: A taxi driver told me last year so it must be true.) πŸ™‚ Also on the sign is an artist’s impression of the new Terminal 2. It looks like a giant inflatable mattress without jetways. I wonder how you’ll be expected to board your plane: Bounce onto the mattress and bounce in through the open door? πŸ™‚