(Originally posted 2008-06-10.)

I had to eat a little bit of humble pie today – because I made an elementary mistake. I’m going to share it with you – to save you making it, too. 🙂 And I’m very sure I’m not going to make it in a customer situation. (Residencies are great for making mistakes in a safe environment.) 🙂

In CFLEVEL 15 and z/OS Release 9 RMF you get a new field at the structure level: R744SETM.

This is the CPU used in the Coupling Facility to process requests to an individual structure. When you add the individual R744SETM values – for all the structures in the Coupling Facility – you get the same value as the sum of the R744PBSY times for all the processors. This is because R744PBSY is defined as the time spent processing requests. (You get this by individual coupling facility processor.) If you were to calculate a capture ratio using R744SETM and R744PBSY you’d always get 100% – so it’s probably not worth the bother. 🙂

Normally one calculates Coupling Facility busy using the formula 100*R744PBSY / (R744PBSY + R744PWAI) and summing over the processors. I’m beginning to think that 100*R744PBSY / SMF74INT is a more useful calculation (and it collapses down to the usual formula for dedicated processors).

So now to my mistake…

I wanted to compare the CPU time by structure to request service times. You can do this if you compare R744SETM to R744SSTM+R744SATM. (SSTM is the sum of Sync service times and SATM the same but for Async.) So I did this comparison and carefully noted that R744SETM is a 8-byte floating point number and the other two are 8-byte integers but that all three are in microseconds over the interval. What I failed to do is to realise that you need to add up the service times over all the z/OS systems connecting to the structure…

R744SETM is for all requests to the CF structure. The others are by z/OS system.

Why does this matter? Because I suffered the embarrassment 🙂 for several fraught hours of not knowing why the CPU time was longer than the elapsed time. 🙂

There are in fact a couple of cases where CPU time isn’t included in the service time. I intend to write those up in the book. But basically they are ones where z/OS has been given the “request complete” signal but there is still some processing to do. In these cases we continue to accumulate CPU but not service time. (And, yes, these cases are logically OK.)

So, I don’t feel too bad about my mistake, particularly as it led to some learning. And I look forward to making the same sorts of mistakes with Coupling Facility Cache Statistics – which are also collected at the “all systems” level. 🙂

(Originally posted 2008-06-04.)

Sometimes reading the SMF manual buys you less than you thought. I had completely ignored a nice little field – and the code I inherited to analyse SMF 74 Subtype 4 Coupling Facility data ignored it as well.

R744SLEC in the Request Data Section for a structure is described by the text “Lock structure only: lock table entry characteristic”. Can you guess what it is? 🙂 If I told you it was a one byte integer would that help?

If I had bolded the word “characteristic” would that have helped?

Probably only if you knew something about floating point numbers. I confess to knowing little about them other than the fact there is a “characteristic” and a “mantissa”. One’s the “power of n” thing, and the other’s whatever you multiply it by. But which is which? Anyhow…

So my first guess is that this field is the number of bytes in a lock table entry – plus one. I see values of 1 and 3 in it. So 2-byte entries and 4-byte entries. Right? Wrong! …

My friend “SuperMario” 🙂 points out to me that I’m looking at a GRS Star ISGLOCK) structure when I see the “3” and further that GRS Star always uses 8-byte lock table entries, no matter how few members (systems) connect to it.

The penny drops…

2³ is 8, right?

So the “characteristic” is really in this case the power of 2 that turns into the number of bytes in each lock table entry. So “3” means 8 bytes (as I just said) and “1” would mean 2 bytes (which figures for the plexes I see it for, being relatively small). So there’s that word “characteristic” again, and the field description (though terse) seems to make sense. 🙂

Now, why’s this important?

For two reasons:

• The installation has a choice of how many bytes to make each lock table entry, depending on the maximum number of lock table entries they want to cater for – without a rebuild. So, too wide a lock table entry and you waste lock structure space – potentially massively. Too narrow and you can’t connect all the systems (or e.g. DB2 members) you want to. Usually not a difficult trade off. But now I can wade into the discussion in a customer with evidence. 🙂
• The almost-neighbouring field R744SLTL gives the maximum number of lock table entries. Multiply the two together and you get the size of the lock table portion of the lock structure. The remainder – R744SSIZ minus the lock table entry size – is for record portion of the lock structure. There are various ways to make a hash (pun intended) of this. Which I won’t bore you with here. The point is you can do this calculation, armed with what the characteristic is.

Well, I thought this was a step forward in our understanding of how lock structures were actually allocated.

On the morrow (otherwise known as “presently”) I’ll write this up in the Redbook.

I’m taking the slightly unusual stance of assuming the readership can get the SMF manual out and follow along. Or at least wave around an SMF field name like it’s their “new best friend”. 🙂 Actually I’m using field names for clarity. There are – as I point out in the Redbook – at least 4 valid size fields for a structure. It’d be nice to be clear which one we’re talking about.

So we’re having fun in Poughkeepsie, going deep into the instrumentation. And we’re having nice discussions about what all the fields mean (and what they don’t mean). Here’s an example of what they don’t mean…

Field R744SSTA sounds like it ought to mean the number of Sync requests converted to Async. But in fact it’s almost always zero, despite the z/OS R.2 Dynamic Request Conversion – which we know goes on all the time. How do we know? I’m not sure really, except that in many situations we see plenty of Async requests where the exploiter is highly likely to have requested Sync. So this field turns out not to be the R.2 algorithm “in play”. It’s the older, possibly best dubbed “Static Conversion” algorithm. So, such things could easily trip you up. Or maybe it’s just me that stands to be tripped up. 🙂

Like I say, we’re having fun here in Poughkeepsie. 🙂 And if you happen to think this sort of thing is fun a residency could be right for you. Now, you might think as a customer (or other non-IBMer) you don’t have the possibility of being selected. In fact the “wall of fame” has photos of quite a few non-IBMers on it. No, I don’t know how it works with non-IBMer nominations. But it seems to be a non-zero-yield process.

(Originally posted 2008-05-27.)

The other three residents are busy doing extensive setup work – and we hope to have some nice measurements later on. (I’m not sure how much later on there actually is, mind.) 🙂

So I have no RMF or DB2 SMF to play with yet. 😦

>

Meanwhile I’m beginning to come up to speed again after the glorious experience of recovering from moving five time zones to the west. 🙂

So I actually got to writing today. And it’s the first actual content.

So I’ve concentrated on getting the structure of what I’m writing right. While my brain continues to accelerate. 🙂 I wonder whether I should use my brain in its present state to do the important work of defining the structure. After all that sets the tone and affects whether the material is accessible or not. Or whether to write some “disposable” content. After all it’s the content you want to read. But “disposable” in that I’ll probably consider today’s content trite and dull.

So I have written some stuff about the SMF records needed to do performance analysis for Data Sharing. This I consider to be “no brainer stuff”.

I’ve fretted a bit about comprehensibility and UK vs US English and whether I’m saying anything useful at all. :-)And how to get Framemaker to do my bidding (and whether that’s the right bidding). 🙂

In the end the best advice is to just get writing and to accept that much of the early stuff mightn’t survive.

And having recovered from almost hosing my Thinkpad last night I should just be grateful to be able to write at all. 🙂

And, finally, Twitter has proved today to be a useful resource for throwing out DB2 questions and getting them answered (with actual discussion). You can, as always, find me here. And if you do read Twitter you’ll discover the standard is pretty low – if my contributions are anything to go by.

(Originally posted 2008-05-23.)

So, I’m about to start writing. At last!

As a team we had a discussion yesterday about how to deal with performance numbers. One of the roles I’m playing on the team is “the guy who writes about performance numbers”. So we came to the following conclusions:

• We really can’t talk about products made by other people – such as SAS/MXG or TMON for DB2. That’s right out – because of our basic provenance and sponsorship.
• It’s not very helpful to customers to talk about eg Tivoli Performance Reporter (or whatever it’s called these days) much – as it’s not an assumed given. So it wouldn’t really help a SAS/MXG customer much if we talked about such stuff. Remember: This isn’t a Redbook about reporting tools. It is about how to manage Parallel Sysplex Performance. So we need to do that in a way that’s accessible to all customers.
• It seems to us reasonable to have one or two examples of RMF Postprocessor reports: We do assume access to RMF or at least the SMF record types that RMF produces – and CMF does do a pretty good job of creating the same records (though there may be some differences). So we don’t think we’re disenfranchising anyone with that.
• More than one or two RMF reports is unhelpful and makes the book unnecessarily turgid and besides it’s tough to format Postprocessor reports so they fit into a book page.
• SMF record fields are canonical in that they’re high up the data flow chain. So we think it’s fine to mention them – so long as we do it in a way that makes them comprehensible.

Taking all that into account we propose the following approach:

• Have a section that talks about performance numbers – including their sources and perhaps provenance. And in this section it’s OK to include a small number of RMF Postprocessor examples and maybe Omegamon XE for DB2 examples.
• For each test run that we document to use the field name or some alias for it. We’re assuming at this point that you’ve read and understood the “performance numbers” chapter.

The aim at the end of the day is to have a Redbook that works for you. So I hope this structure works for you. If you have thoughts on it feel free to comment below. Or to email me or comment at me via Twitter. My handle there is “MartinPacker”. But you’ll probably have to follow me first and thus nudge me into following you. As I think I’ve said before, I’d like to use the new media – eg Twitter and this blog to make the creation of this Redbook a more interactive and responsive affair.

(Originally posted 2008-05-20.)

I feel like I’m on “home turf” at UKCMG… My use of English can be looser 🙂 and it’s so nice to catch up customers with whom I’ve a long and (I hope) fruitful history. I did my (slightly updated) “Memory Matters” and “Much Ado About CPU” presentations – and I really do lose track of how much change there’s been in each of them (though it seems to me I fiddle with them A LOT. 🙂 Actually I sat in a couple of presentations from Glenn Anderson which I HAVE seen before and actually technical topics DO bear repetition. And Glenn did, in my opinion, a GREAT job of the two I saw: One on Websphere Application Server Performance and the other on Unix System Services Performance. (So much so that I’m making sure I read them on the plane(s) tomorrow.)

It was also nice to see an external view of DB2 Version 9.

I’m conscious it may be time to either radically shake up my existing presentations or to do some new ones. I’ve an idea for ONE, sort of inspired by Glenn’s presentations. The working title is “z/OS Application Zoo”. Kinda like the “Particle Zoo” to my perhaps unfocused mind.

And I think there was some value in my Twittering in the sessions. Folks, you can follow my (perhaps marginally useful) Twitterings under my handle: MartinPacker. And I hope some of YOU will sign up to Twitter. I know of at least one customer and two very respected DB2 pundits who have: Willie Favero and Craig Mullins. (And just today a very good friend of mine who left IBM last year signed up – so it’s a great informal way of staying in touch.)

So, onwards:

Tomorrow sees me fly to Poughkeepsie to work on a Redbook on Parallel Sysplex Performance. I hope to try some of the material out here (and to use Twitter to ask questions or toss out odd thoughts that might influence the course of the book).

And I’ve already forewarned many of my Poughkeepsie friends of my imminent arrival… So they can clear out of town. 🙂 Seriously, if there ARE things that need raising with developers in Poughkeepsie feel free to contact me. Face to face there is SO much better for such discussions. But TRY not to abuse the offer. 🙂

(Originally posted 2008-05-09.)

I’m going to try something different this year – and composing this using OpenOffice Writer on my new ASUS EEE PC while flying home is certainly different. But what’s new is not documenting all the bits and pieces I learnt. Instead I’m going to give some impressions:

• On Monday afternoon we had an enforced evacuation from the conference centre and much of the environs – because of a World War II bomb (that the local paper said was dropped by the US Air Force.) I think everybody was forced to tour Dresden  – which is well worth visiting. It’s a great cultural city.
• Next time I’m going to ditch the jacket as it made me think I had to drag my backpack on wheels one mile each way each day. Far better to carry it – except in San Antonio last year when it was far too hot and humid.
• Next time I’m going to follow the conference instructions and book my hotel well in advance so I don’t have to do the long commute on foot each day. Yeah right. 🙂
• Next time I’m going to be fit enough to go the distance. Yeah right. 🙂
• It’s handy to sit in sessions you already know a fair amount about – to get you thinking. Example: Peter Enrico’s excellent presentation on WLM samples. (I think my code needs working on as a result of that.) 🙂
• There is lots going on in the “Web 2.0 and z/OS” space. Thanks to Kevin Keller and Holger Wunderlich for that.
• Perhaps I shouldn’t assume everyone knows what CFCC Dynamic Dispatch is. I got a question on that.
• There might be “unintended consequences” in z10 HiperDispatch disabling IRD’s Logical CP Management. And there was a good question on that.
• Colin Paice from Hursley really is a very good speaker.
• Twitter is useful as a way of raising consciousness about conferences going on -and it’s also good at helping people find each other.
• It was good to run into Aneel Lakhani in person. He’s in the category of “I don’t know precisely why I know him but I know him from BlogCentral and I’m glad I know him” people. (BlogCentral is IBM’s internal blogging site.)
• To the customer who said she was doing a thesis on Web 2.0 yes we really do encourage all IBMers to dive head first into Web 2.0 and social networking in general. Yes, we have guidelines (which I contributed a little to) but nobody finds them restrictive. I wish more companies were like this.
• It’s great to have a tour guide who grew up in this part of the world and who went to university in Dresden. Thanks Barbara!
• My manager is following me on Twitter. But that knowledge doesnt seem to have inhibited me in the slightest. 🙂 I’m still goading him into (hopefully) joining me there.
• I got a chance to forewarn lots of Poughkeepsie folks I’ll be in town in a couple of weeks. If they scarper, given this amount of notice, I will take it personally. 🙂
• I found lots of customers considering doing Data Sharing at non-trivial distances. So the Redbook ought to reflect that.
• People agreed with me that the CF Structure CPU support in CFLEVEL 15 and RMF will make life much easier for Parallel Sysplex customers.

So, roll on UKCMG in only a few days time. When I won’t have to watch my language so carefully. 🙂

And my experience of composing this on the EEE is:

• The machine is nice and small – so it doesn’t feel cramped on the tray table.
• The keyboard will take some getting used to as it’s small (but the screen, though small, is just fine).
• OpenOffice is a new package to me so I’m finding its quirks. (But I have to find them somehow anyway.) 🙂
• OpenOffice’s “Export to HTML” seems to export in a fairly style-rich way when I’d rather have had minimalistic HTML. I expect I can easily fix that.
• OpenOffice’s “Export to PDF” works really nicely (and quickly). I was only messing about when I tried it but I was impressed.

I really do like this EEE. It’s a cheap Linux machine that’s easy to do LAMP stuff on. (On the way to Dresden I learnt a bit of Ruby.) But this has been the first (and I think successful) test of using it to do real work. If you can call blogging real work. 🙂

And the motivation for trying out Linux and OpenOffice is because I expect to be using them on my main machine within the next month or two. I have to start on the learning curve somewhere (or at least some time).

And finally thanks to all the customers and fellow presenters who made this a great conference for me.

(Originally posted 2008-05-03.)

I wonder how many of you recognise the terms “READ_COCLASS”, “WARM”, “RFCOM”,”WRITE_DATALIST” and “CASTOUT_DATALIST”.

Granted that in that list are two pairs of terms that I believe to be synonymous, but all five are relatively recent enhancements to Parallel Sysplex infrastructure that have a positive impact on DB2 Data Sharing performance (or at least the management of Performance).

As I mentioned in a previous post I’m going to Poughkeepsie soon to work on a Parallel Sysplex Performance Redbook – which will deal with topics such as “Data Sharing at a Distance” and “New Parallel Sysplex Technologies”. In preparation for this (not that I do much preparation for such things) I read through the various sections on Data Sharing Performance in the various DB2 Performance Topics Redbooks, starting at the beginning – with Version 4.

It might be significant that I don’t really understand these Coupling Facility commands. But then again it might not be… given that there are (without being self-deprecating) far better proponents of DB2 Data Sharing performance. In any case I feel the material that’s out there falls short in terms of describing when these commands are used and what benefit they bring (and what the downsides, if any, might be. So I hope to write some material in this area and ask (what I hope to be) searching questions of Development.

Now, these commands were first exploited by DB2 Versions 6 and 8. So the material might end up on the cutting-room floor. After all we’re all interested in System z10, CFLEVEL 15 and DB2 Version 9. Right? 🙂 What I’m hoping to do – in any case – is to share some of that material through this blog. If I succeed in doing that I hope you’ll help me by feeding back on the blog entries. Corrections, additional questions, etc.

And the fun really begins on May 22nd. Just after UKCMG.

(Originally posted 2008-05-03.)

Tomorrow I fly to Dresden for the annual European System z Technical Conference. It’s at just the time of year when I can decide in my head that Spring has sprung and that Summer won’t be long coming. 🙂

This is a great conference and I always enjoy attending it (and presenting at it). Talking of which, I have three sessions this year:

• Much Ado About CPU which has a fair modicum of new stuff, including material on System z10.
• Memory Matters in 2008 for which the updates are mainly in the System memory area, particularly the 1MB Pages support in System z10
• DB2 Data Sharing Performance For Beginners which actually isn’t much changed. (As I will be spending late May and much of June in Poughkeepsie working on a Parallel Sysplex Performance Redbook maybe the next iteration will be substantially changed.)

I’m also looking forward to a number of other sessions. No doubt I’ll get to say a few words in those as well. 🙂 In particular Kevin Keller’s session on “Web 2.0 and z/OS” has caught my fancy. (As you probably know I really do feel we in the mainframe community ought to pay attention to things like Web 2.0 and bid for z/OS to run the infrastructure.)

So I hope to see lots of you in Dresden.

(Originally posted 2008-04-27.)

As some of you will know IBM has a twice-yearly “HackDay” programming fest. And the latest – HackDay5 – was last Friday. Couched as a competition it isn’t really… It’s a heck of a sight more cooperative than that. (Else I wouldn’t have got involved.) I’ve participated in all five we’ve had over the past two years. And it’s been great fun. This time – for the second time – I’ve physically gone to Hursley to take part in the local event.

New for HackDay5 was the “zHackDay” initiative – which I take “responsibility” 🙂 for…

The idea was that mainframe-related hacks were labelled as such. Partly to goad mainframe folks into participating, and partly to point out to non-mainframers that we’re just as capable of doing fun hacks as everyone else.

So, was my project exciting?

Probably not – but then I wasn’t personally aiming to set the world alight. I’m more interested in getting useful stuff done that I wouldn’t ordinarily get around to. So my project is basically to map the SMF 88 System Logger records.

This has been done before – and indeed there’s a z/OS-supplied utility to report on System Logger from Type 88 data.

So why would I “reinvent the wheel”?

The main reason is that it allows me to put the data together with other things. Such as SMF 74 Subtype 4 Coupling Facility data. If a logstream uses a coupling facility structure the SMF 88 record tells you the structure’s name. (This is similar to a recent trick where the 74-2 XCF data tells you the structure name and my code looks up the 74-4 view of the structure.)

Actually I know relatively little about System Logger – but I’m learning fast. And that’s the other reason I did this hack… To enable me to learn.

Finally, if I discover something’s missing from the data I can use real customer situations to discuss with Development what’s needed.

So, how far did I actually get? I got the Type 88 Subtype 1 (Logstream Activity) data mapped and summarised at the (e.g.) hourly level. And some very simple reporting. I haven’t (yet) done the same for the Type 88 Subtype 11 (Coupling Facility Structure) data. But that should be easy and straightforward.

So, if you’re sending me data I’m going to be asking you to send me your Type 88 data as well. I hope you’ll find what I do with it to be useful.

(Originally posted 2008-01-28.)

We’ve been issued new guidance:

• Delete SPAM rather than mark it invisible.
• Disallow posting comments after, say, 30 days.

The latter appears to work at the individual entry level. Three of my entries have become particularly troublesome (for some unknown reason). So I’ve set them to “no more comments”. If you find a comment of yours bounced then please do let me know. I’m sure we can fix it.

Deleting all the SPAM comments that have arrived so far for these three posts has been a right pain in the neck. I’m sure a little Firefox Extension programming could make it easier. 🙂

And now back to some real content. I hope.