(Originally posted 2005-06-13.)

I’d made the assumption (in my code) that IRD would vary offline the top logical engines when it decided to vary an LPAR’s logical engine offline.

In fact that isn’t the case. I have data from a US company where it’s engines 2 and 3 (out of the range 0-11) that tend to be offline more than all the others.

In principle even Logical CP 0 can be offline. (Indeed some of the test systems in Poughkeepsie have no Logical CP 0.) Now this was a problem for us because I used the “Online Time” (SMF70ONT) for Logical CP 0 to determine the RMF interval length in STCK format. I use that to determine how much of the interval any Logical CP is online (using their SMF70ONT). In theory I could be dividing by 0. In practice I haven’t seen that.

Now APAR OA05798 documents a change in the order in which Logical CPs are varied offline. It talks about starting with CP 1, not CP 0.

But it does show how the implementation has already evolved and how wrong basic assumptions can be. 😦

Now I feel a new PMMVS chart coming on. 🙂 I have several times now plotted how much of the time each logical CP is online. This is, at this stage, of pedagogical interest. But clients do ask me to explain how IRD works. It is counter-intuitive.

(Originally posted 2005-05-25.)

This is mostly to say a big thank you to all the customers and other consultants at UKCMG (which has just ended). It’s great to catch up with you all. And also to spend time with Don Deese and Mike Moroz.

>

I genuinely believe in user groups and so I’d like to encourage everyone to attend next year. And, just to be balanced, to support GSE as well – I get to go to both conferences and count myself as lucky to do so.

(Originally posted 2005-05-09.)

Here’s another breakage of Accounting Trace that’s been fixed:

APAR PK03905 describes a problem with DB2 Parallel Task Rollup that causes the Group Buffer Pool Accounting data for GBP 0 to be missing. (Its title talks about IFCID 148 but in the detail IFCID 3 is roped in. IFCID 3 means “Accounting Trace base record”.)

(Originally posted 2005-05-04.)

APAR PK04803 describes a problem with RRS attaches:

The value in qwacejst may be zero or less than the one in qwacbjst for rrsaf threads that execute in multiple task. This could involve records with qwacrinv equal to 6 and/or 16.

The fix for this one isn’t due until July.

(Originally posted 2005-05-04.)

I just saw the following APAR close. PQ99525 fixes a couple of problems for “nested” DB2 access i.e stored procedures, triggers and user-defined functions (UDFs).

For anyone thinking “this is not my shop” I think you should consider that these functions are appearing in many modern applications and your Applications people probably won’t think to tell you when they start using them.

To quote from the APAR description:

1. Class 1 accounting logic for UDFs, stored procedures and triggers could capture and record small fractions of class 2 in db2 time. This would result in QWACASC and QWACAJST having non-zero values when class 2 accounting was NOT active.
2. UDFs and stored procedure require in db2 time to connect and disconnect UDF or stored procedure tasks to db2. This time was not being accounting for in class 2 in db2 times (QWACSPTT, QWACSPEB, QWACUDTT, QWACUDEB). Class 3 suspension time is clocked during this connect and disconnect processing and thus class 3 time could be significantly greater than class 2 time.

It’s hard enough keeping track of the nested time without problems like this. We described how to do this in the “Stored Procedures: Through The Call And Beyond” Red Book.

(Originally posted 2005-05-03.)

My thanks to Greg Dyck for pointing out the following on IBM-MAIN:

“The Store-CPU-Identifier now allows for a 8 bit partition identifier but the ESCON I/O protocols only allow for a 4 bit identifier. This is the reason that multiple channel subsystems must be implemented if you want more than 15 partitions on a CEC. FICON does not have this limitation.”

And to think ESCON’s only 15 years old. 🙂 I remember at the time having the EMIF support explained to me – including the LPAR number field. At the time 15 LPARs seemed an awfully large number, particularly as people were still grappling with how to configure machines with 2 or 3 LPARs for performance.

(Originally posted 2005-04-28.)

I’m reminded by a question on MXG-L Listserver that many people don’t understand how LLA works – and in particular how to interpret the statistics in VLF’s SMF 41 Subtype 3 record.

You really have to understand the exploiter to make sense of the statistics. Here’s how it applies to LLA…

LLA (when it was Linklist Lookaside, prior to becoming Library Lookaside) could cache load library directories in its own address space. You get no statistics for this behaviour. 😦

Then in December 1988, when MVS/ESA 3.1.0e was released, LLA got renamed to “Library Lookaside”. The key difference was that you could (selectively) enable exploitation of VLF cacheing of modules. Compare this with “XA” LLA which only cached directories. You enabled module cacheing by specifying in your COFVLFxx member a new class (CSVLLA) with an EMAJ of LLA and a MAXVIRT (which defaulted to 4096 pages or 16MB). 16MB was large in 1988. Now it’s puny.

Now here’s why the VLF statistics are deemed “meaningless” for (CSV)LLA: LLA only asks VLF for something if it knows it’s going to get it. So you always get something close to 100% hits with LLA’s exploitation of VLF. But you probably would regard the “successful find” rate as a reasonable metric of benefit.

It’s much better in my opinion to look at the load library EXCPs or SSCHs. After all those are what you’re trying to get rid of. I know this is old news but a few years ago I got into a dialogue with the LLA component owners. They suggested that installations start with a VLF specification of 128MB for LLA – and then worked up from there.

So that’s what we do – when it’s deemed relevant to look at this sort of thing. (I wrote this particular piece of our code in 1994.)

(Originally posted 2005-04-26.)

A friend of mine from the ITSO in Poughkeepsie is contemplating a Red Paper on Coupling Facility Capacity Planning. I think this is a splendid idea – and would be willing to contribute myself.

But I’m wondering what y’all think of the idea. And of what you’d like to see in such a Red Paper.

(Red Papers are less formal than Red Books – which we all know and love.)

(Originally posted 2005-04-14.)

Session TSP11: Performance of the TotalStorage DS8000, Speaker: Lee La Frese

Of course a new disk controller from IBM is going to appear awesome. 🙂 DS8000 is no exception.

Currently 2-way and 4-way POWER5 processors. Statement of Direction for an 8-way machine. POWER5 can go up to a 64-way.

I/O access density has levelled out over the last five years (2000 to 2005), having historically decreased (over the 1980 to 2000 period).

There’s a new adaptive cacheing algorithm (called ARC). Very effective at low hit ratios (for Open). Likely to have less benefit at higher cache hit ratios (eg z/OS workloads).

A small number of customers in the room have FICON Express2. There is a paper from Poughkeepsie on this.

Channel Measurement Block granularity has recently decreased from 128us to 5us. Which actually has been known to cause an increase in response times reported by RMF. But this doesn’t affect Lee’s performance numbers which come from the controller itself and aren’t subject to mainframe processor considerations.

PPRC (Sync Remote Copy) performance has improved dramatically. The big reductions were in ESS. But a 0.38ms cost for DS8000 over 8KM has been measured. This is not really showing what happens at distance. 303km distance showed as about 4ms and 0km as about 0.5ms. Lee noted that you could interpolate reasonably well as the elongation with distance is pretty much linear. The above numbers are at low load. Load doesn’t alter the picture very much – until you hit the knee of the curve.

Session Z16: Sharing Resources in a Parallel Sysplex, Speaker: Joan Kelley

Actually this was rather more about CF performance than actual resources. But it was very useful nonetheless.

Joan talked about shared CFs. Her “classic” scenario is a “Test” CF whose performance is not important sharing with a “Prod” CF where is important. Recall: each CF needs its own Receiver links – so need to configure accordingly.

Delayed requests can even break Duplexing – as a consequence of timing out a request.

Dynamic Dispatch puts the CF to sleep for 20ms at low utilisations, less time for higher utilisations.

A good design point for this sharing case is to turn Dynamic Dispatch on for “Test” and off for “Prod”. (D DYNDISP shows whether Dynamic Dispatch is on for a CF. (Obviously you can also infer that from RMF as well.))

She had a good foil on responsiveness for different “Test”/”Prod” LPAR weight regimes. It showed that at low Test request rates the weights don’t matter much. At higher rates, though, weights of eg 5/95 produce much better responsiveness than eg 30/70. With System-Managed Duplexing you should set weights so that the Prod (duplexed) CF is dispatched 95% of the time – to avoid the timeouts I mentioned earlier.

With Dynamic ICF Expansion the CF can expand to additionally use a shared engine. If the shared engine is totally available – i.e no other LPAR wants it – the performance is close to that additional engine being dedicated.

Because ICF engines, IFL engines and IFA/zAAP engines share the same pool it is possible for an IFL or IFA to acquire cycles at the expense of the ICF LPARs.

There were several foils on CF links. I’m going to have to learn up on this as well: It’s got a lot more complicated. 🙂

Session ZP12: DB2 Memory Management in a 64-Bit World, Speaker: Me

This went reasonably well. I did get one question which was about what value to set MEMLIMIT at (which I think translates into what REGION value to have for the DBM1 address space). At present the answer has to be “I don’t know.” 😦 That’s because I don’t know if Extended High Private users can expand down into memory below the REGION value. If that makes any kind of sense. 🙂 I clearly need to research how REGION interacts with Extended High Private (which is what DB2 uses – mainly).

Session Z20: What’s New in DFSORT?, Speaker:Me

A small but reasonably engaged audience. One good question was essentially “if I’ve set my DFSORT parameters in particular ways years ago which do I need to change now?” Basically it would take a review of the parameters but most of them wouldn’t need to change. But some of newer ones are worth looking at to see how helpful they could be.

(Originally posted 2005-04-13.)

Session ZP05: Parallel Sysplex Tuning, Presenter: Joan Kelley

Joan gave a good summary of CF levels and general-but-detailed tuning recommendations.

Enhanced Patch Apply on z890 and z990 allows different CF levels on the same CEC. But a customer suggested that in their environment re-IPLing one of the CF LPARs brought it automatically up to the newer level. Joan will try to reproduce in the lab.

CF Level 14 dispatcher changes are to help with scaling to larger request rates. For example duplexing CF to CF signalling does not get stuck behind other requests.

SMF 74-4 CF utilisation no longer includes polling for work. This is a more realistic view of CF busy. SMF 70 CF
LPAR busy is not changed.

CF busy ROTs based on at what utilisation the service time doubles.

Joan positioned duplexing as having the most benefit for structures that don’t support rebuild. Also for IXGLOGR
structures duplexing reduces the need to have staging (disk) data sets, which should improve Logger performance.

Joan offered a rule of thumb of duplexing recovery being 40 times faster compared to log recovery, and 4 times
faster compared to rebuilding the structure. Of course this comes at a cost but Joan reiterated what we already
know: Smart installations have some “white space”, especially on CPU and memory for the failure takeover case.

ISCLOCK is a good structure to do performance tests with as it’s the fastest structure because requests carry no
actual data.

IC Links (internal to the processor complex) are processed by any physical processor – whether Pool 1 (CPs) or Pool
2 (ICF/IFL.zAAP). Two links are usually enough. Limit the number to the sum of Pool 1 and Pool 2 engines – 1. More
than that can cause performance degradation.

If you increase the ISGLOCK structure and get no more entries look at APAR OA10868. It fixes a bug.

Joan mentioned that XCF traffic is increasing in general – because of new exploiters. So it might be worth keeping a
closer eye on it.

Session TSP08: New RMF Measurements for the DS6000 and DS8000, Speaker: Siebo Friesenborg

Siebo used to work in our worldwide consulting team but now he works for the Storage product division.

If you haven’t seen Siebo present I suggest you do – whatever the topic. Just take your sense of humour along with you. It’s wonderful to listen to / watch him. And Siebo was on top form today.

If you specify ESS in RMF 74-8 records will get written with a series of counters for ECKD, PPRC or SCSI. NOESS is
the default. Siebo strongly recommends you specify ESS.

74-5 enhanced Raid Rank statistics to give numbers for each physical volume – for DS6000 and DS8000 (2107).

Session G10: zSeries vs UNIX vs AMD/Intel Platforms: Which one is best?, Speaker: Phil Dalton

This was a cheering marketing presentation but with lots of good information. Here are a just a couple of factoids:

In the last few years the zSeries application portfolio has grown dramatically (particularly if you include Linux on
zSeries).

Phil knows of customers with in excess of 8,000 Linux servers under z/VM on a single machine. Obviously it’s not
nearly as many as the “gee whiz” demo we used to do. But it is a real customer having a mind-boggling large number
of Linux servers on a single footprint.

Session TSS18: DFSMS Catalog Overview and Diagnostics, Speaker: Becky Ballard

This was for me a good review of ICF Catalogs. (It’s been a while.)

Catalog Address Space (CAS) normally is ASID 000C or 000D. If not it’s not necessarily a problem but it will
indicate a restart. Restarting CAS actually is only minimally disruptive: It shouldn’t harm users that already have
a data set open. Catalog requests will be redriven by CAS on restart if they were in-flight.

Becky uttered the words “experience the IDC3000I message”. I have nothing against that particular message – I just
thought it was a lovely turn of phrase.

LISTCAT basically formats the VVRs (VSAM Volume Records). There are two types of VVR: Primary for the cluster and
secondary for each component. LISTCAT draws information from both.

F CATALOG can be very useful in checking maintenance levels, performance information, etc. Also can actually modiFy things. 🙂 You can even modiFy the CAS service task limit – but I’m not clear why you’d need to.

Informational APAR II10752 gives good advice on Catalog performance.