(Originally posted 2007-01-26.)

I presented to the Large Systems GSE meeting in Dudley on Wednesday on “DB2 Data Sharing Performance For Beginners” and an interesting question came up…

“At what distance does performance begin to suffer when moving machines in a parallel sysplex apart?”

As it happens I am dealing with exactly that question in a customer right now. It seems it’s a popular thing to attempt. (I know: One user group question and one customer engagement do not a trend make.) 🙂

So I obviously need another foil or two in my presentation – as if it wasn’t long enough already. 🙂 Here are some early thoughts. (Actually you’re not the first set of guinea pigs – as I drafted something along these lines for my internal IBM blog.) 🙂

The “standard answer” is that performance begins to deteriorate above 10km. That’s fine – but I think I want a little more detail than that…

Disk Performance

The question of disk performance at a distance was raised in the GSE discussion. It’s my view that, although the speed of light does come into play here, it’s less of an issue by far than for coupling facility (CF) requests: An elongation of 30 microseconds is far more serious for a CF request than for a disk I/O. (And there are probably far more CF requests a second than there are disk I/Os.) Ignoring protocol differences – such as the number of exchanges per request – disk is probably OK for one or two more orders of magnitude of distance. But don’t ignore disk considerations.

Now let me talk about CF requests…

A synchronous request to a coupling facility accessed using Internal Coupling (IC) links can complete in, say, 30 microseconds. That’s because the link is rather more logical than physical. Sync requests to close-at-hand CFs using Integrated Cluster Bus (ICB) links are longer than that. Sync requests to CFs using ISC links are going to take longer still. ISC links allow much greater distances than ICB links (>> 7 metres). So, just to be using ISC links probably means a step down in request performance. And as the ISC link gets longer the speed of light comes into play. I also think it’s important to note that a CF request involves a number of signals up and down the link.

Coupling Facility Request Arithmetic

So here’s some easy (but probably inaccurate) maths:

Suppose each request requires 10 signals of some sort or other. And suppose the speed of light is 300,000 km/sec. Every kilometer of extra distance requires 10km of extra signal travel distance. Which takes about 35 microseconds (assuming signals propagate at the speed of light – which is the best case). So each request has an additional 35 microseconds of service time.

Further note that a synchronous request causes a z/OS processor engine to spin – so the time of a synchronous request also causes an equivalent CPU time “wastage”. z/OS, since Release 2, has used an adaptive algorithm, converting sync requests into async requests as appropriate. This helps minimise the coupling CPU cost of z/OS engines. And, as the distance increases, z/OS is more likely to convert your request to async.

Async requests don’t cause the z/OS processor to spin but they do take longer to complete. So longer distances may well have a knock on effect on request response time.

(You can measure the times and the sync vs async request rates – at the structure / z/OS level – using the RMF Coupling Facility Report data (SMF 74-4).)

So the basic conclusion is that distance does matter, and probably significantly below 10km – at the request response time level.

Applications

Now let’s think about what that means for applications…

If the request is from a z/OS image to a proximate CF structure the response time is going to be lower than if it were to a remote CF. And the difference is obviously going to increase with distance. But the impact on an application (such as DB2 (and its applications in turn) depends on the access rates and characteristics. For an application that always uses a local CF structure its performance will be better than one that uses a mixture of local and remote accesses, and still better compared to an “all remote” access pattern. And the fewer CF accesses per “transaction” the lower the impact.

But you might not be able to choose “all local” access patterns. And you might not have much choice about access intensity – but the latter is a major DB2 Data Sharing tuning theme. So don’t discount that possibility. In any (DB2) case you can use DB2 Accounting Trace to monitor and tune DB2 applications’ use of CF resources.

CF Structure Duplexing

Let me conclude by talking about structure duplexing…

First we need to review how CF Duplexing works. (I’ll use the DB2 Data Sharing example to illustrate it.)

System-Managed Duplexing

There are two copies of the structure – in separate coupling facilities, on separate machines.

For every request both structures have to perform the same amount of work, and the two CFs coordinate via a dedicated link. The request’s completion is signalled only when the request has been processed in both CFs.

One request is always to a remote CF. The other may or may not be. So in all cases a request is performed at effectively “remote speed”. Which, as I said, elongates with distance.

DB2 Locking (via the LOCK1 structure) and GRS Star are good examples of structures affected by this.

User-Managed Duplexing

There is only one exploiter: DB2 Group Buffer Pools (GBPs).

In the User-Managed case only the writes are processed by both copies of the structure… An async write to the secondary is followed by a sync write to the primary. When both have completed the request is signalled as having been completed. So writes always go at remote speed.

Reads are always from the primary. At an individual z/OS image level this could be remote or local depending on which machine the z/OS image is on and which machine the primary structure is on.

So the effects here are perhaps less severe. “Perhaps” because it all depends on access rates and patterns. But, for a read-only subsystem, local to all the GBPs all requests are going to be local. For another read-only subsystem accessing the GBPs remotely the CF response times will be higher. (So perhaps balancing the GBPs across the 2 CFs might help keep the application response times consistent.)

The bottom line with duplexing is that it stands to increase CF response times and sensitivity to distance. But equally you can tune DB2 usage, perhaps reducing GBP and locking traffic.

I’ve deliberately written this in a “design in public” style as I’m seeking early customer experiences and perhaps corrections to my thinking. I suppose that’s one of the things blogs are good for.

(Originally posted 2006-12-13.)

I’ve just been looking at some fascinating SMF 70 records. They come from a System z9 EC S54 machine and the LPAR they’re from is running z/OS Release 8. I say “fascinating” because:

• I’ve not seen an S54’s data before. (This is the largest z9 machine – with all 4 processor books fully populated with engines.)
• There are changes that I asked RMF (and PR/SM) Development to add into Release 8 that I’m delighted to see in the data. (It’s nice when a developer says “sure thing, Martin, it’ll be in Release 8” but it’s even nicer when you get to see Production SMF data with the changes in.)

Some of the changes described below are not, in fairness, at my behest.

Engine Counts

The final section of the SMF 70 record used to simply be a lookup table for “pool number” versus “characterised processor type name”. So, for example zAAPs would be in Pool 3 – for z9. (For z990 they’re in Pool 2, alongside all the rest of the non-GCP processor types. But modern machines often have enough logical processors (and LPARs) to cause some of the Logical Processor sections in the record to overflow to second (and perhaps third) records. This means working out the configuration of a machine from SMF 70 requires you to process several records as one. Certainly our code (and probably that of other programs) doesn’t process them as one. So in Release 8 there’s a very useful enhancement…

Each of the “lookup table” sections now has a count of the number of each engine type. So, for example, the ICF lookup table section has the number of ICF processors on the physical machine.

Hardware Model

I asked for this a while back and I thought I had it in z/OS Release 7. However I’ve seen a number of sets of R.7 where this field isn’t in the record. But in this case the field is populated with “S54”. I assume this has little to do with z/OS Release 8 vs Release 7, but rather more to do with microcode levels on the processor. And the importance of this is that I can be sensitive to the number of books installed on the machine when recommending things like upgrades.

Machine Serial Number

This, which I asked for, has two purposes:

• I can look up the Vital Product Data (VPD) for the machine in one of our internal databases. (Customer Engineers have access to a nice tool from Montpellier called VPDFWIN, which formats VPD into a useful display.) Amongst many other things it tells me the book count (see above), the number and size of memory cards on each book, and the I/O subsystem features on the machine. So, again, I would hope to do a better consulting job.

• I can correlate the SMF 70 view of CPU with the SMF 74 Subtype 4 view of CPU. (SMF 74 Subtype 4 gives you a wealth of very useful information about Coupling Facilities.) So, for instance, I can tell which machine a CF is on. So I don’t have to ask you the stupid question. 🙂

  Now, I only have SMF 89 and 70 for this machine so far, so I can’t also test if the machine serial number has shown up in the 74-4 record as well. I’m sure it has – as I asked for it at the same time as for it to be in the SMF 70 record.

I concede these are small “fit and finish” items but they do make the SMF 70 data just that little bit more useful and usable.

(Originally posted 2006-11-30.)

In z/OS Release 8 Real Storage Manager (RSM) implemented brand new algorithms. One of the more significant changes was the adoption of a “new style” UIC value. This behaves much more like the old Expanded Storage Migration Age. Accordingly the old UIC value (SMF71ACA) isn’t really appropriate to use anymore. Or at least it won’t behave the way it used to.

And before I go any further the SMF Type 71 record is the one that accompanies the RMF “Paging Activity” report. But most people process it into a database and graph its fields from there.

Now, the point of this blog entry is to alert readers who like to look at low-level SMF records that the Paging Data section in SMF 71 has been extended from 1040 bytes (ending at SMF71AFB) to 1120 bytes. The new fields can be summarised as:

• Information on shared pages, typically in 8-byte floating point fields.
• New numbers on UIC values, as 4-byte integers.

It’s the latter I want to focus on…

There are a whole host of these UIC fields, giving much more detail on how UIC varies through time. When the documentation refers to “Average current system UIC during the interval” (SMF71UAC), for example, it’s important to bear in mind that the interval consists of some number of samples. So the “Average” refers to averaging across the samples, and the “current” refers to the value at the end of one of the sample periods. It’s rather confusing terminology – but that’s the nature of the beast.

I expect, in a subsequent blog entry, to talk more about the new RSM algorithms themselves.

(Originally posted 2006-11-22.)

In Mainframe v-Business Boas Betzler threw out a challenge: “Demonstrate SecondLife talking to CICS”. I’ve picked this up on behalf of IBM. Now, why on earth would you want to do that, Martin?

To get beyond the “because it’s fun” answer (which is also true) let me tell you my view on SecondLife…

First, it is NOT a game. Emphatically not. Though it is a lot of fun. By way of analogy consider the graphics and sound cards that have crept into most PCs and are certainly built into most laptops. Those were originally for gamers. Now the analogy here is that things that look like fun now will probably become standard in a few years time…

Why shouldn’t we collaborate interactively in 3-D space in a way that’s far better than the current browser and application experiences can offer? Even with the collaborative aspects of things such as Web 2.0 and instant messengers like Sametime. Well maybe SecondLife isn’t there yet but it will be.

(And at this point I should reveal I’m “Timnar Mandelbrot” in SecondLife, in case you feel you want to befriend me.)

So, we have the “better UI” aspect. Maybe compelling, maybe not.

But we also have the “filthy lucre” element, and that could be more important:

I was talking to some friends in May in Washington about how people were making real money in SecondLife. The very next day I heard of an IBMer making real money selling photographs for people to use in SecondLife. That’s a simple example. A number of companies have set up shops in SecondLife: You pay for what they sell (whether for use “in game” or for delivery to your real doorstep) in Linden Dollars. These are directly exchangeable for US Dollars.

Which brings us almost to CICS. So, if a real company is trading in SecondLife, don’t you think they’d want to do it in a robust fashion? Well, this is where the ability to script objects in SecondLife. particularly linking to the outside world (via the llHttpRequest() script function), comes in handy.

Back to Boas’ challenge…

Wouldn’t it be nice to connect SecondLife transactions, requests for information, etc. to real business systems? With all the robustness and qualities of service that the mainframe, z/OS, CICS, DB2, Websphere, Websphere MQ etc bring. If real companies are trading for what is ultimately real money in SecondLife they need this stuff. So the link to CICS is actually quite important:

So, I’ve found a CICS system in Montpellier I can use. I’ve talked to an expert in the HTTP support in CICS – who is enthusiastic about this. I’ve talked to a number of people about SOAP (and it works well in CICS but rather less well in SecondLife). I’ve created a simple scriptable object on IBM’s private Hursley Island. And I think I’m all set.

So, I’m going to be putting all the pieces together over the coming weeks. Well enough to demonstrate that you can talk to CICS transactions from within SecondLife, using “Raw HTTP”. With that basic piece under my belt I’ll branch out and demonstrate some other z/OS-based products in the mix.

Yes, it should be fun. But it should also demonstrate that the mainframe has a role to play for organisations that are going to participate in SecondLife – whether to trade there or to provide a useful 3-D experience. Probably both in fact – if you recall how the web evolved from pretty browsing to business transactions.

In fact it went further than that with some of the Web 2.0 elements: providing interactive experiences and social networking. So next time (which may be the first time) someone invites you to a meeting or lecture or presentation in SecondLife, I’d take it if I were you.

(Originally posted 2006-11-08.)

Well, I don’t know about that. Maybe this contribution to the debate won’t add much. But there must be at least one performance person out there who hasn’t heard of the discussion around RMPTTOM. But would I want such a person amongst my readership? 🙂

I’m not going to rehearse the whole story. It is pretty well understood that the old default was creaking somewhat.What I am pleased by is the way it happened…

Sitting on the sidelines of this one I saw.

• A customer reported experimenting with the RMPTTOM value and getting good results on MXG-L.
• Other customers experimented and also got good results.
• Bernie Pierce of SRM / WLM Development wrote a very fine piece on the matter in MXG-L. And I agree with everything he said in it.
• A WSC Flash on the subject, distilling Bernie’s thoughts.
• z/OS changes to the default.

You might argue that IBM should’ve been proactive in this matter. Personally, as I wasn’t involved, I can only guess that SRM / WLM developers had a hunch that the default value of RMPTTOM would benefit from revisiting. And that a customer coming along and providing proof was the catalyst. That’s genuinely a pure guess. In any case we do have to rely on input from the field and customers – to some degree. That’s what makes things like conference conversations and newsgroups (such as MXG-L) so much fun. 🙂

Of course, if you still want the technical guts do post a comment to that effect. But wait a while as I bone up on the issue. 🙂

(Originally posted 2006-05-22.)

Here are a couple more SQL processing APARs.

APAR PK22814 for Version 8 ESTIMATED SUBQUERY COST IS TOO HIGH WITH SORT MERGE JOIN.

The APAR text says Correlated subquery cost is over-estimated for sort merge join. A well-performing sort merge join plan might not be chosen by DB2. Correlated subquery cost can be overestimated for sort merge join. This may cause a sort merge join plan that performs well to be overlooked by DB2. The cost calculations for sort merge join with a correlated subquery are adjusted to produce a more accurate estimate. Good sort merge join plans can now be chosen for better query performance.

APAR PK23495 for Version 8 SOME PREDICATES ARE NOT BEING PROPERLY PUSHED DOWN INTO VIEWS.

From the APAR text: Slow access path is chosen for an SQL statement referencing a view or a table expression that contains an outer join, because the predicates are not being properly pushed down.

The bind time code in DB2 is modified to correct the incomplete predicate push down.

(Originally posted 2006-05-15.)

DB2 Version 8 introduced some very significant exploitation of 64-bit virtual storage. It would be naive to believe that was a total conversion to 64-bit and indeed installations’ mileage varies, depending on circumstances. Here are two APARs that show the journey continues…

PK21268 MOVE CURRENT PATH STORAGE OUT OF STACK AND INTO ABOVE BAR

From the APAR Comments: The working storage to hold CURRENT PATH information has been moved to above-the-bar storage. This will reduce below-the-bar storage consumption.

The word stack is significant as it is one area where the conversion to 64-bit was said to increase storage requirements below the bar. You can monitor Stack Storage using IFCID 225 DB2 Statistics Trace records.

PK21237 VIRTUAL STORAGE CONSTRAINT RELIEF IN BUFFER MANAGEMENT

This provides relief in three areas, according to the APAR description…

The buffer manager engines have been modified to use a single common above-the-bar storage pool, rather than having a separate pool for each engine.

Additionally, the default maximum for the number of engines has been reduced.

For both castout engines and P-lock engines, excess stack storage will now be released before suspending awaiting more work to do.

(Originally posted 2006-05-13.)

(Scroll down if you’re not interested in DB2)

This past week I’ve been updating our SQL Analysis code to cope with Unicode SQL statements. (If you bind a package or plan once you’ve gone to DB2 Version 8 New Function Mode the statement text is stored in the Catalog in Unicode.) These statements are in Unicode UTF-8 which is extremely similar to ASCII. So I snaffled some existing code of mine that converts from ASCII to EBCDIC and pressed it into service…

If the first 10 bytes of the statement text (minus the 2 4-byte header fields) contain X’40’ (an EBCDIC null (space)) then I consider the text to be in EBCDIC. Otherwise I deem it to be ASCII – and do the conversion. (You might ask “what about statements like ROLLBACK?” Well, it turns out that in SYSPACKSTMT there is always a trailing blank on the statement text – so that’s alright. 🙂

(In fact my code decodes all the structural information in the SYSPACKSTMT STMT column – including extracting information on host variables.)

Now, my colleagues sometimes just want to take a look at the statement text sometimes – without bothering to run my code. So here’s what I learned when creating a DFSORT job I set up for them to run, converting the statement text from ASCII to EBCDIC.

Converting ASCII to EBCDIC With DFSORT

First, a disclaimer: Certain codepoints might need slight adjustment in some countries – for example if you have dollar versus pound sterling issues.

First you code an ALTSEQ statement:

  ALTSEQ CODE=(0000,0101,0202,0303,0437,052D,062E,072F,
    0816,0905,0A25,0B0B,0C0C,0D0D,0E0E,0F0F,
    1010,1111,1212,1313,143C,153D,1632,1726,
    1818,1919,1A3F,1B27,1C22,1D1D,1E35,1F1F,
    2040,215A,227F,237B,245B,256C,2650,277D,
    284D,295D,2A5C,2B4E,2C6B,2D60,2E4B,2F61,
    30F0,31F1,32F2,33F3,34F4,35F5,36F6,37F7,
    38F8,39F9,3A7A,3B5E,3C4C,3D7E,3E6E,3F6F,
    407C,41C1,42C2,43C3,44C4,45C5,46C6,47C7,
    48C8,49C9,4AD1,4BD2,4CD3,4DD4,4ED5,4FD6,
    50D7,51D8,52D9,53E2,54E3,55E4,56E5,57E6,
    58E7,59E8,5AE9,5BAD,5CE0,5DBD,5E5F,5F6D,
    6079,6181,6282,6383,6484,6585,6686,6787,
    6888,6989,6A91,6B92,6C93,6D94,6E95,6F96,
    7097,7198,7299,73A2,74A3,75A4,76A5,77A6,
    78A7,79A8,7AA9,7BC0,7C4F,7DD0,7EA1,7F07,
    8043,8120,8221,831C,8423,85EB,8624,879B,
    8871,8928,8A38,8B49,8C90,8DBA,8EEC,8FDF,
    9045,9129,922A,939D,9472,952B,968A,979A,
    9867,9956,9A64,9B4A,9C53,9D68,9E59,9F46,
    A0EA,A1DA,A22C,A3DE,A48B,A555,A641,A7FE,
    A858,A951,AA52,AB48,AC69,ADDB,AE8E,AF8D,
    B073,B174,B275,B3FA,B415,B5B0,B6B1,B7B3,
    B8B4,B9B5,BA6A,BBB7,BCB8,BDB9,BECC,BFBC,
    C0AB,C13E,C23B,C30A,C4BF,C58F,C63A,C714,
    C8A0,C917,CACB,CBCA,CC1A,CD1B,CE9C,CF04,
    D034,D1EF,D21E,D306,D408,D509,D677,D770,
    D8BE,D9BB,DAAC,DB54,DC63,DD65,DE66,DF62,
    E030,E142,E247,E357,E4EE,E533,E6B6,E7E1,
    E8CD,E9ED,EA36,EB44,ECCE,EDCF,EE31,EFAA,
    F0FC,F19E,F2AE,F38C,F4DD,F5DC,F639,F7FB,
    F880,F9AF,FAFD,FB78,FC76,FDB2,FE9F,FFFF)

Then you refer to this translation table in your control statements:

  INREC FIELDS=(STMT,TRAN=ALTSEQ)

In this case the field STMT (which happens to be a DFSORT Symbol name) is translated using the ALTSEQ symbol table. (There is one translation table but it can be used by selected fields – those for which TRAN=ALTSEQ is specified.)

TRAN=ALTSEQ has been supported by DFSORT for a number of years. As documented in the Reformatting: Translation section of Beyond Sorting also mentions TRAN=UTOL and TRAN=LTOU, which allow translation between lower and upper case.

In our code I have placed the ALTSEQ translation table in a data set pointed to by a (separate) DFSPARM DD. That makes it easier to manage. Indeed, if you do have code page issues you can address them using several such data sets, each with their own customisation for “local difficulties”.:-)

One final note: In Production I’ll probably use IFTHEN to automate the “blank in the first 10 bytes” test I mentioned above – as you can, with IFTHEN (UQ95213 / UQ95214), reformat records in different ways dependent on the input data. That includes choosing whether to use TRAN=ALTSEQ on individual fields.

(Originally posted 2006-05-06.)

Nope, I’m not going to do a Rumsfeld. 🙂

But there are a couple of things I didn’t know (I didn’t know) about DFSORT. Relating them to you is probably OK because

• It’s not something I’m embarrassed about.:-)
• If you want to understand some of the funkier new DFSORT tricks you probably need to know these things.

Consider these DFSORT examples (which I take no credit for). They are used to deal with groups of records and rely on features introduced about a year ago (with New Function APAR UQ95214 /UQ95213).

Here are the two things you need to know if you are ever to understand these examples…

OVERLAY items are applied in sequence

By this I mean that the first field specified is overlain first, then the second, then the third, etc… Now, the significance of this (and the reason for the bold emphasis) is that the result of the first overlaying can be used in the second overlaying. Consider the following fragment from the first example (here):

  IFTHEN=(WHEN=NONE,
    OVERLAY=(152:SEQNUM,8,ZD,
       144:144,8,ZD,SUB,152,8,ZD,M11,LENGTH=8))
 

Here’s how we process this instruction (ignoring the WHEN=NONE IFTHEN piece):

1. We overlay positions 152-159 with a DFSORT-generated record sequence number (and more on that later in this entry).
2. We overlay positions 144-151 with the current value in 144-151 minus the just-generated value in 152-159 (not whatever was in 152-159 before we started OVERLAYing).

And this turns out to be critical to the way this smart trick works.

Every IFTHEN stage that uses SEQNUM has it’s own separately-incremented SEQNUM counter

Consider the whole IFTHEN pipeline in the smart trick:

   INREC IFTHEN=(WHEN=INIT,OVERLAY=(144:SEQNUM,8,ZD)),                   1
         IFTHEN=(WHEN=(2,4,CH,EQ,C'RPT.'),
                 OVERLAY=(134:2,10,144:SEQNUM,8,ZD)),                    2
         IFTHEN=(WHEN=NONE,
                 OVERLAY=(152:SEQNUM,8,ZD,                               3
                   144:144,8,ZD,SUB,152,8,ZD,M11,LENGTH=8))

Each of these SEQNUMs, as I say, is a separately-incremented counter: If the WHEN condition is satisfied the OVERLAY (or BUILD) processing is performed on the record. And each SEQNUM counter is incremented…

1. This one is applied to every record so is the true count of the records that INREC processed.
2. This one is incremented only if (2,4,CH,EQ,C’RPT.’) evaluates to true.
3. This one is incremented if no other IFTHEN condition (other than WHEN=INIT conditions) is satisfied, that is if (2,4,CH,EQ,C’RPT.’) evaluates to false.

Here are some other things you might not know about SEQNUM…

• For a long time now you’ve been able to specify a starting value and an increment. e.g.

    OUTREC BUILD=( ... ,SEQNUM,8,ZD,START=5,INCR=5, ... )
  

  creates a sequence number that starts at 5 and goes up by 5 each time it’s incremented (i.e for each record it’s applied to.
• With UQ92514 / UQ92513 you can specify a RESTART subparameter. e.g.

    INREC OVERLAY=(1:SEQNUM,8,ZD,RESTART=(16,8))
  

  restarts the sequence number at 1 each time the value in positions 16-23 changes.
• You can sort on a sequence number – to preserve the original sequence of records, perhaps. e.g.

    INREC FIELDS=(SEQNUM,8,PD, ... )SORT  FIELDS=(1,8,PD,A)
  

• You can perform arithmetic on a SEQNUM field – assuming you have the arithmetic expression support built into APAR UQ90053. Now, I will take credit for suggesting that MOD be supported so that sequence numbers could be written 1, 2, 3, 1, 2, 3, … (That might allow you to subsequently separate neigbouring records by sorting on this MODed sequence number, for example.) e.g.

    INREC FIELDS=(SEQNUM,8,PD,MOD,+3, ... )SORT  FIELDS=(1,8,PD,A)
  

• Not only does each stage in an IFTHEN pipeline have its own SEQNUM counter but INREC, OUTREC and each OUTFIL statement all have their counter (or, for IFTHEN, set of counters).

So, lots more flexibility with sequence numbers than you might imagine.

(Originally posted 2006-05-02.)

Saw an APAR described as INCORRECT OUPUT 🙂 the other day.

That reminds me of a time long ago when a friend had a foil titled Qualtiy.:-)