Rexx’Em

(Originally posted 2018-05-29.)

In a sense this post follows on from I Must Be Mad – where I talked about some of the subtleties of processing SMF. In another sense it’s writing down in public a briefing I want to give one of my mentees.¹

She’s about to prototype some code to go against a record subtype we’ve not handled before. In fact we throw any records of this subtype away – today.

Rewind…

… all the way back to a z/OS 2.1 presentation my friend and co-conspirator Marna Walle gave in 2013. (I had to look that one up.) 🙂

In it she mentioned z/OS TSO REXX being able to process Variable Blocked Spanned (VBS) data. The best use case for this is, of course, SMF.

I was delighted to see this support. But I’ve done nothing with it until now.

Scroll Forwards²

Actually I have some REXX code that processes SMF but I don’t really like it: It copies SMF 70-1 to a VB (not Spanned) data set and then a REXX exec processes this data. There’s a very good reason for not liking this: It risks truncating records. SMF 70-1 records can be very long, so this is a potentially serious problem.

So, processing VBS would avoid the VBS-to-VB copy and eliminate the risk of breakage.

The rest of this post is about some of the coding techniques for processing SMF with REXX. I actually developed them while processing SMF in VB format, but they are equally valid with VBS.

Processing SMF With REXX

While REXX is reasonably fast, I wouldn’t use it for high volume record types. My use case was extracting the LPARs on a machine which are deactivated. You get this from SMF 70-1 where the Logical Partition Data Section says there are no Logical Processor Data Sections.

This is a low volume case – as 70-1 records are only generated on an RMF interval and there are relatively few of them. My code processes 70-1 in a second or two.

Record Offsets Versus REXX Variable Substrings

This is probably the area that has the greatest potential to cause confusion:

• Records begin at Offset 0, including the 4-byte Record Descriptor Word³ (RDW). So the first byte of data after the RDW is at Offset 4.
• In a REXX string the first byte is at Position 1. When REXX reads the record with EXECIO the RDW is discarded. The position is used when extracting substrings – with substr().

So, to convert offsets to positions you subtract 3.

There are a couple of approaches:

1. Keep the “subtract 3” thing in your head. (Or use a routine.)
2. Prepend 3 bytes and use position as if it were offset.

In my code I chose the former – without the benefit of a conversion routine.

Extracting The SMFID

Because the SMFID is at offset 14, and bearing in mind the “subtract 3” point, you can extract it from a record in a variable with:

smfid=substr(myRecord,11,4)

It’s a 4-byte character string – so it needs no further processing.

Parsing Triplets

A section is a portion of the record with a fixed layout – and hence a fixed length. There might be more than one of a given layout.

Sections within a record are pointed to by data structures called triplets. As the name suggests, they consist of three fields:

• 4-byte offset – how far into the record the first section of this type starts
• 2-byte length – how long every section of this type is
• 2-byte count – how many sections of this type there are

To get to the sections of a given type you use the offset and then process them linearly, using the length to extract them, and to skip to the next.

In the SMF record header is a vector of triplets. So, for example, in SMF 70 Subtype 1 the vector starts at offset 28. The first few triplets in the vector are:

• 28 ( X‘1C’ ) – RMF Product Section
• 36 ( X‘24’ ) – CPU Control Section
• 44 ( X‘2C’ ) – CPU Data Section

When I process a section I extract all three portions of the triplet separately:

/* Extract CPU Control Section position, length and count */
ccs=c2d(substr(myRecord,33,4))                              
ccl=c2d(substr(myRecord,37,2))                              
ccn=c2d(substr(myRecord,39,2))

Reminder: The control section starts at position 33 in the variable – which is offset 36.

Processing A Section

I extract the first (and only) CPU Control Section itself:

section=substr(myRecord,ccs-3,ccl)

I’ve used the offset and the length in this substring operation.

Here offset 0 in the section is at position 1:

/* Extract Plant Of Manufacture */
pom=strip(substr(section,75,4))

In the above the “pom” field is at offset 74 for 4 in the CPU Control Section. I use strip to remove any white space. In reality the Plant Of Manufacture is a character string like “51” (Poughkeepsie) or “84” (Singapore).

The Machine Serial Number is a bit more complex to extract:

csc=substr(substr(section,79,16),12,5)

In principle it’s a 16-byte character string, starting at offset 78 in the CPU Control Section. In practice it’s only the last 5 characters we want. Hence the second substr call.

Handling numeric fields is a bit trickier. When extracting the triplet fields you will’ve noticed the use of the c2d function. You use this “Character To Decimal” function to convert a string of bytes into a usable decimal number.

Handling Timestamps

Timestamps come in a wide variety of formats, but let’s just concentrate on the SMF Timestamp – in SMF Date And Time (SMFDT) format.

Extract the date portion of the SMF timestamp with:

dat=substr(c2x(substr(myRecord,7,4)),1,7)
year=substr(dat,1,4)-100
julian=substr(dat,5,3)
date2=date(,year""julian,"J" )

This is a little difficult to explain but I’ll give it a go:

1. Extract the 4 bytes at offset 10. Convert to hex with c2x and throw away the trailing nybble (always ‘F’ ).
2. Year is in nybbles 1-4 of that but we need to subtract the century.
3. Julian day of the year is in nybbles 5-7.
4. Construct a date in the format that the date function wants and call date, saying “This is a Julian Date”.

The result will be a string like “2 Jun 2016”. I like to uppercase the month with:

parse upper value date2 with day month year

So much for the date. Let’s now do the time.

tim=c2d(substr(myRecord,3,4))
mins=trunc(tim/6000)
hours=mins%60
mins=mins//60

1. Extract the 4 bytes at offset 6. They, converted to decimal, are hundredths of a second since midnight.
2. Minutes are got by dividing by 6000 and rounding down.
3. Hours are got from minutes by dividing by 60 and rounding down – with %.
4. Minutes are got from minutes using the remainder function (//)

I’ve thrown away the seconds and hundredths of seconds but they’re not difficult to capture.

Input / Output

I use EXECIO – which is built into TSO REXX. It can read a single line or the whole file.

In terms of output formats you could create a CSV file, just by the appropriate syntactical sugaring. Similarly, using careful reformatting you could create a flat file that contains the numeric fields in binary format, etc.

Record Selection

While I wouldn’t recommend this approach for filtering all the records your systems cuts, you can do very sophisticated filtering. But I’ll stick to record types and subtypes here.

Record type is a single byte at offset 5, so you want an if statement like

if c2d(substr(myRecord,2,1))=70 then do
  /* SMF 70 */
  ...
end

Record subtype is generally⁴ two bytes at offset 22:

if c2d(substr(myRecord,2,1))=70 & c2d(substr(myRecord,19,2))=1 then do
  /* SMF 70-1 */
  ...
end

Conclusion

You can readily process SMF with REXX, starting with z/OS 2.1. I wouldn’t be keen to do it with high volume records – but most records cut by RMF are low volume. Exceptions are mostly SMF 74-1 (Disk/Tape Activity) and 74-5/8 (Disk Controller & Cache).

But for prototyping it’s quite a good match.