(Originally posted 2018-05-29.)
In a sense this post follows on from I Must Be Mad – where I talked about some of the subtleties of processing SMF. In another sense it’s writing down in public a briefing I want to give one of my mentees.1
She’s about to prototype some code to go against a record subtype we’ve not handled before. In fact we throw any records of this subtype away – today.
… all the way back to a z/OS 2.1 presentation my friend and co-conspirator Marna Walle gave in 2013. (I had to look that one up.) 🙂
In it she mentioned z/OS TSO REXX being able to process Variable Blocked Spanned (VBS) data. The best use case for this is, of course, SMF.
I was delighted to see this support. But I’ve done nothing with it until now.
Actually I have some REXX code that processes SMF but I don’t really like it: It copies SMF 70-1 to a VB (not Spanned) data set and then a REXX exec processes this data. There’s a very good reason for not liking this: It risks truncating records. SMF 70-1 records can be very long, so this is a potentially serious problem.
So, processing VBS would avoid the VBS-to-VB copy and eliminate the risk of breakage.
The rest of this post is about some of the coding techniques for processing SMF with REXX. I actually developed them while processing SMF in VB format, but they are equally valid with VBS.
Processing SMF With REXX
While REXX is reasonably fast, I wouldn’t use it for high volume record types. My use case was extracting the LPARs on a machine which are deactivated. You get this from SMF 70-1 where the Logical Partition Data Section says there are no Logical Processor Data Sections.
This is a low volume case – as 70-1 records are only generated on an RMF interval and there are relatively few of them. My code processes 70-1 in a second or two.
Record Offsets Versus REXX Variable Substrings
This is probably the area that has the greatest potential to cause confusion:
- Records begin at Offset 0, including the 4-byte Record Descriptor Word3 (RDW). So the first byte of data after the RDW is at Offset 4.
- In a REXX string the first byte is at Position 1. When REXX reads the record with EXECIO the RDW is discarded. The position is used when extracting substrings – with
So, to convert offsets to positions you subtract 3.
There are a couple of approaches:
- Keep the “subtract 3” thing in your head. (Or use a routine.)
- Prepend 3 bytes and use position as if it were offset.
In my code I chose the former – without the benefit of a conversion routine.
Extracting The SMFID
Because the SMFID is at offset 14, and bearing in mind the “subtract 3” point, you can extract it from a record in a variable with:
It’s a 4-byte character string – so it needs no further processing.
A section is a portion of the record with a fixed layout – and hence a fixed length. There might be more than one of a given layout.
Sections within a record are pointed to by data structures called triplets. As the name suggests, they consist of three fields:
- 4-byte offset – how far into the record the first section of this type starts
- 2-byte length – how long every section of this type is
- 2-byte count – how many sections of this type there are
To get to the sections of a given type you use the offset and then process them linearly, using the length to extract them, and to skip to the next.
In the SMF record header is a vector of triplets. So, for example, in SMF 70 Subtype 1 the vector starts at offset 28. The first few triplets in the vector are:
- 28 ( X‘1C’ ) – RMF Product Section
- 36 ( X‘24’ ) – CPU Control Section
- 44 ( X‘2C’ ) – CPU Data Section
When I process a section I extract all three portions of the triplet separately:
/* Extract CPU Control Section position, length and count */ ccs=c2d(substr(myRecord,33,4)) ccl=c2d(substr(myRecord,37,2)) ccn=c2d(substr(myRecord,39,2))
Reminder: The control section starts at position 33 in the variable – which is offset 36.
Processing A Section
I extract the first (and only) CPU Control Section itself:
I’ve used the offset and the length in this substring operation.
Here offset 0 in the section is at position 1:
/* Extract Plant Of Manufacture */ pom=strip(substr(section,75,4))
In the above the “pom” field is at offset 74 for 4 in the CPU Control Section. I use
strip to remove any white space. In reality the Plant Of Manufacture is a character string like “51” (Poughkeepsie) or “84” (Singapore).
The Machine Serial Number is a bit more complex to extract:
In principle it’s a 16-byte character string, starting at offset 78 in the CPU Control Section. In practice it’s only the last 5 characters we want. Hence the second
Handling numeric fields is a bit trickier. When extracting the triplet fields you will’ve noticed the use of the
c2d function. You use this “Character To Decimal” function to convert a string of bytes into a usable decimal number.
Timestamps come in a wide variety of formats, but let’s just concentrate on the SMF Timestamp – in SMF Date And Time (SMFDT) format.
Extract the date portion of the SMF timestamp with:
dat=substr(c2x(substr(myRecord,7,4)),1,7) year=substr(dat,1,4)-100 julian=substr(dat,5,3) date2=date(,year""julian,"J" )
This is a little difficult to explain but I’ll give it a go:
- Extract the 4 bytes at offset 10. Convert to hex with
c2xand throw away the trailing nybble (always ‘F’ ).
- Year is in nybbles 1-4 of that but we need to subtract the century.
- Julian day of the year is in nybbles 5-7.
- Construct a date in the format that the
datefunction wants and call
date, saying “This is a Julian Date”.
The result will be a string like “2 Jun 2016”. I like to uppercase the month with:
parse upper value date2 with day month year
So much for the date. Let’s now do the time.
tim=c2d(substr(myRecord,3,4)) mins=trunc(tim/6000) hours=mins%60 mins=mins//60
- Extract the 4 bytes at offset 6. They, converted to decimal, are hundredths of a second since midnight.
- Minutes are got by dividing by 6000 and rounding down.
- Hours are got from minutes by dividing by 60 and rounding down – with
- Minutes are got from minutes using the remainder function (
I’ve thrown away the seconds and hundredths of seconds but they’re not difficult to capture.
Input / Output
EXECIO – which is built into TSO REXX. It can read a single line or the whole file.
In terms of output formats you could create a CSV file, just by the appropriate syntactical sugaring. Similarly, using careful reformatting you could create a flat file that contains the numeric fields in binary format, etc.
While I wouldn’t recommend this approach for filtering all the records your systems cuts, you can do very sophisticated filtering. But I’ll stick to record types and subtypes here.
Record type is a single byte at offset 5, so you want an
if statement like
if c2d(substr(myRecord,2,1))=70 then do /* SMF 70 */ ... end
Record subtype is generally4 two bytes at offset 22:
if c2d(substr(myRecord,2,1))=70 & c2d(substr(myRecord,19,2))=1 then do /* SMF 70-1 */ ... end
You can readily process SMF with REXX, starting with z/OS 2.1. I wouldn’t be keen to do it with high volume records – but most records cut by RMF are low volume. Exceptions are mostly SMF 74-1 (Disk/Tape Activity) and 74-5/8 (Disk Controller & Cache).
But for prototyping it’s quite a good match.
Actually, hopefully more than one will find it useful – in time. And by “mentees” I think I mean “friends I’d like to share The Joy Of SMF with”. 🙂 Well, something like that. :-) ↩
Yes, the juxtaposition of “Rewind” and “Scroll Forwards” is awkward; Glad you noticed. 🙂 Or at least are reading footnotes. :-) ↩
The first two bytes of the four-byte RDW are the length of the record. ↩
Subtype location is actually a convention, which a few record types break. ↩