(Originally posted 2011–04–21.)

Thanks to Scott Drummond for reminding me of last Autumn’s DFSORT Function PTFs – UK90025 and UK90026. They’re mentioned in the preview for z/OS Release 13 so now is not such a bad time to be talking about them. So let me pick out a few highlights:

---

Translation Between ASCII And EBCDIC, And To And From Hex and Binary

For a long time DFSORT has been able to translate to upper case (TRAN=LTOU), to lower case (TRAN=UTOL) and using a table (TRAN=ALTSEQ).

Now you can convert from ASCII to EBCDIC (TRAN=ATOE) and back (TRAN=ETOA). Translation is performed using TCP/IP’s hardcoded translation table.

Other “utility” translations are added: BIT, UNBIT, HEX and UNHEX. For example TRAN=HEX would translate X’C1F1′ to C’C1F1′ and TRAN=UNBIT would
translate C’1100000111110001′ to X’C1F1′.

---

Date Field Arithmetic

DFSORT already had some nice functions for handling dates and times. But here are some new things. This isn’t an exhaustive list:

• You can add years to a date field – with ADDYEARS.
• You can subtract months from a date field – with SUBMONS.
• You can calculate the difference between two dates – with DATEDIFF.
• You can calculate the next Tuesday for a date field – with NEXTDTUE.
• You can calculate the previous Wednesday for a date field – with PREVDWED.
• You can calculate the last day of the quarter – with LASTDAYQ.

---

JCL Symbols In Control Statements

You can now construct Symbols incorporating JCL PROC or SET symbols. These can be used in DFSORT and ICETOOL control statements, just like other symbols. You specify this by coding JPn“&MYSYM” in the PARM parameter of the EXEC statement. (In fact there need be no JCL symbol in this so you could pass in other strings this way and an expected use is for JPn to contain a mixture of JCL Symbols and fixed text.) n can be any one of 0 through to 9.

This support is in addition to the ability to use System Symbols (introduced with UK90013).

---

Microseconds In Timestamps

You can use the new DATE5 keyword to create a timestamp constant at run-time in the form ‘yyyy-mm-dd-h.mm.ss.nnnnnn’. DB2 folks might recognise this as the timestamp format for DB2 Unload and DSNTIAUL. You can use this for things like comparisons.

---

Chunking And Stitching Together Records

You can use the new ICETOOL RESIZE operator to:

• Split records into fixed-sized output records. For example, take a RECFM=FB, LRECL=500 file and create a RECFM=FB, LRECL=100 file – creating 5 new output records from each input record.
• Join together fixed-sized input records. For example, take a RECFM=FB, LRECL=100 file and create a RECFM=FB, LRECL=500 file – in effect reversing the above by joining 5 input records together to make an output record.

In each case you can see there could be problems with partial output records. DFSORT “does the right thing” using blanks.

---

Begin Group When Key Changes

I mentioned WHEN=GROUP here, in particular BEGIN=. With BEGIN= you get a new group when the condition you specify is satisfied. Now, with KEYBEGIN= you get a new group when the value in a particular field changes. For example:



SORT FIELDS=(1,12,CH,A,13,8,CH,D)
OUTREC IFTHEN=(WHEN=GROUP,KEYBEGIN=(1,12),
PUSH=(13:13,8,31:ID=3))






sticks a group number (3 characters wide) on the end of each record. The group number is incremented when there’s a new value in the 12-byte field that begins in position 1.

---

There are lots of other (in my opinion) smaller enhancements in this PTF.

If you want to know whether the appropriate PTF is on look for the following message in a DFSORT run:



ICE201I H RECORD TYPE …



If you see an “H” then you’re all set.

And you can read about these enhancements in more detail in User Guide for DFSORT PTFs UK90025 and UK90026 (SORTUGPH).

(Originally posted 2011–04–20.)

William Gibson’s “The future is already here — it’s just not very evenly distributed” applies very well to HTML5. It’s even more true of CSS3. Despite that (or maybe because of it) it’s a good time to dive into HTML5 – before everyone else does. 🙂

So, a few months ago I bought, read and inwardly digested 🙂 Mark Pilgrim’s HTML5 Up and Running, published by O’Reilly in August 2010. I have a rule of thumb: If a topic is covered by an O’Reilly book it’s probably ready for prime time. If it’s in “For Dummies” it’s probably too late. (With apologies to other similarly fine publishers and, of course, to the publishers of “For Dummies”.) It’s a glib rule but it’s mine. 🙂

So how does this rule of thumb work out for HTML5? Well, if you make a “poor choice” of browser then not very well. “Poor choice” is in quotes because:

• No one browser fully implements HTML5.

• It’s not just about which browser but about which level.

• You might be happy with one browser’s implementation but not another’s.

• There’s a degree of ambiguity, development of the state of the art, etc about HTML5 itself. I’d characterise it as a (slowly) moving target.

And note that this book is from last year. So things will have changed. But:

• It goes to great pains to describe the support by each browser.
• It makes the point in doing so that browser support is variable and you soon get the drift as to what each browser’s maker’s attitude is to HTML5.

Having been around a bit I know something about technology adoption: 25 years ago it would take at least 18 months from a product’s announcement to it being implemented in most installations. I really don’t think anything much has changed. So you wouldn’t write code that depended on a feature your customers (or users) don’t have yet. Well actually you would: You’d just expect it to take a while for them to catch up. And you certainly wouldn’t make it immediately mandatory.

And so it is with HTML5. Mark helps out considerably with the “support of unsupportive browsers” issue by recommending the Modernizr HTML and CSS Feature Detection package. (You can follow them on Twitter: here) I’m sure there are other techniques for handling this but Mark’s right in pointing out this one hunts for capability rather than a named browser. Given some of the features of HTML5 are awfully similar to those provided by javascript frameworks such as Dojo, I’d not be surprised if these frameworks could be relied on in the future to do feature detection (simulating it if not present in the browser). Good examples of this are input elements in forms, much enhanced in HTML5.

At this point I’m reminded I haven’t outlined what’s in HTML5. So here’s a high-level list:

• The Canvas drawing surface (which I have a “Production” application built around).

• Offline applications (which I’ve experimented with).

• Video (which looks a mess and set to remain one).

• Geolocation (which I’ve not used in the HTML5 context but have, of course, in Social Networking and other applications).

• Local Storage (a better replacement for cookies).

• Forms enhancements (which look really nice). Rather than my attempting to create graphics of them see here for some input type examples.

• Microdata (a replacement for the non-standard Microformats and the highly-incomprehensible RDFa – as a way of annotating parts of web pages with structured data).

So lots of really valuable things. But how does the book do?

Given the earliness of its publication I think it does very well. From the above I think you can see it pragmatically handles the issue of support – which is going to be key. It also describes each feature very well, with good clear examples. It also adds a historical backdrop – particularly when talking about how unknown elements are handled – so it gives you a good idea how we got here.

So, I think HTML5 is more than ready to be played with and this book is a very good one to get you started. (I’m assuming you’re not starting from a “zero knowledge of HTML” position.) It doesn’t tackle CSS3 and I’ve yet to find anything that does. When I find such a book I’ll probably buy it and review it here.

(Originally posted 2011–04–19.)

I wouldn’t want you to decide not to come to Vienna, just because I’ve made my slides available on Slideshare:

• I’d hope you’d come to Vienna anyway. It’s a great place and it’s going to be a great conference.
• Lots of people can’t make Vienna and I don’t suppose having the slides to hand is going to tilt the playing field significantly away from coming if you weren’t going to anyway.

I don’t think there’s any prohibition in uploading slides: It’s more important the messages get out. So here are the four presentations I’m giving. If you DO spot errors please let me know. Thanks!

• Memory Matters in 2011
• DB2 Data Sharing Performance for Beginners
• Parallel Sysplex Performance Topics
• Much Ado About CPU

And, I know many of you will have seen these before. If you’ve not seen them recently I hope they’re sufficiently evolved since you last saw them. I do have plans to do brand new presentations this year. More on that, in due course, in this blog.

(Originally posted 2011-04-15.)

Every once in a while I come across a particularly good idea: Where someone has done something particularly clever to solve a problem. Here’s one very recent example:

On the iPhone (and iPod touches) you can only display icons. They can be augmented with a numeric counter, but that’s all. (The idea is the app shows you the number of unread emails, for example.)

But here’s the idea, in a nutshell: If you tried hard enough you could display any number you wanted (actually any integer).

Those clever people at International Travel Weather Calculator have produced a pair of iPhone apps – Celsius and Fahrenheit. (I guess they needed to create two for the hard-of-converting.) 🙂

These two apps display the local (or remote) temperature permanently – without you having to open the app.

So they’ve subverted the counter – and that’s the clever part (weather apps being ten-a-penny).

(And this post is the first one I’ve created using Ecto. I mention it because it creates better HTML than the one built in to developerWorks and I hope it formats better. You can be the judge of that.)

(Originally posted 2011–04–12.)

First a word of thanks to Ferdy for his insightful comment on Batch Architecture, Part Zero. And also to my IBM colleague Torsten Michelmann for his offline note on the subject.

As I indicated in Part Zero I hoped to talk about jobs in a subsequent post. And this is that post. In particular I want to discuss

1. Viewing jobs as part of distinct applications, and
2. Generating a high-level understanding of individual jobs

Mostly I’m talking about using SMF 30 job-end records, but consider also:

• SMF 30 step-end records.
• SMF 16 DFSORT-invocation records (and, for balance, those for Syncsort).
• SMF 101 DB2 Accounting Trace.
• Scheduler Information.
• Locally-held information about jobs.

(When I talk about jobs I’m aware there are other activities, not running as z/OS-based jobs. These include other actions on z/OS, such as automated operator actions, recovery actions, and jobs running on other platforms. In this post I’m more focusing on z/OS-based batch jobs.)

---

Grouping Jobs Into Applications
There are lots of ways of grouping jobs into applications…

Most installations claim a job naming convention. For example:

• First character is “P” for “Production”, “D” for “Development” and “M” for “Miscellaneous”.
• Second through fourth characters denote the application. (Maybe there’s a business area as the second character and the other two are applications within that.)
• Last character denotes the frequency, e.g. “D” for “Daily”, “W” for “Weekly”, “M” for “Monthly”.
• The remaining characters (often numeric) are an identifier within the application.

Sometimes I see naming conventions that are the other way round. I would recommend – if you have the choice – having this way round. So status and application are at the front. The reason I recommend this is it makes it much easier to code queries against any instrumentation – whether you’re using SAS, Tivoli Decision Support or the home-grown code I use. (If you’re merging batch portfolios and have to pick a naming convention this is the one I’d definitely go for.)

Your workload scheduler may well have different names for operations (in Tivoli Workload Scheduler parlance) so some care is required with those.

An interesting question is how well an installation observes their naming convention. As the old joke goes “we love naming conventions: We’ve got lots of them”. 🙂 Analysis of SMF 30 should give you view of whether the naming convention is being observed.

As well as job names it’s sometimes interesting to see which userid(s) jobs are submitted under. Often Production batch is submitted from a single userid, according to Type 30. Similarly you can see which job class, WLM workload, service class and report class a job runs in.

Sometimes the programmer name field in Type 30 reveals application information.

Within a window it is occasionally the case that when a job runs is closely related to which application it’s in, though usually applications are intermingled in time – to some degree.

… And the above are just examples of characterisation information.

---

Understanding Individual Jobs – At A High Level

Whether you’ve grouped jobs into applications or are just looking at individual jobs it’s useful to characterise them. Typical characterisations include:

• Whether jobs are long-running or short. Likewise CPU- or I/O-intensive.

• Whether jobs are in-essence single step. (“In essence” alludes to the fact many jobs have small first and last steps, for management purposes.)

• Whether jobs have in-line backups (the presence of e.g. IDCAMS steps being a good indicator).

• How data sets are created and deleted for steps (e.g. IEFBR14 steps between processing steps).

• Whether jobs use tape or do non-database I/O (visible in tape and disk EXCP counts).

• Reliability statistics.

• Use of DB2. (Slightly tricky for IMS DB2 jobs but still can be done.)

• Clonedness.

• Sort product usage.

The above are all discernable from job- and step-level information. At a slightly lower level (because it requires the use of data set OPEN information) is characterising the data access as being to VSAM, or BDAM, or QSAM/BSAM (or some combination thereof).

A lot of the characterisation of jobs is centred around standards. For example, how jobs are set up by the installation features heavily in the above list. Other sorts of standards can only be seen in things like JCL.

While the above obviously applies to individual jobs it can equally be applied to applications (as identified above) but it’s obviously a bit more work.

---

This post has talked about how to use instrumentation to group jobs into applications and the like. It’s also included some thoughts on how to characterise individual jobs and applications.

I hope in the next part to talk about relationships between applications. And to dive deeper into the application’s data.

(Originally posted 2011–04–11.)

I’m working on my presentations for System z Technical University – Vienna, May 2–6 and I’m reviewing the agenda. As well as my four presentations there are lots of other goodies. These range from the Management level down to the purely technical. (I guess mine are towards the latter end of the scale – but I’d say there’s lots of pressure on us all to work on cost so detailed information on e.g. CPU has real impact.) In the other dimension there’s a very wide range of topics.

For the record I’m speaking on the following topics:

• Memory Matters in 2011
• Much Ado About CPU
• Parallel Sysplex Performance
• DB2 Data Sharing Performance For Beginners

These are all what I call “rolling” presentations: They evolve with time. If you haven’t seen them for a couple of years they’re substantially different. (Actually that’s probably true if it’s only been a year – as it will be for some of the luckier attendees.)

I’ll be a day late to the conference as I’m seeing Brian May and Kerry Ellis in concert at the Albert Hall the day before so won’t travel until the Monday. ( This concert is for a great cause: Leukemia and Lymphoma Research.)

I always enjoy these conferences: They’re generally in nice places but, more to the point, it’s great to run into old friends (customers, vendors and IBMers) and make new ones. And it’s always nice to hear things like “I saw you last year in Berlin and I’ll be in Vienna this year” (said by an Austrian customer back in February).

So, I think this conference is a great investment of time and money. And I feel very lucky to be attending yet again. See you there!

(Meanwhile I hope to be publishing my “Batch Architecture, Part One” post some time this week. I’m working on two batch situations that hopefully will inform the post, even if they delay it.)

(Originally posted 2011-04-04.)

Inspired by two of Bob Leah’s posts on QR Codes – here and here I started experimenting with creating and consuming QR codes.

But what is a QR  code? In short it’s a two-dimensional barcode that can contain e.g plain text or a URL. In the latter case a QR code reader can pick up the URL – maybe from a real-world object – and open it in a browser.

Creating QR Codes

In my experiment I created the barcode differently from how Bob did: As my laptop is running Ubuntu Linux I looked for a command-line tool. In my case I used the qrencode package. This takes a string and encodes it as a PNG graphic. Here is an example:

This is rather small – which might be handy from the perspective of printing labels.

Command line is important to me because it means I could automate generating QR codes – maybe a page of labels at a time.

Reading QR Codes

On my iPhone I installed a nice QR Code reader app: qrafter (in fact the free version). Although the QR code above is rather small it could read it perfectly well. I’m sure there are QR code readers for all kinds of mobile devices. Nowadays anything with a camera can do all sorts of things like barcode reading, QR code reading, document scanning (with or without OCR).

Possibilities

The ultimate aim of the experiment is to be able to tag objects: If you can tolerate sticking a small QR code label on an object you can annotate it: You could stick a URL on the object and then your device of choice could read the URL and open the page in a browser.

But what could the URL be? In my imagination it could be in two parts:

1. The URL points to a web server that maintains a database of information about objects. (In fact the URL points you to a page where you can view the information about the object – and optionally edit it.)
2. The search string is the object number. Each QR code has a different number. Actually it need not be a number, strictly speaking.

Of course you COULD do this with RFID tags. But this seems to me a lighter-weight way to get started. Of course there are many objects you wouldn’t or couldn’t stick paper labels on: Such as clothing. But there are lots of things you could annotate this way.

There are lots of possibilities here. I was just experimenting – admittedly in my hotel room on a Sunday night. I’d be interested in ideas and thoughts on this.

(Originally posted 2011-03-30.)

In 2007 I posted twice on memory metrics. The original posts are

• Memory Metrics Now That z/OS Release 8 Is Upon Us

and

• Mor(e )on UIC

I should probably have posted an update some time ago. In the latter I said "Obviously copious free frames would suggest no constraint." That’s true but I would invite installations to consider something else…

Capturing a dump into virtual memory backed by real memory is much faster than capturing it into paging space. (And that in turn is much faster than capturing it into constrained paging space.) Over the past couple of years I’ve progressively updated my "Memory Matters" presentation to cover Dumping and Paging Subsystem design – to reflect this.

So it’s important to consider what your stance on Dumping is. For some customers Availability will be the over-riding consideration and they’ll configure free memory to dump into. For others it’ll have to be a compromise – for machine lifecycle and economic reasons. The point is decide on a stance on provisioning memory for Dumping. And do it at the LPAR level.

Meanwhile, z/OS Development haven’t neglected this area. I’ve documented the z/OS Release 12 enhancements in "Memory Matters" but in short they are:

• Batch page-in I/O operations during SDUMP capture eliminates much of the I/O delay.
• Data captured will no longer look recently referenced. This data will be paged-out before other potentially more important data.
• Certain components now exploit a more efficient capture method in their SDUMP exits. For example GRS for SDATA GRSQ data, IOS for CTRACE data, and configuration dataspaces.

I’ve had foils on page data set design, Dumping control parameters etc for some time.

But the important thing is that dump speed is an important thing to factor in to memory configuration and monitoring.

And the thing that caused me to write this post – at last – is a discussion today on MXG-L on UIC. So thanks to the participants in that.

(Originally posted 2011-03-29.)

I’m not an architect. I don’t even play one on TV. 🙂 In fact real architects would probably say I’m in the babble phase, architecturewise.

But I’ve been involved in a few situations over the past year or so (and I’m involved in a couple starting round about now) which have led me to the following simple conclusion: Many installations would  benefit from drawing up a Batch Architecture. I don’t think this is specific to z/OS-based batch, though we do tend to have more complex batch environments than other platforms. (And modern environments seem to have z/OS-based and other batch mixed together, often in the same application.)

As I say, I’m not an architect so some of what follows will seem to real architects a lot like Officer Crabtree. 🙂 But it’s my thinking – and hopefully some of it resonates with you.

So what do I mean by a Batch Architecture? To me it contains the following elements:

• A description of the operating environment. This contains things like the LPARs Production Batch will run on, the database systems, message queuing systems and the like. You’d also rope in things like transmission networks, tape subsystems, job schedulers and printers supporting the batch. You might include commentary such as "we use PRDBATHI WLM class for critical production batch, PRDBATMD for most of it, and PRDBATLO for stuff that can run slowly but is still classified as Production".
• An inventory of applications. Though I’ll talk about this element more in a subsequent post I’ll note the minimum would be a list of names of applications and a description for each application. Also the job names (or rule for classifying jobs into a particular application).
  
• An understanding of the interfaces between applications. For example "PABC990D in ABC and PXYZ010D mark the boundary between the ABC application and the XYZ application, the former needing to update the General Ledger before the latter can begin to produce reports". Again, something I hope to write about some more.
  
• A view of when the window is – if there still is such a thing as a window. And what has to get done by when – with business justification such as "we have to post status with the Bank of England by this time in the morning".

The above, far from exhaustive, list enables you to think about your batch in a structured fashion. Done right, and it doesn’t really matter which tooling you use, it begins to enable you to:

• Talk about your batch at a level above the individual job.
• Think about the impact of growing business volumes.
• Plan for merging batch workloads (particularly topical at the moment).
• Plan for splitting off workloads.
• Think about how you can move work around.
• Consider what happens if there is a problem – whether an application failure or a disaster.
• Put some structure on modernisation efforts.
• Tune batch in a structured way.
• Collate the understanding of your batch that so often is in the heads of very good application experts.
  

Now, I appreciate most customers have huge batch inventories – often in the tens of thousands of jobs a night – and I think many customers are doing elements of this already. So what’s left to do that’s actually doable? I think quite a lot – and of course it varies from installation to installation.

But I do think some architectural thinking about batch would be really useful for most customers – and I’m certainly going to be thinking more about this myself (including seeking the wise counsel of some real architects).:-) At a practical level I’m going to post on how to do some of the building of a batch architecture.

(Originally posted 2011-03-25.)

I hope you do. Special thanks are due to Bob Leah, Victoria Ovens and David Salinas for getting me this far:

• Bob created the new template I’m using (and it is discussed further here.
• Victoria created the new blog header graphic (of a z196 and a zBX) and David put it up for me.

You’ll have noticed I’m also blogging again – after a gap of about a year. (I talk about that somewhere in the middle of here.) So, a fresh look seems like a very good thing.

developerWorks blogs are built using Lotus Connections Blogs (in turn built with Apache Roller/Velocity) – a flexible platform I’m beginning to learn how to tailor. So now my tweet stream is embedded – for one.

I’d like to add a blogroll and a set of other useful links in. Basic stuff, I know. It’ll take a little figuring out.

So here’s a question for you: Is there anything else you think I should do to make my blog more useful to you? Apart from (or maybe including) some useful content. 🙂