Prepare To Evacuate

It is rare to need to evacuate hardware. But being rare is not an excuse to neglect to plan for it.

Obviously I’ve been sensitised to the question by real customer situations – though I don’t intend to describe the situations in any detail; This post should prove useful enough without doing that.

When I say “rare” hardware generation upgrades usually require giving the machine over. So I suppose one could make a distinction between such things and repair actions and emergencies.

Perhaps more subtly, concurrent repair and upgrade actions are possible. With these you don’t hand the entire machine over. In fact it’s a valid question whether to or not.

So let’s discuss two options:

  • Evacuating a whole machine
  • Evacuating a processor drawer

(Obviously for a single drawer machine the two are the same. I tend to advocate customers buy machines with at least two drawers.)

Evacuating A Whole Machine

Evacuating a whole machine means taking down all of the LPARs of whatever description on the machine. Some of those LPARs might be running specific workloads, but others might be integral to the whole enterprise, for example coupling facility LPARs with unduplexed structures in or LPARs with DVIPAs in.

The former lead to homes being needed for the work they would’ve run, or for that work to be foregone. The latter need the vital components moved – whether explicitly or automatically.

Most customers I work with have more than one machine for a given workload; Obviously if you don’t the situation is more serious.

Evacuating A Processor Drawer

I won’t go into the mechanics of how to evacuate a drawer; In fact I don’t know how it’s done. (To state the obvious, I’m not a hardware planner nor a customer engineer. I try to nod wisely, though. šŸ˜Š)

It should be noted that a drawer might need to be pulled out for reasons other than repairs. The prime example I can think of is adding memory cards. Though not all increases in purchased memory require additional hardware, some do. The memory cards have to be accessed from the top of the drawer so it would need to be pulled out – and would need to be deactivated.

PR/SM allocates IFLs and ICFs from the top drawer down. It allocates GCPs and zIIPs from the bottom drawer upwards. (Nomenclature for drawers varies – even within IBM infrastructure – so it might be best to refer to them by their location in the frames, but that’s cumbersome.)

When a drawer is removed any characterised processors would have to be relocated (if possible). Likewise any memory (if possible). It should be expected that PR/SM will rework physical resources as well as LPAR allocations. (Lots of other things cause similar rework – but that’s beyond the scope of this post.)

A well planned configuration will have paths from each processor drawer to each I/O device – though not necessarily directly to each I/O drawer. While the standard planning tools strive to achieve redundant connectivity it is worth checking. (Again not my domain, if you’ll pardon the pun.)

While individual z17 DPUs (Data Processing Units) own specific channels, the loss of a DPU (actually most likely its PU chip) does lead to reworking. Perhaps one day I’ll write about SMF 73 Channel Path information and DPUs.

When I talk about connectivity and processor drawers similar things could be said about I/O drawers.

Capacity Planning

While evacuating a whole machine might lead to more loss of capacity than a single drawer, both scenarios need planning for. There needs to be enough remaining capacity to run the important workloads. I’m finding the conversation about shedding workload in an emergency is very similar to the conversation about making work displaceable. Many customers find it very difficult to find work to shed.

Of course, with the exception of older machines, it might be feasible to temporarily add capacity on surviving machines.

You would want to schedule such activities when there is little work on the affected machine – but such a time is becoming increasingly difficult to find. Particularly if you make the pessimistic assumption that it will take longer than scheduled.

Conclusion

Whether evacuating the whole machine or a single processor drawer planning for it is important. I’d advocate doing the planning at a time when the question is theoretical; You don’t want to do it in an emergency.

I would also advocate keeping an eye on where LPARs’ logical processor home addresses are in the machine. This is much easier with z16 and later machines.

One final thing: A long time ago, when Concurrent Drawer Repair was new, I presented it to a customer. Their response was “That’s very interesting, Mr Packer, but if it’s all the same to you, we’d like to continue with the practice of handing over the whole machine to you”. I wonder how many customers feel the same today.

Making Of

I wrote this post on two flights between London and Barcelona, my first time to that city. And with a new-to-me customer. My aim – for once, it seems – is to publish almost immediately.

What’s Important? CICS Transaction vs Region Goals

When assessing how well a WLM policy protects key work itā€™s necessary to understand how much CPU is used at each importance level. \ If there is very little work at a lower importance (higher importance number) a CPU crunch will leave work at this importance level vulnerable to CPU queuing.

Let me give a real, recent example:

Db2 ā€œEngineā€ (DBM1, MSTR, DIST address spaces) were in a service class at Importance 1 – with CPU Critical enabled. This is good. However, these address spaces were the only users of zIIP. (DBM1 mostly for Deferred Write and Prefetch, MSTR for Log Writes.) It is fortunate the zIIPs were lightly used. If they had been heavily used Db2 might well have slowed down. And that would cause Db2 clients e.g. CICS to slow down – even though these clients werenā€™t directly using zIIP.

In that example, Db2 being the only zIIP user is what I call ā€œa fact of lifeā€. You canā€™t tune that situation away. Fortunately, adding newer kinds of work that exploit zIIP almost certainly would create displaceable work. So growth in zIIP usage might cure the problem all by itself.

So the key thing we seek is indeed displaceable work.

And I deliberately gave a zIIP example to show itā€™s not just GCP CPU.

Measuring Displaceable Work

So, how do we measure displaceable work?

Letā€™s keep this simple by only looking at GCP CPU. (My standard code graphs CPU, zIIP-on-zIIP, and zIIP-on-GCP the same way.)

With RMF Workload Activity Report data (SMF 72-3) we can see CPU By Service Class Period. We can also see Importance for that workā€™s goal.

So, my code sums CPU by Importance – over all the service class periods. It plots it by time of day – as the picture in the batch window is often very different from the online peak.

But you mustnā€™t forget SYSTEM and SYSSTC service classes. And you mustnā€™t forget Discretionary. On the latter you could further divide between SYSOTHER and other discretionary service class periods. Personally I donā€™t.

A Twist With CICS

(What Iā€™m about to say is true of IMS as well.)

CICS allows you to define (single period) transaction service classes. If you do so the regionsā€™ own goals will be ignored and CICS work will be managed to the transaction service classā€™ goals.

Actually the dispatching priority of the region will be in support of the transaction goals. If there are multiple transaction service classes served by the same region the most important one will drive things. And the other transactions will come along for the ride. This is because it is the address space that has a dispatching priority, not an individual transaction.

So, if the transaction goal overrides the region goal, how do you tell what Importance the CICS CPU is at?

Unless you can tell the Importance of the predominant transactions in the region and the regionā€™s CPU itā€™s difficult. The latter can, of course, be obtained from SMF 30. But the correspondence between transactions and regions isnā€™t in RMF or in SMF 30. You could get it from SMF 110 CICS Monitor Trace – but that would be expensive. And it doesnā€™t speak to CICS transactionsā€™ service classes.

Fortunately, in most CICS customers, the service class of the transactions has the same WLM Importance as the regions. But not always. For example, a transaction might flow from a TOR (perhaps at Importance 1) to an AOR (perhaps at Importance 2).

And this, in a nutshell, is why I am careful about assessing displaceable work when CICS and IMS are involved.

Conclusion

Understanding the amount of CPU at the various Importance levels is important when understanding how resilient the WLM setup is. But itā€™s not always straightforward.

A few other notes:

  • Importance is not the same as dispatching priority – though it is the most scalable way of assessing displaceability.
  • Having little displaceable work should suggest you canā€™t run the machine all that busy. And you have to look at GCPs and zIIPs separately.
  • Iā€™ve not really talked about what Importance work should be classified to. You can have all the displaceable work in the world but if work is misclassified it could still avail you little.

One final point: I often ask customers questions like ā€œif you lost a machine what work could you sacrifice?ā€ To my mind this is similar to which work is displaceable. I have in mind another blog post on this subject.

Making Of

Again another ā€œwritten on a planeā€ post. For the first time ever, though, I had to restore the Markdown from a prior version. Somehow the text had mostly disappeared. Thankfully Drafts stores versions in iCloud. So, no harm done. Still, it freaked me out. Be careful to have automatic backups, folks.

Actually it was written on two plane journeys, some months apart. Iā€™d had it on my task list to complete but somehow never got round to it. Well, now I have.

Remember Memory

In Engineering – Part 8 – Remote Access Detection I talked about reasons why LPARs might go cross-drawer.

One reason I didnā€™t give was about memory.

(I should perhaps remind you that on modern machines a logical processor accessing Level 4 Cache or memory in another processor drawer incurs a significant performance penalty, showing up as an increase in Cycles Per Instruction (or CPI) in SMF 113. Engineering – Part 8 – Remote Access Detection talks about techniques to look at cross-drawer.)

I was reminded recently of something I already knew but hadnā€™t seen much incidence of: LPAR memory configuration leading to LPARs split across drawers.

An Example

Here is a theoretical example that illustrates the point:

A 2-drawer machine has 2TB of memory. (Actually this is a little modest these days.) There are three LPARs:

  • LPAR A with 800GB of memory and fewer than half the number of online logical cores that would fit in a drawer.
  • LPAR B with 400GB of memory and even fewer logical cores than LPAR A.
  • LPAR C with 700GB of memory and again fewer logical cores than half a drawerā€™s worth.

(When counting cores you need to add together all the cores in all the pools. So, for z/OS that would be both GCPs and zIIPs.)

Ignore the cores, except to note that any pair of LPARs taken together has fewer cores than would fit in a drawer.

With two drawers and three LPARs PR/SM has an interesting challenge: There are three possible pairs of LPARs to try to fit into a drawer. (More than one LPAR In a drawer is not a problem, in principle.) The three combinations are:

  • A + B: 1200GB
  • A + C: 1500GB
  • B + C: 1100GB

In fact all three LPARs add up to 1900GB, so there is enough memory on the machine to fit all three in the two drawers.

It is obvious that some LPAR will have to have memory allocated in both drawers.

PR/SM Decision Making

In the z17 Technical Guide I clarified the following thing: When allocating machine resources – memory and cores – PR/SM allocates them to the LPAR with the highest entitlement first, then the next highest, and so on. So far no clarification. The clarification is what the term entitlement means:

  • For dedicated core LPARs think of it as the number of cores (in all pools).
  • For shared core LPARs think of it as the number of coresā€™ worth of weight ( in all pools).

A general piece of advice is not to try too hard to second guess what PR/SM would do – but to aim for sensible LPAR design.

The example in this post does not represent the best possible design, though it might be necessary.

How Could This Situation Be Improved?

There are two main approaches, and each has its own downside:

  1. Buy more memory. That obviously costs money – so maybe not popular. But when buying a new machine this sort of thing should be taken into account.
  2. Reduce the memory footprint of the LPARs – but that has performance implications.

When I look at LPAR memory I generally see a lot spare – but not always. This can be established with SMF 71. But it is important to consider things like recovery scenarios: That apparently spare memory might be there to allow, for example, for a Db2 subsystem to be recovered, along with maybe some CICS regions. This is one reason I take an interest in Architecture and what uses installations put their hardware and software to.

Conclusion

LPAR Design is an interesting topic – but it canā€™t be treated as an exact science.

I would also note in closing two things:

  • You canā€™t from RMF tell how much memory is purchased in each drawer. Vital Product Data (VPD) is something a Customer Engineer has access to – and it does document this.
  • If an LPAR moves drawer it can take some time to move. The larger the LPARā€™s memory (including Virtual Flash) the longer the more it can take – with a potential for cross-drawer memory access.

Perhaps you shouldnā€™t just assign all purchased memory to an LPAR.

Making Of

Iā€™m writing this on a plane journey home, having spent the week with two rather sophisticated customers. Some of what Iā€™ve written is inspired by them, some not. Writing, to me, is an assimilation of what Iā€™ve experienced and thought about – often over many months. These two might just recognise themselves in bits of this but nobody else would. And the example is a simplified version of something real.

That last sentence was hard fought as I experimented with picking the iPad up and writing on it directly with the Apple Pencil. That didnā€™t work very well. In general, though, writing with the Pencil is getting a little easier.

And something is causing suggested text replacements to be better – and the acronym ā€œLPARā€ to be better recognised. Itā€™s probably AI as I think itā€™s hallucinating words into existence. šŸ˜€

Two Of These Are Not Like The Others

They were simpler times back then.

ā€œBack when?ā€ You might ask.

When PR/SM got started – in the mid 1980ā€™s – a machine might have two or three LPARs. Similarly, when Parallel Sysplex got started the number of members was very small.

For reference, a z17 ME1 can have up to 85 LPARs, and a Parallel Sysplex up to 32 members. I rarely see a customer approach 85 on a machine and Iā€™ve never seen a 32-way Parallel Sysplex.

However, I am seeing increasingly complex environments – both from the point of view of machines having more LPARs and Parallel Sysplexes having more members.

And, of course, Db2 was brand new – and now look at how complex many customersā€™ Db2 estates have become. A dozen or more members in a Datasharing Group is not unheard of, and multiple Datasharing Groups is very common. (And itā€™s not just Development vs Production.)

So How Do You Handle Such Complexity?

Thatā€™s the important question. Iā€™ll admit Iā€™m slowly learning – by doing. Iā€™d like to think I do each study better than the last – especially where the environments are complex.

What I donā€™t want to do is to say the same thing over and over again for each of the systems or Db2ā€™s. I think I would at the machine level – as thatā€™s relatively few times.

But I want any conversation to flow nicely – for all concerned.

There are a couple of approaches – and I try to do both:

  • Establish Commonality
  • Discern Differences

So letā€™s talk about them.

Establish Commonality

If you have 10 systems they might well have things in common. For example, they might have Db2 subsystems in the same Datasharing Group. Or cloned CICS regions.

There might be symmetry between LPARs on a pair of machines. This is very common – though asymmetry tends to creep in, particularly with older systems.

By finding commonality and symmetry itā€™s possible to tell the tale with economy of effort and reduced repetition.

Discern Differences

But symmetry might be broken and often Parallel Sysplexes are pulled together from disparate systems. This was particularly so in the early days – to take advantage of Parallel Sysplex License Charge, as much as anything.

Nowadays Iā€™m seeing a growth in differentiated systems within a Parallel Sysplex. Thankfully Iā€™m seeing pairs or quartets of such systems. Examples include:

  • DDF workloads, with their own Db2 subsystems alongside CICS or IMS systems. (Alongside meaning sharing data.)
  • Different CICS applications. Quite common is the ā€œBanking Channelā€ model.

So ā€œspot the differenceā€ is a good game to play.

The Importance Of Good Tools

Good tools enable me to see, among other things:

  • Architectural structure
  • Differences in behaviour

This post is not to boast about the quality of my tools – as most of them are in no fit state to sell or give away.

Further, I wouldnā€™t say my tools are perfect. Which is why I have to maintain a posture of continual improvement. Youā€™ll see an example of that later on.

Architecture

Over the years Iā€™ve taught my tools to produce architectural artefacts, such as which Db2 subsystems on which LPARs are members of which Datasharing Group. Further, a view of each Db2 is accessed by what. Likewise, which LPARs have which Service Classes and Report Classes being used.

Very recently Iā€™ve got interested in the proliferation of TCP/IP stacks – which I see as different address spaces, plus Coupling Facility structures.

Right now my nursery of ā€œinteresting address spacesā€ is growing.

Difference

You probably wondered about the title of this post.

Seeing differences in behaviours between supposedly similar things can be instructive.

Take this graph. Itā€™s brand new – and the only editing is removing system names and the title.

A few notes on how to read the graph:

  • Each series is for a different system.
  • Each data point is for a different RMF interval.
  • It shows how the velocity of a service class period varies with the GCP CPU used.
  • The green datum line is the goal velocity for the service class period. (If it varies itā€™s suppressed.)

You might have seen something like this before – but then each series wouldā€™ve been for a different day, not a different system. The question Iā€™m solving with this one is ā€œdo all the systems behave the same?ā€ rather than ā€œdoes this system behave the same way every day?ā€

(The idea of plotting a three-dimensional graph where the two horizontal axes are GCP CPU used and zIIP CPU used had occurred to me – but I consider it problematic both presentationally and technically. Maybe Iā€™ll experiment one day. And I did try out a 3D column chart in a recent engagement.)

But what does it show?

I see a number of things (and you might see others):

  • Two of these systems perform worse than the others. Hence the blog post title.
  • These two systems perform worse for the same sized workload.
  • These two systems have – much of the time – much more CPU consumption.
  • Even the better-performing systems struggle to meet goal.
  • You could argue all the systems scale quite nicely – as their velocity doesnā€™t drop much with increasing load.

With such a systematic difference you have to wonder why. A couple of thoughts occur:

  • System conditions might be different for these two systems. They are in fact larger LPARs – with lots of other things going on.
  • These two systems might be processing different work in the same service class. (Iā€™m not going to say ā€œperiodā€ anymore as these is a single period service class.) This is indeed a ā€œBanking Channelā€ customer.

Iā€™ve encouraged customers to judiciously reduce the number of service classes. The word ā€œjudiciouslyā€ is doing a lot of heavy lifting in that sentence. This might be a case where an additional service class is needed.

Still, vive la difference! It certainly shows the value of this graph.

One final point: The graph is for a velocity goal. Doing something similar for response time goals might be a bit more fiddly.

Here we have two subtypes: Average and Percentile variants. Compared to Velocity. So thatā€™s two more graphs to teach my code to construct. If I only want one itā€™d have to be Performance Index that is plotted – but thatā€™s too abstracted, I feel. Perhaps Iā€™ll experiment with this – probably in early 2026.

Conclusion

It is possible to tell the story of more complex environments in a relatively succinct way – and thus make discussions more consumable. But it takes some thought – and some code.

And my storytelling continues to evolve – which helps me want to keep doing this.

Making Of

This post started out as wanting to show off that graph. While I do like it a lot my thoughts went a lot wider in writing this. And I had the time for them to go wider as Iā€™m on a flight to Istanbul, to meet with a couple of my regular customers.

I was going to try my handwriting out again but somehow I lost the tip of the Apple Pencil on the plane before I got started. I did find it on landing – so all good now.

I still think some automation in my writing tool – Drafts – could help tidy up what I wrote. Iā€™ll have to think about that. Thatā€™s probably a good thing to play with on my flight home. Javascript at 35,000 feet.

Modern Machines, Modern Metrics

Modern Machines, Modern Metrics

In z17 Sustainability Metrics – Part 0 I wrote about the new z17 Sustainability Metrics, or “Power Consumption”, if you prefer.

This post isn’t part 1 – as I don’t intend to go into much detail about what I’ve learnt so far. I have learnt things, of course.

During the Summer it occurred to me that there are a number of things that are new in z/OS instrumentation on z16 and z17. Regular readers will know that I like to write about them.

I suggested to my friend John Baker of the IntelliMagic team that we could do a conference presentation about them. He readily accepted. (John has about the same level of interest in such things as I do – and sees different customers soour collective experience base is wider.)

And so Modern Machines, Modern Metrics was born.

As I alluded to “modern” is both z16 and z17. At least for now.

So what does “M4”1 cover?

It starts with a pair of z17 topics:

  • Sustainability Metrics
  • DPU ( Data Processing Unit) or “I/O Engine” as some of us like to call it

Then we talk about a couple of z16 topics.

As l write this, they are:

  • Home Addresses
  • AIU (Artificial Intelligence Unit)

I say “as I write this” because, as with all my presentations, this one will evolve – as we gain experience. We’re expecting the z17 topics to gradually crowd out the z16 ones.

I think it would be ambitious to hope to schedule this as a two parter, but you never know – and we’re still learning things about how z16 metrics behave. In fact older metrics can still spring surprises.

We gave this presentation a few weeks ago at GS UK Annual Conference. It went very well, I think. We’re about to give the first topic in a meeting with the Development team for Sustainability Metrics. I think they’ll find it interesting. We do have some questions to ask them, around data sources and interpretation – but I think you’d expect that.

And I’d like to think we’d evolve this for future machines.

Making Ofī…—

I’m writing this on a plane back from Madrid, mostly using an Apple Pencil Pro. I seemed to have it tamed. But then what is euphemistically called “light chop” šŸ˜Š intervened. I was doing well, really I was. šŸ˜Š The Bay Of Biscay is a harsh mistress.2

This is a new keyboard on the iPad Mini. More expensive than the previous one but it’s well worth the money.

But, again, some exotic (not really in my opinion) keys require strange finger acrobatics.

One cute thing is the keyboard can be made to light up – in different colours. And cycle between them. šŸ˜Š

  1. M4 is of course the name of a motorway, running west from London. But it’s a handy shorthand. ↩

  2. Hopefully some of you will get the reference. šŸ˜Š ↩

md2pptx 6 Is Another Big Step Forward

If the purpose of Version 5 was to add the ability to use Python in the context of md2pptx the purpose of version 6 is to extend md2pptx’s automation capabilities still further.

This time instead of Python it’s AppleScript. The context is different too:

  • With Version 5 the Python support runs the user-supplied code as part of the main md2pptx run – before the PowerPoint presentation is saved.
  • With Version 6 the AppleScript code is generated by md2pptx, saved, and then run – after the PowerPoint presentation has been saved.

But why do it this way?

The answer is that it enables some things that md2pptx can’t do – because the python-pptx package it builds on can’t enable them. And AppleScript – as a postprocessor – can.

It’s unfortunate that this is AppleScript only – restricting the function to Macs. That’s because I don’t have a Windows machine to test or develop Powershell scripts on. I might stand more of a chance with Linux – and something like Open Office. That’s because I have Raspberry Pi’s galore.

So, what can you do with the new support? As I write this you can – with v6.1:

  • Reopen the presentation in PowerPoint to a specific slide
  • Insert slides from other presentations
  • Run your own arbitrary AppleScript

Each of these has their own motivation:

  • When I rebuild a presentation I often want to reopen it in PowerPoint at a specific slide. So I created a bookmarking capability.
  • Users of python-pptx often want to embed slides from other presentations – so I built this more for them than me. But I actually do have standard educational slides in my workshop presentations – so I might well use it myself. A prime example of this is my WLM presentations: I start by explaining terms such as Performance Index and Velocity. I include some optional topics – which often become live – at the end.
  • As with Python for Version 5, I often want to experiment with AppleScript – so I made that easy to do. I also think some adventurous users will write their own.

As with Python, there is an obvious health warning: Running scripts you didn’t inspect can be dangerous. Any I ship will readable and well commented. If not that’s a defect and can be raised as an issue. As can any coding inefficiencies.

I actually ship an AppleScript file that contains a routine to copy slides from another presentation. I plant calls to it – if the user requests md2pptx to do so.

One other limitation of python-pptx is it can’t emulate the PowerPoint layout engine; By opening a presentation, navigating to a slide (or maybe through all slides), and then saving it, AppleScript could force PowerPoint to lay out the slide, text and all. I don’t know how useful it would be – but people have complained of such things. So I’ll have to experiment with this. And now I can.

The net of this is Version 6 opens up yet more automation possibilities for creating PowerPoint presentations.

One final thought: I prefer to add capabilities in Python rather than AppleScript. Further, I would prefer people not to have to use RunPython but rather use Markdown or metadata semantics. This is more user friendly and more widely applicable.

md2pptx 5 Is A Big Step Forward

A while back I experimented with executing user-provided Python. It seemed a small step at the time, but I had a hunch it would turn out to be a much bigger thing.

Coding it was straightforward but two things delayed releasing it:

  • Documentation is tricky.
  • The python-pptx API is (necessarily) not for the faint of heart.

But why bother in the first place?

While it is possible to inject any Python code you like that isn’t really the point. There are things I want in presentations that can’t be expressed with Markdown. Here are two examples:

  • Graphs
  • Table cell fine tuning

Actually the latter can be done with <span> elements and CSS – but it’s not as flexible as I’d like.

So, if I could expose what Python-pptx is capable of, I could make the run-Python support useful.

So that’s what I set out to do.

How To Invoke The New Function

This is actually pretty straightforward. Here is a simple sample of a slide with a bullet point and a trivial piece of Python.

### My Slide Title

* A bullet point

``` run-python
print("Hello world")
```

This code doesn’t do anything useful. But it makes the point you can embed arbitrary Python code. The real challenge was to teach my code to do something useful.

Making Inline Python Easier To Use

I’ve already mentioned it’s not that easy to drive python-pptx so I thought about how to make it easier. I wrote some helper functions, focused on things hard to express in Markdown which might actually be useful extensions.

I haven’t done any research on what people need; The helper functions just do things that patently made my programming life easier.

I’ve also ensured that useful things are exposed. Two examples of exposed objects are:

  • The current slide
  • The rendering rectangle you can use

I expect many use cases revolve around the slide md2pptx is currently creating. Further, it’s useful to tell the user code a safe area to render into.

My initial expected use cases are twofold:

  1. Adding content to a slide. Initially I worked on this for graphing. And this is where the rendering rectangle idea came from. Of course, you could render outside of this rectangle but you might collide with other shapes on the slide.
  2. Modifying content to a slide. A good example of this might be to filter cells in an existing table.

Both of these examples – graphing and table filtering – caused me to create a helper routine to read an external CSV file into an array:

  • You could run md2pptx every day against the latest version of a CSV file and create the same graphs from it, just with fresh data.
  • You could populate a fresh set of tables every day, perhaps turning some rows red – depending on the data.

So this was what I initially released.

Further Developments

Since the initial release I’ve done a number of things, most notably:

  • Added support for checklists and refined it somewhat. So you can – from a CSV file – create a checklist with (optionally coloured) ticks and crosses.
  • Selective bullet removal – as a sort of between-bullets paragraph function.
  • Tweaked graphing to make it more useful.
  • Added a helper routine for drawing arbitrary PowerPoint shapes.

These might be small things but they do illustrate one point: Version 5 is proving to be a great testbed for experimenting with python-pptx capabilities – and some of these did indeed get “shrink wrapped”.

Documentation

For me documentation isn’t that much fun to write. But it has to be done.

It’s the one thing that delays me releasing new levels of md2pptx most of all.

However, there is a brand new section towards the end of the User Guide – with the function descriptions and some examples.

Wrap Up – With An Important Caveat

My companion app mdpre might be useful here. A couple of examples illustrate why:

  • You could include code inline with =include.
  • You could include data inline with =include, too.

In fact I have been doing this for years – to pull in presentation fragments.

A word of warning: Because you can execute arbitrary Python code you need to be careful about where it came from.

Certainly – because it’s open source – you can inspect my helper routines – in runPython.py. And you might well create your own analogue.

Philosophically you might consider md2pptx is a long way from turning Markdown to slides. I’d say it’s still that. But, more generally, it’s turning textual data into slides.

It just got a lot more flexible and powerful in Version 5. And 5.2.2 is out, with more helper functions. I can’t say I’ll add more functions – or what they’ll be – but I probably will; This experiment got fun and surprisingly useful.

I’ll also say my checklist function is in use “in Production”: When I create the stub presentation files (Markdown, naturally) I now create a tracking presentation. It includes a CSV file that contains the checklist data. It’s easy to tick the database builds and presentations off as they get done. It’s a nice way of showing professionalism at the beginning of the workshop – or indeed leading up to it.

Making Of

I originally wrote this post when I had just released Version 5. I’m completing it on a flight to New York, to begin a visit to IBM Poughkeepsie. This actually allowed me to talk about the things that have happened to md2pptx since Version 5 debuted – which is quite a lot. And to show I really am taking the opportunity to experiment – now that I can.

11 Months On

That flight in January seems like a long time ago; It’s been a busy year – what with the z17 launch and a heavy caseload.

But md2pptx did roll on. Within Version 5 , for instance, the “checklist” function got enhanced with custom graphics – which permitted additional checklist item states (to look reasonable).

And Version 6 has been out for a while, with several tweaks within that version. I suppose I should write about it…

mdpre Comes Of Age

I wondered a while back why I hadn’t got mdpre to 1.0. It turns out there were still some things I felt it needed to have it “graduate”.

I suppose I should explain what mdpre actually is. It’s a tool for preprocessing text into Markdown, This text is “almost Markdown” or “Markdown+” so you can think of it as a tool people who normally write in Markdown could value.

I use it in almost all my writing – so it’s primarily designed to meet my needs. For example I often

  • Want to build materials from several files, linking to them from a master file.
  • Have CSV files that I want to turn into tables.
  • Want to use variables.
  • Want to use “if” statements for including files or pieces of text.

So I built mdpre to enable these things – and more.

But these things are common things for authors to want to do so l open sourced it.

I don’t think I’m ever going to consider it finished but at some point it’s worthwhile considering it to be ready. And that time is upon us.

It took one thing for me to declare 1.0: Allowing filenames to be specified on the command line. Now this isn’t something I wanted myself – but it was something people had asked for – and it felt like a completeness item.

So that was 1.0 a couple of weeks ago. But then I considered one of my “pain” points. Actually that’s rother overstating it: As l mentioned, l do quite a bit of creating tables from CSV.

So l thought about things that would enhance that experience, whether it was doing the same things quicker and better or enabling new functions.

Converting A CSV File To Markdown – =csv And =endcsv

If you have a Comma-Separated Value (CSV) file you want to render as a table in a Markdown document use the =csv statement.

Here is an example:

=csv
"A","1",2
"B","20",30
=endcsv

The table consists of two lines and will render as

A 1 2
B 20 30

The actual Markdown for the table produced is:

|A|1|2|
|:-|:-|:-|
|B|20|30|

You’ll notice an extra line crept in. By default, mdpre creates tables where the first CSV line is the title and all columns are of equal width and left-aligned.

If you have a file which is purely CSV you don’t actually need to code =csv and =endcsv in the input file just to convert it to a Markdown table – if you are happy with default column widths and alignments. Just use the -c command line parameter:

mdpre -c < input.csv > output.md

mdpre uses Python’s built-in csv module. Just coding =csv causes mdpre to treat the data as the “excel” dialect of CSV – with commas as separators. This might not suit your case. So you can specify a different dialect.

For example, to use a tab as the separator, code 

=csv excel-tab

“excel-tab” is another built in dialect. Your platform might support other dialects, such as “unix”. If you specify a dialect that is not available mdpre will list the available dialects.

Controlling Table Alignment With =colalign

You can control the alignment with e.g.

=colalign l r r

and the result would be

A 1 2
B 20 30

(This manual uses this very function.)

The actual Markdown for the table produced is:

|A|1|2|
|:-|-:|-:|
|B|20|30|

You can specify one of three alignments: l (for “left”), r (for “right”), or c (for “centre”). The default for a column is l.

If you have a large number of columns you might find it tedious or fiddly to specify them. mdpre has a shorthand that addresses this.

For example, coding

=colalign l rx4 lx2 c

Is the equivalent of 

=colalign l r r r r l l c

The first value is the alignment specifier, the second being the count of columns it applies to.

If there aren’t enough column specifiers for the rows in the table additional ones are implicitly added. By default these will contain the value “l”. You can override this by making the last one have “*” as the replication factor. For example rx* would make the unspecified columns right aligned, as well as the last specified one.

Controlling Table Column Widths With =colwidth

You can control the column widths with statements like

=colwidth 1 1 2

Adding that to the above produces the following Markdown

|A|1|2|
|:-|-:|--:|
|B|20|30|

Here the third column is specified as double the width of the others.

If you have a large number of columns you might find it tedious or fiddly to specify them. mdpre has a shorthand that addresses this.

For example, coding

=colwidth 1x4 2x3 1

Is the equivalent of 

=colwidth 1 1 1 1 2 2 2 1

The first value is the width specifier, the second being the count of columns it applies to.

If there aren’t enough column specifiers for the rows in the table additional ones are implicitly added. By default these will contain the value “1”. You can override this by making the last one have “*” as the replication factor. For example 3x* would make the unspecified columns have a width specifier of “3”, as well as the last specified one.

Note: Many Markdown processors ignore width directives. The developer’s other Markdown tool doesn’t. šŸ˜Š

Applying A CSS Class To Whole Rows With =rowspan

You can set the <span> element’s class attribute for the text in each cell in the immediately following row using =rowspan. For example

=rowspan blue

wraps each table cell’s text in the following row with <span class="blue"> and </span>.

Of course this class can apply any styling – through CSS – you like. But typically it would be used for colouring the text. Some basic examples of what you can do with CSS are in Some Useful CSS And Javascript Examples With =rowspan and =csvrule.

Note: This styling only applies to the immediately following row.

Applying A CSS Class To Cells Based On Rules With =csvrule

You can set the <span> element’s class attribute for each cell that meets some criteria. For example:

=csvrule red float(cellText) >= 99

wraps each table cell’s text that meets the criterion with <span class="red"> and </span>.

Each =csvrule statement is followed immediately by a single-word class name and an expression. The expression is passed to Python’s eval function. It should return a truthy value for the class to be applied.

Only code =csvrule outside of a =csv / =endcsv bracket. Each rule will apply to subsequent tables. You can code multiple rules for the same class name, each with their own expression.

Three variables you can use in the expression are:

  • cellText
  • columnNumber – which is 1-indexed
  • rowNumber – which is 1-indexed

Because mdpre imports the built-in re module you can use matching expressions for the text, such as:

=csvrule blue ((re.match(".+a", cellText)) and (columnNumber == 3))

The above example combines a regular expression match with a column number rule.

You can, of course, do strict string matches. For example:

=csvrule green cellText == "Alpha"

For numeric comparisons you need to coerce the cell text into the appropriate type. So the following wouldn’t work:

=csvrule red cellText >= 99

Speaking of mathematics, mdpre also imports the built-in math module.

Some basic examples of what you can do with CSS are in Some Useful CSS And Javascript Examples With =rowspan and =csvrule.

To delete all the rules, affecting future tables code

=csvrule delete

Some Useful CSS And Javascript Examples With =rowspan and =csvrule

=rowspan and =csvrule assign <span> classes.

mdpre passes CSS and other HTML elements through to the output file. A normal Markdown processor would pass these through into the HTML it might create. The full range of CSS (or indeed Javascript query) capabilities are available to the output of mdpre.

Here are some CSS and Javascript ideas, based off <span> classes.

Colouring The Cell’s Text With CSS

This CSS

.red {
    color: #FF0000;
}

colours the text in a cell with the “red” class to red.

Colouring The Cell’s Background With CSS

This CSS

td:has(.grey){
    background-color: #888888; 
}

colours the background of a cell with the “grey” class to grey.

Alerting Based On A Cell’s Class With Javascript

This Javascript

blueElements = document.getElementsByClassName("blue")
for(i = 0; i < blueElements.length ; i++){
    alert(blueElements[i].innerHTML)
}

pops up an alert dialog with the text of each cell whose class is “blue”.

Flowing A Table – To Shorten And Widen It – With =csvflow

You can widen and shorten a table by taking rows towards the end and appending them to prior rows. You might do this to make the table fit better on a (md2pptx-generated) slide.

The syntax is:

=csvflow <dataRows> <gutterColumns>

For example, if a table has 17 data rows (plus heading row) and the value of the <dataRows> parameter is 10, the last 7 data rows will be appended to the first 7. Three more blank sets of cells will “square up” the rectangle.

If a table has 24 data rows and the value of <dataRows> is 10, there will be three columns – with 10, 10, and 10 rows. The final 6 sets of cells in the third row will each contain blank cells.

All rows will be squared up – so the overall effect is to create a rectangular table – with no cells missing. You could use =csvflow to square up a table where the number of rows doesn’t exceed the <dataRows> value.

The <gutterColumns> parameter is optional and defaults to “0”. If you code “1” a single column “gutter” of cells with just a single space in will be added to the table prior to flowing. (The line-terminating gutter cells will be removed after flowing.) If you coded “2” then two gutter columns would be added – as appropriate.

Conclusion

So a lot of what I wrote above was new a few months ago. It’s made working with CSV-originated tables a bit easier and a lot more satisfying in terms of output.

But there is still more to do.

REXX And Python

A long time ago I wrote blog posts about driving Unix System Services from REXX. My motivation at the time was relatively minor. Now, however, I have a stronger motivation: Python.

You might or might not be aware that z/OS has modern Python support – and I expect it to keep up very well as Python evolves.

For reference, here are the three blog posts from long ago. Probably best to read them first – as it’ll make the code explanation a lot easier.

REXX vs Python

In a world where modern languages have become available REXX still has an important place: REXX supports external function packages. I use two of them – SLR and GDDM. I don’t expect to stop using either of them any time soon – as my toolchain relies on them.

Python, however, has a couple of advantages to me:

  1. It enables data structures to be expressed and manipulated more easily. For example, dictionaries.
  2. It has access to other capabilities – such as the CSV package.

I could add a third: People know Python. That’s not hugely relevant to me personally, though I’ve written a lot of Python over recent years and indeed have several open source projects written in it.

REXX Invoking Python

Here is a simple example.

It consists of two parts:

  1. A driving REXX exec.
  2. A driven Python program.

Here’s the REXX.

/* REXX */

python = "/shared/IBM/cyp/v3r12/pyz/bin/python3"

stdin.0 = 1
stdin.1 = python "python/test.py fred"

cmd = "/bin/sh"

call bpxwunix cmd,stdin.,stdout.,stderr.

interpret stdout.1
say "x=" x
say

Here’s the Python

import sys

print(f"x='{sys.argv[1]}'")

Problems Still To Solve

It would be better if my python code were inline – or at least in a member in my REXX library. That would make backing it up easier, for example.

I could go for (“heredoc”) inline code but that would involve storing the program inside a REXX string in the REXX code itself. That would get messy.

When I tried to pass code interactively to Python it complained it wasn’t UTF-8. It isn’t; It’s EBCDIC. A little taming required, I think.

The example uses interpret – which isn’t clever; That’s an easy problem to solve; You just use a sensible stem variable name for stdout.

It also occurs to me this could be structured so it looks more like Python driving REXX than it currently does today.

Anyhow, I’ll keep experimenting – once I find a real use case. I’m not planning on wholesale refactoring my current REXX code just to introduce some Pythonisms. Real world code has to earn its keep.

Meanwhile I’m continuing to write Python at a rate of knots – on my Mac.

Well, That’s Embarrassing

Iā€™ve had a task on my to-do list for some time now to assess the state of my blog and publish some posts I knew I had drafted.

Typically I write posts while in the air – as thereā€™s precious little else to do. So Iā€™d written several – some with more topicality than others.

I think a general state of being extremely busy with customer engagements and conferences has led to me not getting round to posting them. So, on this flight Iā€™m reviewing them, perhaps editing them a little, and preparing to post them at last.

I also have a few more thoughts to add – in new posts. Weā€™ll see.

So, for some of you itā€™ll be an opportunity to ā€œbinge readā€ – but I doubt I have many completionists in my audience. But most of my hits are from web searches, not least my own. šŸ˜Š1

By the way, the same paucity of time has limited Marnaā€™s and my ability to record. But we do have a nice episode in the advanced stages of planning.

But here we areā€¦

Making Of

Iā€™m writing this on a flight to Madrid. Iā€™m also testing out a new keyboard for this iPad Mini. Itā€™s better made than the last one – so Iā€™ll see how I get on with it. For instance, it took me a while to find the [ character, but the backtick was easy to find.

  1. Very often when I search the web the top hit is a blog post of mine. I walk a lonely road. šŸ˜Š ā†©