filterCSV, An Open Source Preprocessor For Mind Mapping Software

I have a number of ideas for things I want to open source, some directly related to the day job and some not. This post is about one piece of software that I use in my day job but which you probably wouldn’t recognise as relevant to mainframe performance.

To me the rule of thumb for candidates for open sourcing is clear: Something of use outside of IBM but with little-to-no prospect of being commercialised.

filterCSV is just such a piece of software.

What’s The Original Point Of filterCSV?

filterCSV started out as a very simple idea: Our processing often leads to CSV (Comma Separated Value) files with a tree structure encoded in them.

This tree structure enables me to create tree diagrams in iThoughts. iThoughts is mind mapping software I’m (ab)using to draw tree diagrams. Whereas most people create mind maps by hand, I’m bulk loading them from a CSV file. Strictly speaking, I’m not creating a mind map – but I am creating a tree.

iThoughts has a variant of CSV for importing mind maps / trees. It’s documented here. It’s a very simple format that could be confected by any competent programmer, or from a spreadsheet.

So, to filterCSV: I’ve got in the habit of colouring the nodes in the tree I create in iThoughts. Originally I did it by hand but that doesn’t scale well as a method. If I discern a bunch of nodes (perhaps CICS regions) are part of a group I want to colour them all at once.

The very first piece of filterCSV, which is a Python 3 script, compared nodes to a regular expression. If they matched they’d be coloured with a specified RGB value – by altering the CSV file. I would import this altered CSV file into iThoughts.

In a real customer engagement this saves a lot of time: For CICS regions the nodes have the string “RC: CICSAORS” in, for example. “RC” is short for “Report Class”, of course. So the following works quite well as a command line invocation:

filterCSV < input.csv > output.csv ‘RC:CICSAORS’ FFBDAA

So every node with “RC: CICSAORS” in its text gets coloured with RGB value FFBDAA.

If I keep going with this I can find rules that colour all the CICS regions. Then I understand them much better.

Open Sourcing filterCSV

Let’s generalise the idea: You might be creating a mind map and want to colour some nodes, based on a readily-codifiable criterion. Here’s what you do:

1. You export the mind map from iThoughts in CSV format.
2. You throw the CSV file through filterCSV, specifying the regular expression and the colour on the command line.
3. You import the resulting CSV file into iThoughts.

I don’t know how many users of mind mapping software want to do this, but I bet I’m not alone in wanting it. If the effort to open source it were minimal it makes sense to do it, rather than accepting I’m going to be the only user.

So, I put it on IBM’s internal GitHub site – and I was pleased when Christian Clauss of IBM Zurich joined me in the effort. He’s brought a lot of experience and, in particular, testing knowledge to bear.

Then I got permission to open source filterCSV. This turned out to be very straightforward for a number of reasons:

• IBM is keen on open sourcing stuff.
• There is no prospect of this becoming product code.
• The process for open sourcing when there are no dependencies is streamlined.

I’ll also say this is a good practice run for open sourcing things that are of wider interest – and particularly for the mainframe community. Which is something I really want to do.

So it’s now a project on GitHub. I subsequently went through the process with another one – which I’ll talk about in another blog post.

filterCSV Has Morphed Somewhat

I realised a couple of things while developing filterCSV:

1. It’s not just iThoughts that the method could be applied to.
2. What I really have is a tree manipulation tool. In fact that’s essentially what mind mapping software is.

It’s the combination of those two points that made me think the tool could be more generally useful. So here are some things I’ve added to make it so:

• It can import flat text – creating a tree using indentation. That can include Markdown using asterisks for bullets.
• It can import XML.
• You can delete nodes that match a regular expression.
• You can change the shape of a matching node, or its colour.
• You can write HTML in tabular or nested list form.
• You can write XML – either OPML or Freemind.
• You can promote nodes up the hierarchy, replacing their parents.
• You can spread Level 0 nodes vertically or horizontally. This helps when you have multiple trees.

Craig Scott, the developer of iThoughts, kindly gave me the RGB values for iThoughts’ colour palette. So now you can specify a colour number in the palette. (You can actually go “next colour” (or “nc” for short), which is quite a boon when you have multiple regular expression rules.)

Some of these things came to me while using filterCSV in real life; the experience of actually using something you built is useful.

Conclusion

So this has been a fun project, where I’ve learnt a lot of Python. I continue to have things I want to do to filterCSV, including something that should be out in the very next few days. The general “tree manipulation” and “adjunct to iThoughts” ideas seem to have merit. And I’m enjoying using my own tooling.

If you fancied contributing to this open source project I’m open to that. In any case you can find it here on GitHub. The latest release is 1.2 and 1.3 should, as I say, be out soon.

And I have plenty of ideas for things to enhance filterCSV.