edwardaux / pipelines

Java implementation of CMS Pipelines

NO LONGER SUPPORTED

Due to a lot of different reasons, this library is no longer supported. If you're interested in this topic, a more robust and complete implementation based on Swift is available.

Java Pipelines

Years ago, I used to use a product on the IBM mainframe called CMS Pipelines. It is an awesome record-based filtering system that lets you filter and change records as they flow through the pipeline. Having left IBM some years ago, I no longer had access to a mainframe system where I could use this tool.

There is currently a NetREXX-based partial implementation of CMS Pipelines called njPipes, but given that I lack 1) a NetREXX installation, and 2) the inclination to find and install one, I thought that it would be an interesting exercise to try to write one in pure Java.

I had a false start a few years ago when I realised that I didn't fully understand the whole record delay mechanism, but about 6 months ago, I had a situation arise where an implementation of Pipes would have been just the ticket. Obviously that situation has long since passed, but it sowed the seed that I needed, and I re-downloaded all the documentation that I needed.

Current features

While the implementation that I have is not complete, it definitely has the makings of a full featured implementation. I have tried to stick as closely as possible to the original implementation (notwithstanding the obvious Java/Rexx and EBCDIC/ASCII differences). At times this can be hard because I don't have a mainframe implementation to check my work with, but through careful reading of the manuals, I believe that I have created a very faithful reproduction.

There are several features that are worth pointing out, I think:

• The dispatcher is complete with one exception: stall detection. I have designed the stall detection mechanism, but have not yet implemented it.
• Single- and Multi-stream pipelines are fully supported.
• Callpipe has been implemented (but probably needs a little bit of testing). Addpipe has been designed but not yet implemented.
• The dispatcher has the ability to generate events, which can be "listened to". Eventually, this mechanism will be used to implement RUNPIPE EVENTS.
• Full support for MSGLEVEL, TRACE, STAGESEP, LISTCMD, LISTERR, et al (at both the pipe and stage level)
• While I have done very little performance tuning, I have been very conscious of throughput during the whole design and development process.

Implemented Stages

The list below gives a taste for which stages have been implemented:

• <
• ABBREV
• ADDRDW
• AGGRC
• BETWEEN
• BUFFER
• CHANGE
• CHOP
• COMMAND
• CONSOLE
• COUNT
• DUPLICATE
• FANIN
• FANINANY
• FANOUT
• GATE
• HOLE
• HOSTID
• HOSTNAME
• LITERAL
• LOCATE
• NLOCATE
• NOEOFBACK
• NOT
• QUERY
• REVERSE
• SPECS
• SPLIT
• STRLITERAL
• TAKE

How to use?

Download pipe.jar from github.com, and use a command like this one:

java -jar pipe.jar "literal hello there | cons"

Alternatively, if you want to use it from within your existing java program, you could do something like:

new Pipe().run("literal hello there | > blah");

Bug Reports or Problems

Although I have many testcases for each of the stages, there is obviously some chance that bugs have slipped through. If you believe that you have found a bug, or a difference to the mainframe implementation, please email me.

pipelines's Issues

SPECS stage always uses WORDSEP SPACE for all Input-Conv-Output groups when not specified explicitly

Hi,
My (old) z/VM manual CMS Pipelines Reference Version 3 Release 1.0, SPECS stage says:

"If WORDSEPARATOR (or WS) is specified more than once in a SPECS stage, a word separator character remains in effect until the next WORDSEPARATOR(or WS) specification is encountered."

My manual shows this example for WS in SPECS:
pipe literal a-b-c | specs wordsep - w1 1 w2 5 w3 9 | console
a b c
Ready;

For me, on Linux, that pipe produces this output:
$ pipe "literal a-b-c | specs wordsep - w1 1 w2 5 w3 9 | console"
a

$

For each Input-Conversion-Output group, it appears "new RangeSingle()" is called and it uses a hard-coded default of _wordSep = ' ' if no WS is specified. It doesn't use the WS currently in effect.

It's easy to get around the problem (just use explicit WS on every grouping), but thought I should document it.
Thanks,
Keith