Two weeks ago I was on a Project Management course at work, which was highly enjoyable and very involved. Project management for a company this big is evidently a very complex and rigorous process involving all kinds of procedural steps and checks and formal presentations of stuff. That was one aspect of the course: the System, creating a proposal and then delivering it to dangerously executive executives.

The other half of the course was basically: "Agile is good, Agile is great, we surrender our souls as of this date".

There's a lot about modern software development that strikes me as, well, blindingly obvious. As in, "Why would you ever do things differently?" In a small project, deciding what tasks to focus on in the short term is a good thing. Limiting yourself to only a certain amount of development - one iteration - and always ending up with a slightly more useful, stable product at the end, is smart. Listing out all the various tasks ("user stories", "technical debt") that you want to do and having a well-structured discussion about their relative difficulty and the amount of time each will take? Five-minute daily meetings to find out where everybody was yesterday and is today? A clear distinction between the Lead Technical Bod and the Lead Non-Technical Bod? All of this makes perfect sense to me. For small projects.

Whether it works on a product as big as the ones that ICBM manufactures and sells is still as-yet unproven. We seem to still be in a transition phase at the moment, and an organisation this large doesn't turn on a sixpence, changing direction rather more like a petroleum tanker. This stuff has been developed Waterfall Style for decades, and the adjustment process from that to this isn't proving to be without its hitches. Something like CICS has run basically faultlessly for forty years. So do you want to develop something like that in any way other than "slowly and with extreme, extreme care"? We don't know Agile scales yet, but we're being asked to believe that it's perfect, it's the Way Forward. I'd love nothing more than that. I'd love it for everything to turn out fine, because quite frankly, I work here. But if something has to be believed in for it to work properly, it doesn't work properly. This is Britain, nation of sceptics.

*

One week ago I began the System z certification masterclass crash course, which the previous bunch of grads took nine months over and which we're doing in eight weeks flat. System z is a hardware platform, the latest iteration of which is the z10 mainframe, combined with the z operating system, z/OS.

ICBM System z is the Stonehenge of computing platforms.

It is monolithic: z10s are enormous square standalone black things with ridiculous power requirements, more closely resembling the monoliths of 2001: A Space Odyssey than any machine to date, both in form factor and processing power. It is ancient, tracing its roots all the way back to System/360 which was released in 1963 and was programmed using punch cards. It is terrifying. Unwieldy. Impassive. Inscrutable - mainframe programming is on a whole other planet from your conventional Linux and Windows and miscellaneous systems ("distributed platforms"), and I will shortly share a few of the major brain-freezes which I've encounted so far in my limited experience with the topic. It's phenomenally reliable: it's been doing what it's been doing since the dawn of time. And finally, you will never, ever be able to make it go away.

System z is a living fossil; it is computing archaeology; computing palaeontology. System z is an example of the principle "If it ain't broke, don't fix it" carried to its logical extreme. System z and its predecessors haven't broken for forty-five years: the "z" stands for "zero downtime" and some of these machines have been running continuously for decades without a reboot. Nor, therefore, have any of them ever been "fixed".

It uses EBCDIC, a character encoding wholly incompatible with ASCII featuring such delights as non-contiguous letter sequences. It was, from 1983 to 2000, a 31-bit operating system. (Yes, thirty-one.) It does things with virtualisation of system resources and operating systems themselves that were thirty years ahead of their time, but all the interaction with z machines is still performed through an archaic green screen system from the 1970s, running in an emulator, with a 24x80 character screen. The screen is 80 characters wide because that is how many characters you could fit on a punch card.

The output from the Job Entry System, however, is 132 characters wide, because that was how many characters you could fit on the line printer where the output came out. Nowadays, of course, the output comes to the green screen. So you always have to scroll left and right to see all of it. The keys for scrolling left and right are F7 and F8. For scrolling up and down? F10 and F11. No, Page Up and Page Down will do nothing.

To enter a command? You hit the right Ctrl key. "Return" will do nothing. "Enter" will work, of course, but, AS YOU KNOW, "Return" and "Enter" are separate keys on your keyboard. "Return" is just above the right Shift key. "Enter" is way over next to the numeric keypad.

To copy a line of text, you have to go to the small command area which exists to the left of every line and type a "c". Then hit right-Ctrl. Cutting and pasting is about as tiresome. To indent lines in bulk? Also tiresome, but don't worry about it - most code will break if you start doing crazy things like adding whitespace. Don't you love a text editor which requires the use of a command line to work properly?

"Undo" is disabled by default!

z/OS does not have files. Or directories.

Job Control Language, the language through which tasks are started and stopped (oh, everything is a batch job by the way; even the terminal session you're using to interface with this thing) isn't Turing-complete because it can't do loops. It is case-sensitive, by which I mean that everything must be in upper case.

Every valid line of JCL begins with a double slash, "//". Anything else is likely to be parsed as a comment. Because the screen is 80 characters wide but 8 characters are taken up by the aforementioned command area to the left, there are always 8 characters missing off to the right of the screen.

JCL does have subroutines ("procedures"), but get this: every line in a JCL proc has to have a label, and when you call a procedure, you can add extra lines below the procedure call to alter the contents of the procedure on a per-line basis. This was originally the only way to pass a "variable" into a procedure. Got it? That means that, as soon as you create a JCL procedure which is a really useful utility for other people, you can never change it again because someone, somewhere out there, is relying on it to continue to work exactly as it always has, in order for their code to work. If you so much as relabel one line of your procedure between one software version and the next, you will break somebody else's program.

And no, you cannot change things to make them better, because "somebody else" is Walmart, "somebody else" is Barclay's Bank, "somebody else" is the Tokyo Stock Exchange. I don't know that those three use System z specifically, but that is the scale of customer we are looking at here. z is this software developer's nightmare: it is something which can never be improved because so much depends on its extreme backwards compatibility and its pedigree of never failing and never changing, a pedigree apparently entirely at odds with the Agile manifesto.

Back to Blog
Back to Things Of Interest

Discussion (30)

2009-10-30 00:10:54 by rgoro:

Haha! This is what happens when someone with an inclination for writing fiction writes an essay: a plot twist in the end! I do it myself often, although I don't write that good fiction.


(by the way, I tried to post with my full name 'Román Gorojovsky Sánchez', and was told in red letters that 'names can only consist of letters'.....)

2009-10-30 04:28:45 by Jonn:

>a sixpence

What's that in American?

[/joke]

2009-10-30 08:44:02 by skztr:

Ah, z/OS. My boss told me "learn this, and you'll be set for life" (because no one else wants to learn it, and no one will listen to any alternatives). The "green screen" I have very little experience with, however. It's something which existed in another room, but wasn't always plugged in (or at least, rarely turned on). The green screen is something people like to show off during a job interview/tour, but don't actually use. All of my experience with z/OS was through telnet- with colour.
And once you're in telnet, all bets are off in terms of how flexible things can be.
I started by using a "standard" telnet client (you know, "telnet"), but soon I found that everyone used "intelligent" terminals which knew where they were, added menu options and keybindings, etc.

So the path from there was pretty obvious:
 - Rebind all the keys to something sane (for example, paging up and down with pgup/pgdown)
 - Write an FTP Proxy which translated z/OS's FTP into a format Windows could understand (then use vim to edit files)
 - Treat any format which was as painful to work with as JCL as a "compiled" format, and write scripts to build the JCL files. (though the JCL I was working with was very, very simple)

All of the issues with z/OS would have gone away by now, but the real problem with z/OS, much worse than any user interface issues, is the culture. Everyone who uses z/OS has spent a lot of effort knowing how to use it, and generally does nothing but it all day. No one is interested in coming up with "better solutions" because they either don't want to learn something new, or because they have a vested interest in making sure things aren't easier for other people.


At least, that takes care of the UI side of things. Now just try writing something that runs cross-platform on it. Of course, another major issue is getting any development done, and it's another important reason why things are "still" the way they are. There is no way to test code, because you either need access to a Real z/OS machine, or you need to pay obscene licensing fees for a (mostly non-working) virtual machine. They know their business model is to sell huge expensive machines to yet-huger companies, so they really have no interest in allowing anyone else to get anything done.

2009-10-30 09:39:33 by Overmind:

I actually used such a system way back when. I was actually comfortable with those old green screens, still have a computer here that uses one (ah, ten meg hard drive, was wonderful). And as much as I knew all the commands (yes the F keys for moving left/right and such as all), I would still never want to work with such a system. It would be so easy to build better interfaces over that exact same back-end, but yea, no one wants that for some reason...

2009-10-30 13:17:34 by dankuck:

I will probably have nightmares now.

2009-10-31 13:36:12 by Vertigo:

Sounds exciting.

My parents, back in the day, worked with System360. I've heard the stories, and we still have boxes upon boxes of punchcards in their basement. their friends, for their wedding, filled mom's old car with punch card chad. It *Still* spits chad out of the air conditioners when you fire it up.

2009-11-02 23:04:28 by Graham:

As an amateur programmer, this scares me. Badly. No whitespace? I might could manage. But that whole thing with the variables... -shivers-.

2009-11-05 14:07:29 by Gonsalu:

I work with one of these! :)

2009-11-05 14:34:47 by rgw:

you people have got to be kidding

2009-11-05 14:58:21 by Rob:

Right!

And all the other OS's make perfect sense.

- Windows which will have you chasing your tail if there is any sort of OS problem because there is very poor error messages to help debug what is happening.

- UNIX - not really sure that this is all that different. Except for the fact that it can't do some of the things that z/OS takes for granted. VI, VIM etc etc are just as arcane as ISPF possibly more so.

Development - Eclipse based tools are used for mainframe development. Just get your head out of the sand. There are a number of cross platform development tools. JAVA development works just fine on the mainframe as on unix or windows.

Files ? Take your pick Unix files are there, traditional "data sets" (mainframe way to store data) are also there.

TN3270, SSH using PUTTY, FTP, sFTP, Crypto services with actual FIPS-140 level 4 certification... I could go on and on..

While the article is worth a laugh.. only serves to breed FUD.

2009-11-05 15:10:41 by SteveConway:

I've made my living on mainframes since 1977. The article is interesting, getting the perspective of a person new to the platform. Google for zNextGen, and get in touch with a group of people who are also new to the platform. See what they've learned and how to make your time on z productive.

The IBM z/OS Basic Skills Information Center will also give you independent study resources at http://publib.boulder.ibm.com/infocenter/zos/basics/index.jsp.

Sure, z has warts due to backward incompatibility. A program that ran on OS/360 in 1970 will most likely run today on z/OS 1.11, unchanged, not even requiring a recompile. The business value of that is hard to compute.

EBCDIC translation is definitely an issue, no matter how many code pages you come up with, until you get to the Grail of Unicode. It is especially annoying doing cross-platform computing. Expect IBM to continue to work at ways to lessen the impact of this disparity.

As for screen geometry, 24x80 is the lowest common denominator, it's what we ran on dedicated terminals. It is NOT what I run with on a daily basis. As another poster pointed out, TN3270 (telnet 3270 emulation) lets you customize to your heart's content. I run 140 characters wide, and only scroll in the Unix environment on z.

Like any platform, z has its strengths and weaknesses; sometimes the two are so intertwined there's no separating them. It all comes down to the business needs. The trade-offs inherent in selecting one platform over another must be evaluated based on businedd needs and service levels required.

Welcome aboard.

Cheers,,,Steve

2009-11-05 15:14:55 by zOSdude:

Interesting take on z/OS, if not totally misinformed. IBM's commitment has always been to protect the client's investment in their computer systems. That's why an assembler program written in 1960 still runs on z/OS. How many Windows 3.1 programs run on Vista or Windows 7? Microsoft causes businesses to spend billions of dollars on new software every 2-3 years. Not to mention the training costs associated with retraining users, because as we all know, no PC software upgrade is complete until the UI has been totally rewritten. Mainframe programs don't do this. Any mainframe UI changes from release to release are typically evolutionary, not revolutionary. This preserves the client's investment, as they do not have to spend the money to retrain their entire staff to understand the new UI. The upgrades from Win 95, to 98, then XP, Vista, and now Windows 7 all incurred huge productivity costs as most users spent months getting accustomed to the new UI. How well did your Office 2003 to 2007 upgrade go? How long before the users were able to figure out how to do in the 2007 tabbed interface what they were doing in the 2003 menu interface?

Your 24X80 screen is only that way because you chose it to be. Most TN3270 emulators allow you to define your own screen size. Same for PageUp and PageDown. Most smart emulators make those functions page up and page down. As far as your statement that there's no development environment, that's just plain wrong, or should I say bollocks? Nearly all z/OS installations support robust test environments, where you can test all your programs.

It sounds like you either had a bad teacher, or work in a shop with a sub-standard z/OS environment. I could go on, but I've wasted enough time here. You're so off base it's not even funny.

2009-11-05 15:25:02 by SteveComstock:

Full of mis-information. Clearly you need some high quality training. (Ahem: www.trainersfriend.com)

"but all the interaction with z machines is still performed through an archaic green screen system from the 1970s, running in an emulator, with a 24x80 character screen. The screen is 80 characters wide because that is how many characters you could fit on a punch card."

"The output from the Job Entry System, however, is 132 characters wide, because that was how many characters you could fit on the line printer where the output came out. Nowadays, of course, the output comes to the green screen. So you always have to scroll left and right to see all of it. The keys for scrolling left and right are F7 and F8. For scrolling up and down? F10 and F11. No, Page Up and Page Down will do nothing."

Get a better terminal emulator. Mine is set at 62 lines / 133 characters and works great. I can
map the workstation keys anyway I want, so PgUp and PgDn can scroll up and down, if I would like
to do that. I always map the Enter function to the Enter key. But it's your choice.

"To copy a line of text, you have to go to the small command area which exists to the left of every line and type a "c". Then hit right-Ctrl. Cutting and pasting is about as tiresome. To indent lines in bulk? Also tiresome, but don't worry about it - most code will break if you start doing crazy things like adding whitespace. Don't you love a text editor which requires the use of a command line to work properly?"

To copy a line of text, I enter 'c' in the sequence number field of that line,
scroll to the destination (F7 or F8; PgUp or PgDn; or I can use a 'find' command)
and type an 'a' (for 'after') or a 'b' (for 'before') in the sequence number
field of the target line.

To indent a block of lines, enter '))' in the sequence number field of the first
line in the block, then '))n' in the sequence number field of the last line in
the block (where 'n' is the number of columns to shift; default is 2)

What are you talking about "most code will break if you start doing crazy things like adding whitespace"?
I've _never_ seen this happen. You really do need to get a more even handed
introduction to the platform.

"Don't you love a text editor which requires the use of a command line to work properly?"

Well, yes, yes I do. Especially as opposed an editor with no command line that
requires memorizing arcane key combinations.

"Don't you love a text editor which requires the use of a command line to work properly?

"Undo" is disabled by default!"

For historical reasons. Enter 'recovery on' and this enables undo, and it will
be remembered across sessions, so no big deal.

"z/OS does not have files. Or directories."

Of course it does. There are sequential files (flat files), VSAM files (which
can be sequential, indexed by key, or accessed by randomizing algorithm if you
like), and partitioned files ("libraries" which contain members and a directory).
You want Hierarchical files like UNIX? z/OS supports that too.

Get your facts straight.


"JCL does have subroutines ("procedures"), but get this: every line in a JCL proc has to have a label"

Simply not true.

"z is this software developer's nightmare: it is something which can never be improved because so much depends on its extreme backwards compatibility and its pedigree of never failing and never changing, a pedigree apparently entirely at odds with the Agile manifesto."

Let's see. In the past 10 or so years, z has gone to 64-bit architecture,
added support for ASCII and Unicode character sets, decimal floating point,
and hardware encryption. You can run all the classic z/OS applications _and_
you can run UNIX under z/OS! Yeah, you're right, it never changes.

I have no idea what "Agile" is, but it sounds clueless, depending on lies
and inaccuracies to portray its supposed superiority over a classic system
that is still living, growing, changing, and providing stable, secure,
scalable working applications for every day use.






2009-11-05 16:14:37 by DaveRivers:

Or - do your mainframe programming on your PC or Linux environment.

See http://www.dignus.com

2009-11-05 16:26:55 by Brakkie:

One word: Security!

z/OS has security (RACF) and full audit trails available via SMF records.
z/OS also gives you ability to issue or use digital certificate issued by yourself or by Verisign for example.
Your Credit Cards are protected by encryption schemes used by banks and invented by IBM et al.
z/OS has the power to use FULL audit trails and you can audit your users and datasets used.

z/OS also has very few virusse in the wild. Actually there are NO antivirus vendors for z/OS, but you can have antivirus software running on z/Unix and z/Linux, if you wish.

Speaking of z/Unix and z/Linux - you can choose to authorise access via RACF and/or via access list for each file and folder.

I could go on and on and on about security, but just Google about RACF.

2009-11-05 16:46:29 by zManDo:

A masterclass? Jeez, it's going to be a long course. All the author has learned so far is urban legend and miscellaneous other misinformation.

2009-11-05 17:42:44 by Scott:

Oh, z/OS. My blog on this crap is at: www.inherentlylame.com

I actually think we can get rid of it, because a good 80% of the shit that runs is some forgotten-about thing that is either not used at all, or only partially used as a source of data for something else.

The biggest problem with "getting rid" of the mainframe is that the data dependencies (and documentation) are so nebulous and grey that you need to do some serious analysis.

But if you compare all of the COBOL dreck, used within each department, you'll see that so much of it was just a copy-and-paste template with some hackery in the middle. I'm working on converting/destroying a lot of my departments COBOL and swapping it out with Java. I already promised to have 80% less code, 100% more documentation (from nothing), and fewer redundant jobs.

2009-11-05 18:14:44 by DanEspen:

Not sure what you were getting about every line in a proc having to have a label.

Yes, if you want to code an override, the override will address the proc.step.label, but otherwise no need to label.

I think you were trying to make the point that PROCS in JCL aren't generally useful to other programmers. That's the truth, but the reasons are more complicated. Mainly variable substitution in JCL and in PROCS is extremely weak.

2009-11-05 19:58:15 by Don:

Agile programming: http://www.globalnerdy.com/2007/11/28/dilbert-on-extreme-and-agile-programming/

2009-11-05 20:16:42 by Fjord:

It seems that the z/OS enthusiasts have arrived. I hope that you were wearing your flame-retardant underwear, Sam. It's a bit toasty in here.

2009-11-05 22:17:53 by jayatthecapacityorg:

"featuring such delights as non-contiguous letter sequences"

The letter sequences are actually contiguous, well were, on punch cards.

Good luck, all the things you said (good and bad) are true!

2009-11-06 02:43:57 by dustin:

Some discussion of this post at Hacker News:

http://news.ycombinator.com/item?id=923170

2009-11-07 13:12:49 by spartlow:

Nice article. I think you captured the initial shock of moving from distributed to mainframes well. Of course you had a lot of errors that others have commented on, but you captured the feeling so it made a good read. The real issue is mainframes are DIFFERENT from PCs so switching can be hard. Just like how some old-timer mainframers hate all the new GUIs that can be used on z, a GUI-lover will initially be aghast at all the 80 character lines. Don't despair.

2009-11-08 05:01:03 by ARealProgrammer:

The majority of this blog contains ZERO real information based on actual first hand experience. It's obvious that most of you have never even seen a real mainframe, much less ever used it. Would it surprise you to know that just about every major massively parallel gaming environment is hosted on a mainframe?

One of the reasons why the mainframe is so misunderstood is that access is limited to people who actually know what they are doing. Most large companies trust mainframes for their most critical processing and data storage needs and relegate the commodity processing, like serving up cutsey web pages, to computers you can buy in Wal-Mart, but the actual manipulation of that data is entrusted to real computers.

There's a direct relationship between a company's IT platform and it's ability to survive. It's no surprise that the vast majority of companies that are failing right now are not running their processing on mainframes and the ones making it through the economic crisis are.

As for COBOL and JCL, they are the simplest and most effective languages for the essential nature of business data processing which is financial, not glitzy games or fun web puzzles. By the way, COBOL has been capable of object oriented programming for over a decade. The reason it hasn't caught on is that OO is not the best language for programming financial applications. That's why there are over 5 billion lines of COBOL code in the business world and 5 million new lines of COBOL code are being created every day.

So keep playing games and writing "slopware" web pages for the HR department that don't provide much more than drivel. But, if you want to get into the nuts and bolts of how your company really runs, do anything you can to get on a mainframe programming project. Nearly all of the Fortune 500 CIOs got their start in mainframes.

By the way, most of the emails in my inbox are alerts about new viruses that infect the Windows and Unix platforms that you all claim to be antiquated and junk. Funny how there are never any about mainframe viruses.

Keep your perspective exactly where it is. The adults are laughing at you.

2009-11-10 01:20:17 by atomicthumbs:

"It uses EBCDIC"

:(

2009-11-12 18:14:00 by doomsought:

This will show up in your fiction some time right?
I mean this is just asking for it.

2009-11-16 05:43:24 by dentin:

Having spent five years at IBM here in the states, I understand fully where Sam's coming from. The culture and technology shock is in fact pretty impressive when you switch from Unix/Linux systems to AS/400 or System 36/390 stuff.

With that in mind, pretty much everything he said is valid: the default interface is crap. A lot of the tools are ancient crap. A lot of the reasons that you can't change anything are because the stuff you need to change is ancient crap with huge dependencies.

The things the enthusiasts also said are largely valid: better tools exist; you can change the defaults; and you don't have to use the crap interfaces.

But no number of workarounds will eliminate the fact that there are some impressive warts in old systems like this, and that said warts may in fact make the foray into Agile development a less than perfect match. Yes, it meets the customer's requirements. But it does so at extreme cost.

2009-11-16 05:49:48 by dentin:

ARealProgrammer:

Your comment paraphrased roughly as 'all major parallel MMOs run on mainframes' is misleading at best.

Large games generally have very large databases for the game world, and for very large games those databases generally reside on mainframes; but saying that the game 'runs' on the mainframe is equivalent to saying that my personal compute farm 'runs' on my NAS. The mainframe is simply a data repository. A slightly-less-than-dumb repository, but a repository nonetheless.

2012-10-02 18:46:45 by zOSprogrammer:

Ummm, no. Mainframes are a bit more than simple repositories. Your little PC is just a UI display device. Real programs run on real hardware, which is a zEnterprise EC10 today. See: http://www.informatik.uni-leipzig.de/cs/Literature/Features/report.pdf for a beginning of the differences (it's about two years out of date). The new hardware is based on a 4-core 5.5Ghz chip that allows up to 120 processors in a single system.

2012-10-02 21:21:34 by zOSProgrammer:

Here's another URL with information about the zEnterprise EC12:
http://mainframe-watch-belgium.blogspot.com/2012/08/the-new-ibm-zenterprise-ec12-technical.html
Yeah, it's geeky, but that should fit the crowd well. Mind you, the whole thing is about the size of a big commercial refrigerator. At Sear's Holding Company (division of K-Mart) their mainframe sits alone on one side of a huge room of racks and racks of servers (which they can't cool adequately), yet the mainframe is doing more real work. They have a project to replace most of the physical servers with a single z-box, running thousands of zLinux images under z/VM.

The Windows servers will be next.

The point is: Mainframes have a 30 year head start, and they are still advancing the technologies as fast (or faster) than distributed systems. But, then, the graphic flames really help your application's performance, right. (insert FlashPoint here.)