Free at last!

The punched-card was restrictive in many ways: there were only codes for letters, digits and a small collection of special characters. No provision was made for both upper and lower case letters and a very few codes were designated for control characters. Each of the computer manufacturers intro­duced their own version of the cards to accommodate their code needs. The American Standards Association’s (ASA) X3.2 subcommittee published the first version of the ASCII 8-bit code-set in 1963. The ASCII code-set was an 8-bit character encoding that eliminated most of the shortcomings of earlier code-sets. The ASA had created an earlier 7-bit code-set which included four separator codes. When IBM launched its System 360 it also introduced the EBCDIC encoding for punched-cards, which included the separator control codes introduced in the ASCII 7-bit code-set. In 1968 President Johnson mandated that all government computers would use ASCII, so IBM dropped their many-flavored EBCDIC and embraced the ASCII 8-bit code-set for System 370.

ASCII has since evolved to UTF-8, UTF-16 and UTF-32. Bigger and bigger code-sets that embrace all of mankind’s various alphabets and symbols. Code-sets went from 5-bits to 32-bits in 50 years. A 32-bit code-set allows for 4,294,967,296 different characters! (I don’t know anyone who uses UTF-32. Most programming is done using UTF-8.) The only reason that I even discuss this issue is because ASCII introduced the four very important control characters; the hierarchical separators. Quoting Wikipedia:

Data structuring. The separators (File, Group, Record, and Unit: FS, GS, RS and US: 1C, 1D, 1E and 1F, hexadecimal) were made to structure data, usually on a tape, in order to simulate punched cards. End of medium (EM) warns that the tape (or other recording medium) is end­ing. While many systems use CR/LF and TAB for structuring data, it is possible to encounter the separator control characters in data that needs to be structured. The separator control characters are not overloaded; there is no general use of them except to separate data into structured groupings. Their numeric values are contiguous with the space character (20 hexadecimal) which can be considered a member of the group, as a word separator.

I decided to use these separators to free IT records from their straight jackets. I devised a scheme for using these control characters to allow variable length fields and unrestrained nesting of dependent data. The paper that I wrote about my scheme may be found here (it was presented at the ACM SIC­FIDET Workshop in Houston Texas in 1970). Unfortunately, I have found it impossible to persuade anyone to employ this technique. I am aware of only two implementations of the use of these concepts. Both were in systems that I developed. (Yes – I’ve been worrying this problem for a long time.)