3. Unix

The next system we’ll study is the Unix operating system. First we’ll look at the circumstances surrounding the creation of Unix, and then we’ll examine some of its technical innovations. We will conclude this case study with a survey of the Unix legacy in technical terms and also in design philosophy.

The computing scene in the 1950s and 1960s

Many computers were custom-made, hardware and software, like the Apollo Guidance Computer. Commercial computing could be described as:

Very expensive
Requiring lots of space for installation (e.g. the entire floor of a building)
Single central processor (CPU is not even the right term, because the processor was typically a set of components)
No practical networking (though some manufacturers would eventually offer ways to transfer data only between computers of their brand)
Non-interactive, typically (punched cards, batch mode)
One user at a time (the operator, who ran the machine)

Timesharing

Timesharing is a technique that allows one computer to be shared by multiple users simultaneously. We take it for granted today that a single (core of a) CPU can run many programs at once, but mainframes running in batch mode did just one thing at a time.

When CPUs got fast enough, they could divide their time between the needs of multiple users, where each user had the experience of having the computer to herself. No more going to the basement with a deck of punch cards, and coming back later in the day for a printout.

Expensive computers that ran one job at a time became, with timesharing, expensive computers that could work for many people at once, interactively, all day long. User (programmer) productivity rose; utilization rates for expensive hardware went up.

With multiple users, there was now a need to allocate resources on a per-user basis. This meant that a user had to authenticate themselves, or “log in”. It is the operating system’s job to manage all the system resources, as you know.

Discussion question: What resources does the OS have to manage on a per-user basis, and why?

MULTICS

The MULTICS project was a large well-funded multi-institution effort to build a timesharing operating system that could be ported to many different computers. Recall that at this time, each manufacturer provided their own OS, compilers, text editors, and all other utilities needed to develop software and manage a system. MULTICS was an effort to boost productivity by eliminating all of that variation. Computer users would no longer be locked into using only the tools provided by the hardware vendor.

Discussion question: What happened to the MULTICS project, and how did it set the stage for Unix?

The legacy of Unix

Many features of the Unix design are found today, in various forms, in almost every operating system. Here are a few examples:

Everything is a file, even i/o devices and odd things like /dev/null and /dev/random.
The contents of a file is a sequence (technically a stream) of bytes. They have no other OS-enforced structure.
Each user has their own id and password, and are not expected to share them. The OS defines groups so that permissions can be granted to a collection of users, e.g. to access certain files or run certain programs.
Permissions are granted on a per-file basis (and everything is a file, even a directory!).
The C language was created to make Unix easier to port to new hardware.
The fork and exec system calls may have been original to Unix. We still have them today, in Linux and BSD Unixes (including MacOS). There are other ways to create new processes, e.g. Windows and other OSes do it differently.
Unix became a significant “open source” project before that term came into use. And it happened in an interesting way.

Unix, and hence everything influenced by Unix, was itself heavily influenced by the design of the DEC PDP-11 Mini Computer.

Required reading/viewing (part 1): Context

Mainframes, described here very briefly by IBM, the company that came to dominate the market, were for years the kind of computer purchased by large companies and government agencies.
The long Wikipedia article on mainframes has two really good sections: Design and Characteristics. Please read these sections and note the emphasis on reliability and availability. We discussed these terms in our AGC investigation and they reappear here as measures of quality of commercial products.
Timesharing was an improvement in human-computer interaction compared to the batch processing approach that was standard until the early 1960’s and common for many years afterwards. (Look for an NCSU connection in this article!)

Required reading/viewing (part 2): Unix

Read the listed sections of this retrospective on the design of the PDP-11: Introduction; the People section; and from the heading RISC to the end.
A 20-minute AT&T promotional video on Unix from 1982 shows us what AT&T (the phone company) thought Unix was.
- Optional: For a laugh, compare the Unix video with this one, on the Rockwell Retro Encabulator https://youtu.be/RXJKdh1KZ0w
One aspect of the Unix philosophy was emphasized in the Unix video (above) from AT&T: the ability to construct “pipelines” in which the output of one program is “piped into” another program, possibly as part of a longer chain of programs that together solve a problem. Much has been written about the Unix philosophy, and a short article worth reading is this one, from Wikipedia.

Optional reading/viewing

Early Unix history, is summarized by Dennis Ritchie in this blog-like essay. The content is technical and historical, and is light on design philosophy.
If you are interested in hearing other perspectives on the Unix philosophy and its connection to open source and software development generally, look for articles by Eric Raymond and Richard Gabriel (notably Worse is Better).
In the PDP-11 architecture, note these design points:
- Kernel (privileged) and user modes, which are needed for time sharing (interrupts and traps enter kernel mode)
- Byte addressable memory, though it could also address words, where most other machines of the time could only address full words
- Stack pointer support (push did a decrement and copied data; pop did an increment; therefore stacks had to grow downward)
- Instruction set support for ASCII, EBCDC, and several number representations was common, and complex to implement: Unix simplified by using only ASCII
- Early PDP-11 models could address memory only via 16 bits of address; later, the hardware extended this to 22 bits for a physical memory address, but only 16 bit pointers were supported by the instruction set! Each process, therefore, could only address 64KB. (The kernel could select which block of memory would be used by a given program.)
- The instruction set had auto incr, auto decr, and DEFERRED versions of each, corresponding directly to the C constructs ++j, —j, j++, j—.
- Before the PDP-11, mainframes (including PDP-10) and other early minicomputers had word sizes like 12 bits or 18 bits or even 36 bits. And a memory address pointed to an entire word. The 36-bit word size of the PDP-6/PDP-10 was influenced by the usefulness of having two Lisp 18-bit pointers in a single word. There’s a reference in this article for that rationale: https://en.wikipedia.org/wiki/Lisp_(programming_language)

Investigation questions, part 1

Mainframes.
a. What is a mainframe computer?
b. Roughly when were these developed, and did they exist before timesharing was a thing?
c. Give some notable specifications, such as memory size, speed, physical size, power consumption.
d. How much would a mainframe have cost, e.g. in 1970. What would that amount be today (use an inflation calculator)?
Mini-computers.
a. What was a 1970s-era mini-computer?
b. Give some notable specifications, such as memory size, speed, physical size, power consumption.
c. Were these designed primarily single-user “batch” processing, or multi-user interactive usage?
d. How much might a mini-computer have cost, e.g. in 1980. What would that amount be today (use an inflation calculator)?
Batch processing.
a. What does it mean for a computer to be used as a batch processing device?
b. Describe punch cards and paper tape.
c. Assuming punch cards were the input, and a printout was the output, what steps did a programmer go through to get their program to run? How did they produce the punch cards? Could they go to the computer and load them in at any time, and wait for the printout?
Multics.
a. What was this project? Who funded it, and what organizations participated?
b. Every computer vendor provided some kind of operating system with their hardware. Why was Multics needed?
c. What security feature(s) did Multics pioneer? Why were they needed?
Unix.
a. Where did Unix originate, and why?
b. How did the creators of Unix convince themselves and management that they needed to write an OS?
c. What did a telephone company need computers for in the late 1960s, anyway?
d. Why do you think a telephone company had a research division? What notable inventions came out of Bell Labs?
Early terminals. Imagine an early timesharing system with 5 or 10 terminals attached to it.
a. What is a terminal? Before we had video terminals, what did terminals look like?
b. How was a terminal connected to a computer? What kind of “protocol” went over the wire?
c. When a user pressed a key on the terminal, how did the computer know?
d. Some early terminals had paper tape punches and paper tape readers attached to them. How these would be used by a programmer?
Video display terminals.
a. Eventually, we had video display terminals. Roughly when did this occur? b. Name one or two popular models of video display terminals. What manufacturers made them?
c. Early use of video terminals worked the same as if the terminal was a teletype (typewriter keyboard with paper). Soon, the idea of a “full screen” application occurred to someone. The editor vi is an example. Contrast the editor ed with vi. How was editing different with one versus the other.
d. Describe the ANSI “escape sequences” used to control video displays.

Topics for discussion

Open source by mailing magnetic tapes of source code to those who requested it. We will discuss this in class, specifically how it relates to U.S. federal regulations on the business markets that AT&T could participate in. These regulations were imposed in return for giving AT&T a monopoly on long-distance phone service.
Everything is a file. Unix programs interact with devices as if they were files, in order to simplify the kernel API. A simpler kernel API should translate into a better experience for programmers. (This is true up to a point – see the ioctl system call for an example of where things get complicated.)
Every file is a stream of bytes. Most operating systems of the day supported structured files, i.e. a file was a series of records. A program had to be written with knowledge of the record structure in order to use a file. Furthermore, programs had to be written to interact with specific peripheral devices in a way that was very specific for each device type, make, and model.
In Unix, files and devices and directories and such were simply byte streams. Any program could read or write to any file by manipulating bytes. Sure, you had to know the format of a file if in fact the file was structured, such as if you wanted to open a directory and read the list of files in it. But this was your job as the writer of a “user space” program, instead of it being the job of the OS to provide a special set of APIs. Of course, the Unix source code does provide header files that user programs can use to make this task easier. And the set of Unix system calls did grow over time.
These design decisions were, in my view, largely about where the OS stops and the user programs begin. Unix designers took the view that the OS should ideally be simple and small. For example, a shell (bash, zsh, etc.) is just another program. You can write your own shell. It is not part of the OS in the sense that it is not “special” or “magic” and thus has to be provided by the OS vendor.
The C “machine model” is a PDP-11. This topic is covered well the reading on the legacy of the PDP-11. In addition, there is the C ABI.

Investigation questions, part 2

Text files for configuration. One way to configure an OS is to have a database of configuration settings, and for the OS to supply a special program to read and edit that database.
a. Briefly describe the Windows Registry, for which there are many precedents in the history of operating systems.
b. A different approach is to use a collection of text files, such as: (1) most of the files in /etc on a Unix/Linux system, (2) the various rc files found in user directories, like ~/.bashrc and ~/.vimrc. Choose an example of one system config file and one user config file that we can look at together as examples. What do you think are some pros and cons of plain text versus a binary database for settings?
c. Why not use binary files? Here’s a study in contrasts: JSON was defined as a text format, but there are efforts to define a binary JSON format. Pick one such effort and provide a link to its description. What are the apparent pros and cons of switching from the existing JSON text format to binary?
d. Google defined a “protocol buffer” data representation. Is it text or binary? Describe it briefly.
The C ABI.
a. What is an ABI? Contrast ABI and API.
b. Give some examples of the “rules” that define the C ABI.
c. What is a foreign function interface?
d. Many programming languages have a “foreign function interface” (FFI). The FFI for Python, Rust, Lua, Java, C++, many Scheme implementations, and others are all interfaces to the C ABI. It’s hard to find, e.g. a FFI library for Java that is designed to interact directly with Scheme/Lisp code. Speculate as to why. What does an FFI do, e.g. Python’s cffi (or consider another example if you wish)? It must do something related to data and something related to functions.
Unix programming.
a. How does a new process get created (started) in Unix?
b. Describe the fork system call.
c. Describe the exec system call.
d. Contrast with the Windows approach (shared by many other earlier systems, and some current ones), which is to use a spawn API.
The Posix standard.
a. What is Posix?
b. What kinds of things does it specify?
c. What does it mean to be “Posix compliant”?
d. Are OSes like MacOS X and Linux Posix compliant? (For Linux, pick a popular Linux distro, e.g. Ubuntu, Arch, Mint, RedHat, or whatever is familiar to you.)
Application-level network protocols. Our next topic is networked application design, i.e. systems that use multiple computers which communicate with each other. Low-level network protocols like tcp and udp are “binary” in the sense that they are defined by byte sequences, often with individual bits having special meanings. Application protocols are often text-based.
a. Research HTTP 1.1 and explain the basic format of an HTTP request and response. Give examples of “HTTP headers”.
b. Research SMTP, and give an example of how an email message may be sent (transferred from one computer to another) via SMTP. In other words, what does an email message look like “on the wire”?

“Narrow waist” architectures

In our discussion of the investigation questions above, we encountered the concept of “narrow waist” architectures. If we look at the OSI network model as a conceptual model, and then look at the internet as it functions in reality, we could highlight TCP as the “narrow waist of the internet”: Many application protocols work “over TCP”. Such protocols (mostly) don’t care how TCP is implemented, as long as it provides the TCP abstraction of a bidirectional stream.

The layers below TCP could be ip and ethernet, or SNA and Token Ring, or any number of other technologies. Above TCP are many protocols that rely on it, like HTTP, SSH, SMTP, FTP, etc. The resulting architecture, when one takes a broad view, is shaped like an hourglass, with TCP in the middle.

In practice, internet traffic is based on ip, though the layers below that vary, in order to efficiently move data over copper wires for short distances, copper wire for long distances, fiber optics, terrestrial microwave, and satellite.

HTTP has arguably displaced TCP as this narrow waist. We now run other protocols on top of HTTP. REST is an entire category of such protocols, but there are more.

The C ABI is another integration point, or narrow waist. Foreign Function Interfaces (FFI) allow one language to interface with code written (and already compiled) in another language. Somehow, everyone decided to use the C ABI as the integration standard. Language A generates data that is represented in memory the way C would store it, and they call a library function the way C would call a function. Language B exposes a function that expects to be called using C’s calling conventions, and it expects data in memory to be represented in C’s style. And neither A nor B might be C!

Posix, Wifi, Bluetooth, and other technical standards: These are often both standards and trademarked names. If you build a device that has Bluetooth and you want to advertise that it does, your device has to pass a bunch of compliance tests. When it does, you get certified and can use the Bluetooth name on your product. Certification also gives you confidence that your product will work with all kinds of devices made by other companies that also use Bluetooth.

Last, but not least, there is text! A program that can read/write text can interact with any other program that does the same – no binary encodings are needed. Of course, both sides need to know to expect from the other, but at they don’t need to know what binary encoding it will be in. Plain text has another advantage, which is that is human readable (and writable). But its role as a “narrow waist” is all about how many programs generate text and how many other programs take text as input. Those all speak in plain text.

Software Systems Design (FIRST DRAFT)