1 Introduction
2 Ground Rules

Building a File System
3 File Systems
4 File Content Data Structure
5 Allocation Cluster Manager
6 Exceptions and Emancipation
7 Base Classes, Testing, and More
8 File Meta Data
9 Native File Class
10 Our File System
11 Allocation Table
12 File System Support Code
13 Initializing the File System
14 Contiguous Files
15 Rebuilding the File System
16 Native File System Support Methods
17 Lookups, Wildcards, and Unicode, Oh My
18 Finishing the File System Class

The Init Program
19 Hardware Abstraction and UOS Architecture
20 Init Command Mode
21 Using Our File System
22 Hardware and Device Lists
23 Fun with Stores: Partitions
24 Fun with Stores: RAID
25 Fun with Stores: RAM Disks
26 Init wrap-up

The Executive
27 Overview of The Executive
28 Starting the Kernel
29 The Kernel
30 Making a Store Bootable
31 The MMC
32 The HMC
33 Loading the components
34 Using the File Processor
35 Symbols and the SSC
36 The File Processor and Device Management
37 The File Processor and File System Management
38 Finishing Executive Startup

Users and Security
39 Introduction to Users and Security
40 More Fun With Stores: File Heaps
41 File Heaps, part 2
42 SysUAF
43 TUser

Terminal I/O
45 Shells and UCL
46 UOS API, the Application Side
47 UOS API, the Executive Side
48 I/O Devices
49 Streams
50 Terminal Output Filters
51 The TTerminal Class
52 Handles
53 Putting it All Together
54 Getting Terminal Input
55 QIO
56 Cooking Terminal Input
57 Putting it all together, part 2
58 Quotas and I/O

59 UCL Basics
60 Symbol Substitution
61 UCL Command execution
62 UCL Command execution, part 2
63 UCL Command Abbreviation


Download sources
Download binaries

I/O Devices

Before we discuss the specifics of outputting text to the user's terminal, we need to have a more in-depth discussion of I/O devices in general. In order for UOS to be flexible, both now and in the future, it is important to reduce the number of assumptions we make. Of course, it is impossible to make no assumptions. On one hand, you could define a device as being anything (CPU, cables, memory, concepts, etc). It is hard to imagine how that would be useful. On the other extreme, you define a device down to the level of a specific make and model disk drive. Such a device would be outdated within (at most) a couple of years. It is hard to imagine how that would be useful in a general-purpose operating system, or even possible to keep up with - given the number of new disks that constantly come on the market. So, how do we design an architecture that is both flexible and useful? It is both an art and a science. UOS will use a device architecture that is based on a wholistic knowledge of the current and past technology, and an awareness of trends that point to the future of device technology. In the most basic sense, a UOS device is something which we can either send output to, get input from, or both. But we will exclude system memory and co-processors. They could fit into the generic definition of a device, but they are so tightly integrated to the CPU that we will handle them as separate things.

UOS has a three-tier device architecture. The first tier is the integrated hardware interface. An example of the hardware interface would be the printer port on the motherboard. We call this "tighly-coupled" hardware, since it is part of the base system hardware. The second tier is the attached hardware. An example of the attached hardware is the printer that is connected to the printer port (via a cable). We call this "loosely-coupled" hardware. The third tier is the media. And example of media is the paper that goes into the printer.

The three-tier architecture applies to any hardware. For instance, a floppy disk has the tightly-couple hardware interface, the loosely-coupled floppy drive, and the media is the floppy disk. Obviously, some devices have no media. For instance, a video terminal has no paper or other media (although it might have an attached printer, but this is handled by the terminal without the knowledge of UOS). Even those devices with media may not report that the media is not present (such as paper in some models of printer). In some other cases, all there is of the device is the tightly-coupled hardware, such as the system clock (which is handled entirely by the HAL) and the non-volatile BIOS ROM. But no matter what the device's actual physical characteristics, all devices have these three tiers defined, even if they are not all used.

Using the stores as an example, since we've spent a lot of time discussing them, they are the media portion of the three tiers. That is, the store is located on the media (usually the disk itself, or a DVD disc), which is inside a loosely-coupled drive, which is connected to a SATA interface. Each tier imposes its own restrictions on the overall capabilities of the device. For instance, the first tier hardware could, by design, only allow output (such as with a printer port) - making it write-only. Even if a hard-copy terminal with a keyboard was connected to the printer port, the port has no means of receiving input from the connected device. The second tier hardware could be set as read only (a switch on some USB thumb drives, for instance), even though the first tier hardware allows both reading and writing. The media itself may prevent writing (such as a non-writable DVDROM), even if the first and second tier allow both reading and writing. The read-only and write-only capabilities are, perhaps, the most common characteristic used in regard to devices, but in this article we will see additional characteristics which apply to all three tiers.

Note that file systems and store partitions do not fit into the three-tier architecture - they are simply organizational schemes used on the media. Device drivers don't deal with file systems.

The software that directly interacts with the first tier hardware (and sometimes with the second tier hardware) is the device driver. We will describe drivers in far more detail in future articles. The FiP's TDevice class (not the one used in the HAL and Init) interface with the drivers.

The TDevice_Info structure provided by the HAL was discussed back in the articles about Init. We will now expand this structure to contain additional information about the devices. The TDevice class in the FiP contains a device info structure that is a combined set of characteristics for all three tiers. You could consider it a logical and operation of all three tiers (ie the least common denominator). We will now explore the extentions to this structure.

Device Class
There are several general classes of device. The one we've talked the most about the most are stores. As described before, a store is any device on which data can be written and later retrieved from, under program control. The second class of device is the stream. Streams can be input, output, or both. They differ from stores in that the data output to them is not retrivable under program control - the data from a stream is usually not what was written to it. An example of a stream is a terminal, such as the console. A magnetic tape, however, is a store. It may be accessed sequentially, but the program can reposition to the previously written data and read it back. But, technology being what it is, sometimes the line between stores and streams is somewhat blurred. For instance, data written to a video terminal can sometimes (depending upon the terminal) be retrieved. The PC monitor is often the output device of the console, which means the data is stored in video ram and a program could possible retrieve that data. Although, unless the operation is performed via the device instance, it is still a stream. So, the general characteristic of a stream is that it is a one-way transfer of data - whatever is output to a stream is essentially "lost" to the computer - at least, without human interaction to somehow get the data back via some other means. An example of that would be when output is sent to a printer, and then the user uses a scanner and optical character reader software to get the data back into the system.

The third class of device is "unknown". That is, it is neither a store nor a stream. Anything that doesn't really qualify as a store or stream fits in this category. An example would be an array processor. These read data from memory and perform parallel math operations. The only I/O interaction with such devices has to do with loading up the "registers" to indicate the operation to perform, and the memory locations of the operands and output.

Since the exact characteristics of a device depend upon all three tiers, generic interfaces (such as USB) can completely change depending upon what is connected to the interface. In essence, when not connected, such first tier devices are invisible to the user since they will have no device designation associated with them. If a store (such as a thumb drive) is connected, a new store device will be "created". If a stream is connected, a new stream device will appear.

One last issue of device class should be mentioned, although we won't discuss it in detail until a future article. Some devices - most commonly networks - can serve as a "bridge" to another device which has a completely different class. An example of such "bridged" devices is a network disk. In this case, the network is a stream, but it provides access to a store. Another example are network printers, which demonstrate an I/O stream providing access to an different output stream.

We will change the TDevice_Info structure and constants as follows: Basically, the constants that used to be DT_* (device type) are now DFC_* (device flags: class), and what used to be the Device_Type item in the structure is now Flags. These changes allow us to include several other characteristics without expanding the size of the structure. Thus, the previous

value can be obtained via
Info.Flags and DF_Class_Mask

We've also collapsed some discrete booleans, such as Media_Present, into bits in the Status.

Access method
There are two means of accessing devices: sequential and random-access. The stores we've looked at so far are disks, which are accessed randomly. Stores such as magnetic tapes are accessed sequentially. By definition, streams are accessed sequentially, but it is conceivable that some of the output or input could be buffered and accessed randomly on some stream devices, although I cannot think of any physical device in existance that works this way. It should also be noted that CDROMs and DVDROMs are accessed randomly, but the media itself is technically sequential. CDs and DVDs have a single track that spirals from the outside edge to the inside edge. However the drives can skip across to any point on this spiral track to seek for a specific location. This, in conjunction with the device driver, make the media work like a random-access disk drive when reading from it. Writing to it is a different matter - data is written sequentially, although that only applies to writable media. We will discuss CDs and DVDs in a future article. As far as tapes are concerned, the tape can be moved past the read head in either direction, allowing access to any given record on the tape. However, each individual record between the current position and the target position must be read (or skipped), although there is a rewind mechanism which allows the tape to be rewound at high speed to the beginning. Tapes are another device/media that we will discuss in a future article. Here are the access method flag values:

// Device Class...
const DFC_Non_Existant = 0 ; // Invalid device
      DFC_Unknown = 1 ; // Type not otherwise included here
      DFC_Stream = 2 ; // Stream device
      DFC_Store = 3 ; // Data store device
      DF_Class_Mask = 3 ;

Access mode
Devices can either be written (output) to, read (input) from, or both. This may be a consequence of any of the three tiers. As mentioned before, It doesn't matter what is connected to a printer port - the hardware has no provision for input. Thus, these devices are write-only. Some disk drives have write-lock switches to prevent overwriting important data. Thus, they are read-only. And some media may be read-only (DVDROM), write-only (paper in a printer), or write-once (such as a DVD-R disc). Note that write-once may actually be written to on more than one occasion, but the data that has already been written cannot be overwritten with new data. Terminal devices are read/write, although the data being read most likely is not the data written. Thus, it is important to note that just because a device is read/write doesn't mean that it is reading and writing the same data - at least in the case of streams. The flags are fairly straight-forward except that the DF_Access_Mask must be used to compare against the DFM_* constants. Also, the write-once mode is indicated by the combination of both the read-lock and write-lock flags together. Here are the access mode flag values:

    // Device access mode...
    DF_Write_Lock = 4 ;
    DF_Read_Lock = 8 ;
    DFM_RW = 0 ;
    DFM_Read_Only = 4 ;
    DFM_Write_Only = 8 ;
    DFM_Write_Once = 12 ;
    DF_Access_Mask = 12 ;

Any device with media has a position on that media at any given time. For instance, the read/write heads of a disk or tape drive are positioned over a particular location of the magnetic media. Terminals have a print-head (for hardcopy) or cursor (for video) that indicates where the next character will be placed. In the case of sequential devices, this position can be interrogated by an application. Some devices (such as tape drives) don't keep track of position of the media unless it is at the beginning of the tape or the end of the tape. But UOS will keep track of this information as it performs read/write/other operations on the tape. Likewise, terminal devices usually don't report back their positions, so UOS also keeps track of the current position for them as well. However, this isn't foolproof. For instance, a user could switch a terminal to local mode and change the position, then put it back online. In the case of tapes, the driver usually notifies UOS if the device goes offline, so UOS will mark the position as unknown.

In the case of a random-access store, the current position is of little use to most applications, since they are typically accessed via the file system. Even if the application is accessing the store in a non-file-structured mode, the likelihood of multiple users accessing the store means that the current position is constantly changing. However, UOS still uses this for such purposes as I/O request ordering for performance (a topic we will cover in a future article). For random-access stores, the position will be the linear sector (block). For tapes, the position will be the record offset from the beginning of the tape.

In the case of terminals, the definition of the position depends upon several factors. In the case of graphics terminals, the position might be a pixel position, while non-graphics terminals are positioned via character cell. We will discuss this further in the next article. The position-related flags are in the Status field and are described below.

Other flags
Now let's look at the remaining flags.

    // Other device flags...
    DF_Sequential = 16 ; // Media is sequential, vs random-acccess
    DF_Unloadable = 32 ; // Media can be removed
    DF_Integral = 64 ; // Data transfers must be integrals of minimum size
    DF_Rewrite = 512 ; // Destructive rewrite

DF_Sequential: This flag indicates that the device is accessed sequentially. Otherwise the device is random-access.
DF_Unloadable: This device has unloadable media. Otherwise the device has no media or it is integral to the drive and cannot be removed.
DF_Integral: Data transfers to this device must be a multiple of the minimum data size for the device. This is typical for disk drives. Otherwise, the size of data reads/writes can vary between the minimum and maximum data size values (see discussion later in this article).
DF_Rewrite: Rewriting any data destroys part/all of the rest of the data. Some tape drives don't allow updates of existing data, for instance. Attempts to do will essentially write a logical end-of-tape mark after the write, thus erasing any data that occurred after that point.

Here is the updated TDevice_Info structure:

type TDevice_Info = packed record
                        Flags : word ; // See DF*
                        Status : int64 ; // See DS*
                        Controller : word ;
                        Device_Unit : word ;
                        Position : int64 ;
                        Minimum_Rate : word ; // BPI, Bps, hertz
                        Maximum_Rate : word ;
                        Current_Rate : word ;
                        Minimum_Size : cardinal ; // Minimum atomic data size, in bytes
                        Maximum_Size : cardinal ; // Maximum data size, in bytes
                        Current_Size : cardinal ;
                    end ;

Flags: We've discussed the flags above.
Status: Status flags (see below).
Controller: Controller index (discussed in past articles).
Device_Unit: Unit index on controller (discussed in past articles).
Position: The current position of the media relative to the read/write head. The exact meaning depends upon circumstances. For a store, this is the logical block/sector. For a terminal/printer, the high 32 bits are the row and the low 32 bits are the column.
Minimum_Rate and Maximum_Rate: These values vary based on the device. They indicate the ranges supported by the device. In the case of magnetic tapes, this is the BPI (Bits Per Inch). For serial cable connections, this is the baud rate. For microphones or sound cards, this is the hertz. A value of 0 indicates unknown or not used. A value of $FFFF indicates that this setting is not used.
Current_Rate: This is the current rate for the device.
Minimum_Size and Maximum_Size: These values indicate the minimum and maximum amount of data, in bytes, for a single transfer. For a terminal, the minimum would be 1. For a hard disk with a 4,096-byte sector size, the value would be 4,096. For a network device, this would be tbe packet size ranges. 0 indicates unknown.
Current_Size: This is the current data size, for devices which have a fixed size, such as disks.

The Flags indicate characteristics that do not change (at least until different media is loaded). Status indicates the current state of the device and/or media. These flags will change as the device is used. Here are the status flag constants.

// Device Status...
const DS_Disabled = 1 ; // True if device is disabled
      DS_Media_Not_Present = 2 ; // Media not present
      DS_Timeout = 4 ; // Device operation timed out
      DS_DataCheck = 8 ; // A non-correctable CRC error occurred (parity error on character devices)
      DS_CorrectedError = 16 ; // A corrected CRC or parity error occurred.
      DS_DoorLocked = 32 ; // Door is locked on drive.
      DS_Overrun = 64 ; // Data overrun
      DS_Ready = 128 ; // Device is ready (on-line and not busy)
      DS_PositionError = 256 ; // Seek to invalid position error
      DS_Open = 512 ; // Cover/door is currently open
      DS_OffLine = 1024 ; // Device is on-line
      DS_NoECC = 2048 ; // Don't perform error checking/correction
      DS_Unloaded = 4096 ; // Heads/media not loaded.
      DS_FrameError = 8192 ; // Serial framing error.
      DS_Break = 16384 ; // Break received.
      DS_DualAccess = $8000 ; // Device is in dual/multi-access mode.

      // Position flags...
      DSP_BOM = $10000 ; // At beginning of media
      DSP_LEOM = $20000 ; // At logical end of media
      DSP_PEOM = $30000 ; // At physical end of media
      DSP_Unknown = 0 ; // Unknown position
      DS_Position_Mask = $30000 ;

DS_Media_Not_Present: Indicates that removable media is not present.
DS_Timeout: Indicates the I/O operation timed out.
DS_DataCheck: Indicates a data corruption error.
DS_CorrectedError: Indicates that the operation completed with a corrected error.
DS_DoorUnlocked: Indicates that the drive door is currently not locked.
DS_Overrun: Indicates a data overrun. Usually this indicates that the input is coming in faster than the system is reading it.
DS_Ready: Indicates that the device is ready for I/O operations.
DS_PositionError: Inidcates a positioning error.
DS_Open: Indicates that the drive door is open.
DS_OffLine: Indicates that the device is connected, but offline.
DS_NoECC: Indicates that no error checking/correction is currently enabled.
DS_Unloaded: Indicates that the media is present but the read/write head(s) are not loaded.
DS_FrameError: Applicable only to serial interfaces. Indicates that the tier one data rate differs from the tier two (or three) device.
DS_Break: Applicable only to serial interfaces. Indicates that a break signal was received.
DS_DualAccess: Indicates that the device is in dual-access or multi-access mode. For devices with multiple I/O ports/channels.
DS_Position_Mask: This can be used to mask out the position value from the status, which can be compared to the DSP_* flag values.

For illustrative purposes, here is a collection of various devices and what the corresponding device information flags would be.
Access Access
Device Class Method Mode  Comments
Disk Store Random DFM_RW
Disk Store Random DFM_Read_Only
CDROM Store Random DFM_Read_Only  CD
CDROM Store Random DFM_Write_Once  CD drive with writeable media
DVDROM Store Random DFM_Read_Only  DVD
DVDROM Store Random DFM_Write_Once  DVD drive with writeable media
Paper Tape Store Sequential DFM_Write_Once
Paper Tape Punch Store Sequential DFM_Write_Only
Paper Tape Reader Store Sequential DFM_Read_Only
Mag Tape Store Sequential DFM_RW
Terminal Stream Sequential DFM_RW
Printer Stream Sequential DFM_Write_Only
Network Stream Sequential DFM_RW
Scanner Stream Sequential DFM_Read_Only
Card Reader Stream Sequential DFM_Read_Only
Card Punch Stream Sequential DFM_Write_Only
Camera Stream Sequential DFM_Read_Only

A couple of things are obvious from this table. CDs and DVDs - and, in fact, any other optical media - are indistinguishable in terms of flags. From the viewpoint of UOS, this is fine as there is no operational difference between them. A read-only disk is indistinguishable from a read-only CD or DVD. Again, this is fine, as there is no operational difference between those and a read-only disk. In terms of sequential-access streams, the device flags are not sufficient to distinguish between all of them (for instance, a terminal and a network card appear the same). Thus, streams need additional information, which will we discuss in the next article.

We introduced the FiP's TDevice class in article 26. Here we expand it a bit. So far, we have added some properties and the handlers for those properties. Here is the complete class:

type TDevice = class
                   public // API...
                       // Identification...
                       Index : integer ; // HAL device index
                       Parent_Index : integer ;
                       Info : TDevice_Info ;
                       RInfo : TRAID_Info ; // RAID information
                       Store : TCOM_Store64 ; // Nil if not a store
                       Name : string ; // Named GPT partition

                       // For RAID...
                       RAID : integer ; // RAID level
                       RAID_Check : boolean ; // True if there is an inconsistency between RAID members

                       // For terminals...
                       Attached : cardinal ; // PID attached to terminal (0 if none)

                       // Other...
                       Mounted : boolean ;
                       Owner : TOwnership ;

                       // For stores...
                       FS : TUOS_File_System ; // File system for device (nil if non-structured store)
                       RAM : boolean ; // True if a RAM disk

                   public // Constructors and destructors...
                       constructor Create ;

                   protected // Property handlers...
                       function Get_Device_Class : integer ; virtual ;
                       function Get_Access_Mode : integer ; virtual ;
                       function Get_Position : integer ; virtual ;
                       function Get_Sequential : boolean ; virtual ;
                       function Get_Removable : boolean ; virtual ;
                       function Get_Integral : boolean ; virtual ;
                       function Get_Destructive_Rewrite : boolean ; virtual ;

                   published // Properties...
                       property Device_Class : integer
                           read Get_Device_Class ;
                       property Access_Mode : integer
                           read Get_Access_Mode ;
                       property Position : integer
                           read Get_Position ;
                       property Sequential : boolean
                           read Get_Sequential ;
                       property Removable : boolean
                           read Get_Removable ;
                       property Integral : boolean
                           read Get_Integral ;
                       property Destructive_Rewrite : boolean
                           read Get_Destructive_Rewrite ;
               end ; // TDevice

// Constructors and destructors...

constructor TDevice.Create ;

    inherited Create ;

    Index := -1 ;
    Parent_Index := -1 ;
    RAID := -1 ;
end ;

// Property handlers...

function TDevice.Get_Device_Class : integer ;

    Result := Info.Flags and DF_Class_Mask ;
end ;

function TDevice.Get_Access_Mode : integer ;

    Result := Info.Flags and DF_Access_Mask ;
end ;

function TDevice.Get_Position : integer ;

    Result := Info.Flags and DF_Position_Mask ;
end ;

function TDevice.Get_Sequential : boolean ;

    Result := ( Info.Flags and DF_Sequential ) <> 0 ;
end ;

function TDevice.Get_Removable : boolean ;

    Result := ( Info.Flags and DF_Unloadable ) = 0 ;
end ;

function TDevice.Get_Integral : boolean ;

    Result := ( Info.Flags and DF_Integral ) <> 0 ;
end ;

function TDevice.Get_Destructive_Rewrite : boolean ;

    Result := ( Info.Flags and DF_Rewrite ) <> 0 ;
end ;

In the next article, we will discuss streams in more detail.