Files and I/O#

UNIX systems use files as a common interface for many resources. Regular files, directories, devices, pipes, and sockets have different behavior, but they are all named in the filesystem and many of them support the same basic operations: open(), read(), write(), close(), and select().

Common attributes of all (UNIX) files#

Files live in the filesystem namespace under /. UNIX paths do not use drive letters. Each file has a name, an owning user, an owning group, and permission bits for the owner, group, and other users.

Many filesystems also store extended attributes, creation time, last access time, and modification time. Beyond these shared attributes, file types vary widely in structure and behavior.

Types of Files in Unix#

UNIX file types include regular files, symbolic links, directories, block device files, character device files, named pipes, UNIX domain sockets, and, on Solaris, doors.

Regular Files#

Regular files persist program data in a filesystem. They have ownership, permissions, and a defined size. They can normally be accessed sequentially or with random access through seek operations.

Sparse files are an important exception to the simple size model. A filesystem may represent large unwritten regions without allocating disk blocks for every byte.

Folders#

Directories map names to filesystem entries. Early UNIX implementations represented directories as special files that listed other files. Modern systems still expose some of that model, but programs modify directories through directory-specific system calls rather than by writing directory bytes directly.

Directories do not behave like regular files. The execute bit controls whether a process can traverse the directory and use it in path resolution. Read permission controls whether entries can be listed.

Symbolic Links#

A symbolic link is a file whose contents point to another file or directory. The target may be an absolute path or a relative path. If the target does not exist, the symbolic link is broken.

Most filesystem operations follow a symbolic link and operate on its target. unlink() is the main exception: it removes the symbolic link itself.

Block Device Files#

Block device files expose devices that operate in fixed-size blocks. Common examples are disks, optical drives, USB media, and RAM disks. Block devices usually support random access and buffered I/O.

Operating systems expose block devices through special filesystems or device-management tools. Linux has used several mechanisms over time, including device files and special filesystems such as devfs and sysfs.

Character Device Files#

Character device files expose stream-oriented devices. Common examples include terminals, serial ports, modems, network interfaces, sound devices, video devices, and tape drives.

Most character devices do not support random access. When they do, seeking is often expensive or device-specific.

Named Pipes/FIFOs#

Named pipes are pipes with names in the filesystem. They let unrelated processes communicate even when the processes do not share a parent-child relationship. We cover pipes in more detail in the IPC chapter.

Unix Domain Sockets#

UNIX domain sockets are sockets with names in the filesystem. They are similar to named pipes, but can support stream or datagram communication. Unlike network sockets, they do not use TCP/IP or UDP/IP.

Filesystem System Calls#

Most UNIX systems have many file-oriented system calls. The basic pattern is simple: open a resource, perform operations on its file descriptor, and close the descriptor when finished.

Filesystem System Calls#

Filesystem Calls#
Function	Description
`open()`	opens or creates files and returns a file descriptor
`creat()`	creates new files
`close()`	closes a file descriptor
`lseek()`	updates a file descriptor’s current file offset
`read()`	reads data from a file descriptor into a buffer
`write()`	writes data from a buffer to a file descriptor
`dup()`	duplicates one file descriptor
`dup2()`	updates a file descriptor to point to another one
`fcntl()`	changes file properties such as asynchronous I/O and file locks
`ioctl()`	handles device-specific control operations
`stat()`	returns permissions, size, timestamps, and other metadata
`access()`	tests for read, write, execute, or existence of a file
`umask()`	updates the file creation mask
`chmod()`	updates permission bits

More Filesystem System Calls#

Filesystem Calls#
Function	Description
`chown()`	changes file user or group ownership
`truncate()`	changes the length of a file
`link()`	creates a hard link
`unlink()`	removes a name from the filesystem
`rmdir()`	deletes an empty directory
`remove()`	combines `unlink()` and `rmdir()` behavior
`rename()`	renames a file, possibly moving it to another directory
`symlink()`	creates a symbolic link
`readlink()`	reads the value stored in a symbolic link
`utime()`	updates access and modification times
`mkdir()`	creates a directory
`opendir()`	opens a directory for reading
`readdir()`	reads the next directory entry
`rewinddir()`	resets directory reading to the beginning
`closedir()`	closes a directory descriptor
`chdir()`	changes the current working directory
`getcwd()`	gets the current working directory
`sync()`	flushes filesystem buffers to disk

Opening Files with open()#

open() opens a file and returns a file descriptor.

int open(const char *pathname, int flags, mode_t mode);
int open(const char *pathname, int flags);

pathname is the path to the file. flags controls how the file is opened. Common flags include O_APPEND, O_ASYNC, O_CREAT, O_DIRECT, O_SYNC, and O_TRUNC.

When O_CREAT is used, mode supplies permission bits. These are usually written in octal. For example, 0700 gives the owner read, write, and execute permission, while group and other users receive no permissions. 0660 gives the owner and group read/write permission.

The return value is a file descriptor on success and -1 on error.

Closing files with close()#

close() releases a file descriptor.

int close(int fd);

The argument is a descriptor returned by calls such as open(), dup(), or pipe(). The return value is 0 on success and -1 on failure.

Writing to a File#

write() copies bytes from a user buffer to a file descriptor.

ssize_t write(int fd, const void *buf, size_t count);

fd is the file descriptor, buf is the source buffer, and count is the number of bytes to write. A return value of -1 means an error occurred. A successful call usually returns count, but code that needs to be robust should handle partial writes.

Writing Example#

The systems-code-examples/read-write-demo directory contains small separate examples for direct read() and write() calls.

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>

int main(int argc, char* argv[])
{
    const char *data = "foobar";
    int fd = open("file", O_CREAT | O_TRUNC | O_RDWR, 0666);
    size_t length = strlen(data), offset = 0;
    while(length > 0)
    {
        size_t written = write(fd, data + offset, length);
        offset += written;
        length -= written;
    }
    close(fd);
}

Key points:

open() creates or truncates the file named file.
strlen() is used to compute the number of bytes to write.
The loop handles the possibility that write() writes fewer bytes than requested.
offset advances through the source string as bytes are written.
close() releases the descriptor after the write loop finishes.

Reading from a File#

read() copies bytes from a file descriptor into a user buffer.

ssize_t read(int fd, void *buf, size_t count);

The return value is -1 on error, 0 at end of file, and otherwise the number of bytes read. The number of bytes returned may be smaller than count.

Reading Example#

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>

int main(int argc, char* argv[])
{

    int fd = open("file", O_RDONLY, 0666);
    size_t length;
    char buffer[5];
    while((length = read(fd, &buffer[0], 5)) != 0)
    {
        write(1, &buffer[0], length);
    }
    close(fd);

}

Key points:

open() opens the existing file for reading.
The buffer is deliberately small so the loop has to run more than once for longer input.
read() returns 0 at end of file, which terminates the loop.
write(1, ...) writes the bytes to standard output.
This example omits detailed error handling so the read/write pattern is easy to see.

Complete File I/O Example#

The systems-code-examples/file_rw example combines creation, writing, reading, closing, and removing a file.

#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>

const char *file_data = "1234567890abcdefghijklmnopqrstuvwxyz";
const char *file_name = "temp.dat";

int main(int argc, char* argv[])
{

    int fd = open(file_name, O_CREAT | O_TRUNC | O_RDWR, 0666);
    if(fd == (-1))
    {
        printf("open returned (-1)\n");
        return (-1);
    }

    size_t length = strlen(file_data);
    size_t offset = 0;
    while(length > 0)
    {
        size_t written = write(fd, file_data + offset, length);
        if(written == (-1))
        {
            printf("write returned (-1)\n");
            return (-1);
        }
        printf("wrote %ld bytes\n", written);
        offset += written;
        length -= written;
    }

    if(close(fd) == (-1))
    {
        printf("close returned (-1)\n");
        return (-1);
    }

    printf("file contents written.\n");

    fd = open(file_name, O_RDONLY, 0666);
    if(fd == (-1))
    {
        printf("open returned (-1)\n");
        return (-1);
    }

    char buffer[5];
    //read in a loop until zero bytes are read
    //zero return value means EOF
    while((length = read(fd, &buffer[0], 5)) != 0)
    {
        if(length == (-1))
        {
            printf("read returned (-1)\n");
            return (-1);
        }
        write(1, &buffer[0], length);
    }
    printf("\n");

    if(close(fd) == (-1))
    {
        printf("close returned (-1)\n");
        return (-1);
    }

    if(unlink(file_name) == (-1))
    {
        printf("unlink returned (-1)\n");
        return (-1);
    }

    return 0;
}

Key points:

The first open() creates temp.dat, truncating any existing file with the same name.
The write loop keeps going until all bytes from file_data are written.
The file is closed before it is reopened for reading.
The read loop writes each chunk to standard output as it is read.
unlink() removes the file’s name from the filesystem at the end of the example.

Seeking within a File#

Not all files support seeking. Regular files usually do. Pipes, sockets, and many character devices do not.

lseek() changes the current file offset for a descriptor.

off_t lseek(int fd, off_t offset, int whence);

whence is usually SEEK_SET for the beginning of the file, SEEK_CUR for the current offset, or SEEK_END for the end of the file. Seeking beyond the current end of a file can create a sparse file on filesystems that support sparse allocation.

Standard File Descriptors#

Every process starts with three standard file descriptors.

stdin: Standard input. The default descriptor value is 0.
stdout: Standard output. The default descriptor value is 1.
stderr: Standard error. The default descriptor value is 2.

The parent process may redirect any of these before the child program runs.

Duplicating File Descriptors#

dup() duplicates a file descriptor and returns a new descriptor number.

int dup(int fd);

Programs use duplicated descriptors when they need another reference to the same open file. They are also useful in shell-style redirection.

Redirecting File Descriptors#

dup2() redirects one descriptor to refer to another open file.

int dup2(int oldfd, int newfd);

If newfd is already open, dup2() closes it first. After the call, newfd refers to the same open file description as oldfd. This is how shells redirect standard input, output, and error.

Reading Folders#

Directories are read with directory-specific library calls such as opendir(), readdir(), and closedir().

Directory Traversal in C#

The systems-code-examples/dir_list example lists entries in the root directory.

#include <stdio.h>
#include <sys/types.h>
#include <dirent.h>
#include <errno.h>

int main(int argc, char* argv[])
{
    const char *dir = "/";

    DIR *d = opendir(dir);

    struct dirent *de;
    while((de = readdir(d)) != NULL)
    {
        printf("name: %s\n", de->d_name);
    }

    closedir(d);
    return 0;
}

Key points:

opendir() returns a DIR* handle for reading directory entries.
readdir() returns one struct dirent at a time.
The loop ends when readdir() returns NULL.
closedir() releases the directory handle.
The example prints entry names, not full paths.

Directory Traversal in C++#

The systems-code-examples/dir_list_cpp example performs the same basic operation with C++17 filesystem support.

#include <iostream>
#include <filesystem>

int main() {
    std::string dir = "/";

    try {
        for (const auto& entry : std::filesystem::directory_iterator(dir)) {
            std::cout << "name: " << entry.path().filename().string() << "\n";
        }
    } catch (std::filesystem::filesystem_error& e) {
        std::cout << "An error has occurred: " << e.what() << '\n';
    }

    return 0;
}

Key points:

std::filesystem::directory_iterator wraps the directory traversal pattern in a C++ range-like interface.
The exception handler catches filesystem errors, such as permission problems.
entry.path().filename() extracts just the entry name.
The underlying work still depends on operating system directory operations.

Text Input#

Text-oriented programs often read a line, split it into words, and then process those words. C gives you low-level tools for this, which means programs must be careful about buffer sizes, terminators, and ownership of allocated memory.

Bounded Line Input Example#

The systems-code-examples/better_gets example implements a bounded line input function. The surrounding program also demonstrates signal and child-process handling, but the relevant part for this chapter is better_gets().

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <unistd.h>
#include <sys/wait.h>
#include <sys/types.h>

#define MAX_PID (1000)

int childProcesses[MAX_PID];

void addProcess(int childPid)
{
    printf("add(%d)\n", childPid);
    for(int i = 0; i < MAX_PID; i++)
    {
        if(childProcesses[i] == (-1))
        {
            childProcesses[i] = childPid;
            printf("added child pid = %d\n", childPid);
            return;
        }
    }
}

void removeProcess(int childPid)
{
    printf("remove(%d)\n", childPid);
    for(int i = 0; i < MAX_PID; i++)
    {
        if(childProcesses[i] == childPid)
        {
            childProcesses[i] = (-1);
            printf("removed child pid = %d\n", childPid);
        }
    }
}

int countProcesses()
{
    int count = 0;
    for(int i = 0; i < MAX_PID; i++)
    {
        if(childProcesses[i] >= 0)
        {
            count += 1;
        }
    }
    return count;
}

void clean_up_child_process(int signal_number)
{
    printf("SIGCHLD recieved.\n");
    int status;
    pid_t pid;
    while((pid = waitpid(-1, &status, WNOHANG)) > 0)
    {
        if(pid > 0)
        {
            printf("pid %d exited\n", pid);
            removeProcess((int)pid);
        }
    }
    return;
}

char input[12];

int read_from_stdin()
{
    char buffer;
    int error;
    while((error = read(0, &buffer, 1)) == (-1))
    {
        if(error == 0)
        {
            return EOF;
        }
    }
    return buffer;
}

void better_gets(char *buff, int len)
{
    int i;
    for(i = 0; i < len-1; i++)
    {
        int val = read_from_stdin();
        if(val == EOF)
        {
            break;
        }
        buff[i] = val;
        if(buff[i] == '\n')
        {
            break;
        }
    }
    buff[i] = (char)0;
    //printf("read: [%s]\n", buff);
}

void parent()
{
    printf("parent continuing....\n");
    for(int i = 0; i < 5; i++)
    {
        printf(">");
        better_gets(&input[0], 12);
        printf("you typed: %s\n", &input[0]);
    }
    while(countProcesses() > 0) {}
    printf("parent exiting\n");
}

void child()
{
    printf("child starting.\n");
    sleep(5);
    printf("child exiting.\n");
    exit(EXIT_SUCCESS);
}

int main(int argc, char** argv)
{
    for(int i = 0; i < MAX_PID; i++)
    {
        childProcesses[i] = (-1);
    }

    struct sigaction sigchld_action;
    memset (&sigchld_action, 0, sizeof (struct sigaction));
    sigchld_action.sa_handler = &clean_up_child_process;
    sigaction (SIGCHLD, &sigchld_action, NULL);

    for(int i = 0; i < 5; i++)
    {
        int childProcess = fork();
        if(childProcess == 0)
        {
            child();
            return 0;
        }
        else if(childProcess > 0)
        {
            addProcess(childProcess);
        }
        else
        {
            printf("fork failed\n");
        }
    }
    parent();

    return 0;
}

Key points:

better_gets() accepts both a buffer and a maximum length.
The loop stops before the buffer is full so there is room for the null terminator.
Input stops at newline, end of file, or the configured length.
The implementation reads from descriptor 0, which is standard input.
The rest of the example shows why reusable input helpers matter in larger systems programs.

Tokenizing Input with strtok()#

The systems-code-examples/strtok example splits a string into tokens using whitespace delimiters.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int SplitString(char* str, char* tokens[], const int maxTokens);
void testSplitString();
void assertEqualsStr(const char *expected, const char *actual);
void assertEqualsInt(const int expected, const int actual);

int main(int argc, char* argv[])
{
    if(argc == 2 && strcmp(argv[1], "--test") == 0)
    {
        testSplitString();
        printf("tests pass!\n");
    }
    return 0;
}

void testSplitString()
{
    const int maxTokens = 20;
    char testStr[] = "hello\tworld testing 123\nhello again";
    char* tokens[maxTokens];

    int tokenCount = SplitString(testStr, tokens, maxTokens);

    assertEqualsInt(6, tokenCount);
    assertEqualsStr("hello", tokens[0]);
    assertEqualsStr("world", tokens[1]);
    assertEqualsStr("testing", tokens[2]);
    assertEqualsStr("123", tokens[3]);
    assertEqualsStr("hello", tokens[4]);
    assertEqualsStr("again", tokens[5]);
}

void assertEqualsStr(const char* expected, const char* actual)
{
    if(strcmp(expected, actual) != 0)
    {
        printf("expected <%s> but was <%s>\n", expected, actual);
        exit(1);
    }
}

void assertEqualsInt(const int expected, const int actual)
{
    if(expected != actual)
    {
        printf("expected <%d> but was <%d>\n", expected, actual);
        exit(1);
    }
}

int SplitString(char* str, char* tokens[], const int maxTokens)
{
    int i = 0;
    for(i = 0; i < maxTokens; i++)
    {
        char* token = strtok(str, " \t\n");
        str = NULL;
        tokens[i] = token;
        if(token == NULL)
        {
            break;
        }
    }
    return i;
}

Key points:

strtok() modifies the input string by replacing delimiters with null terminators.
The first call receives the original string. Later calls pass NULL to continue tokenizing the same string.
The example stores pointers into the original string; it does not copy each token.
The test function makes the expected token sequence explicit.

Dynamic String Buffer#

The systems-code-examples/strbuffer_t example builds a small dynamic string buffer. It supports appending characters and returning a heap-allocated C string.

/*
 * Many students come to C from Java.
 *
 * This is a demonstration of how to create a StringBuffer lite in C.
 * The buffer is used by declaring a strbuffer_t, which acts as a handle for all operations.
 */

#ifndef _STRINGBUFFER_H_
#define _STRINGBUFFER_H_

typedef struct _strbuffer_t
{
    char* data;
    int capacity;
    int increment;
    int pos;
} strbuffer_t;

#define MIN_CAPACITY 80
#define MIN_INCREMENT 10

extern void strbuffer_new(strbuffer_t* buffer);

extern void strbuffer_init(strbuffer_t* buffer, int capacity, int increment);

extern void strbuffer_resize_if_needed(strbuffer_t* buffer);

extern void strbuffer_append(strbuffer_t* buffer, char c);

extern char* strbuffer_tostring(strbuffer_t* buffer);

extern void strbuffer_reset(strbuffer_t* buffer);

extern char* strbuffer_getline(strbuffer_t* buffer, int *eof);


#endif

#include "strbuffer.h"

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

void strbuffer_new(strbuffer_t* buffer)
{
    strbuffer_init(buffer, MIN_CAPACITY, MIN_INCREMENT);
}

void strbuffer_init(strbuffer_t* buffer, int capacity, int increment)
{
    if (capacity < MIN_CAPACITY)
        buffer->capacity = MIN_CAPACITY;
    else buffer->capacity = capacity;

    if (increment < MIN_INCREMENT)
        buffer->increment = MIN_INCREMENT;
    else buffer->increment = increment;

    buffer->data = (char*) malloc(buffer->capacity * sizeof(char));
    buffer->pos = 0;
}


void strbuffer_resize_if_needed(strbuffer_t* buffer)
{
    if (buffer->pos < buffer->capacity)
        return;

    int new_capacity = buffer->capacity + buffer->increment;
    //printf("Resizing to %d\n", new_capacity);
    buffer->data = (char*) realloc(buffer->data, new_capacity * sizeof(char));
    buffer->capacity = new_capacity;
}

void strbuffer_append(strbuffer_t* buffer, char c)
{
    strbuffer_resize_if_needed(buffer);
    buffer->data[buffer->pos++] = c;
}

char* strbuffer_tostring(strbuffer_t* buffer)
{
    char* repr = (char*)malloc((buffer->pos+1) * sizeof(char));
    memcpy(repr, buffer->data, buffer->pos * sizeof(char));
    repr[buffer->pos] = 0;
    return repr;
}

void strbuffer_reset(strbuffer_t *buffer)
{
    buffer->pos=0;
}


char* strbuffer_getline(strbuffer_t* buffer, int *eof)
{
    int c;
    *eof = 0;
    strbuffer_reset(buffer);
    for(;;)
    {
        c = getchar();
        if (c == '\n' || c == '\r' || c == 0 || c == EOF)
            break;
        strbuffer_append(buffer, c);
    }
    if (c == EOF) *eof = 1;
    return  strbuffer_tostring(buffer);
}

Key points:

strbuffer_t stores the backing array, capacity, resize increment, and current position.
strbuffer_append() checks whether the buffer needs to grow before storing the next character.
realloc() is used to grow the buffer while preserving existing contents.
strbuffer_tostring() returns a separate heap allocation, so callers must eventually free it.
strbuffer_getline() builds a line one character at a time.

String Buffer Input Example#

The systems-code-examples/strbuffer-demo directory shows how the string buffer can be used to read lines and split them into words.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

/*
 * This example shows a strbuffer_t-enabled getline() like function
 * and how to use strtok() to separate text with whitespace delimiters.
 */

#include "strbuffer.h"

#define DELIMS " \n\r\t"

int main(int argc, char* argv[])
{
    strbuffer_t buffer;
    strbuffer_init(&buffer, 80, 10);
    for (;;)
    {
        int eof;
        char* line = strbuffer_getline(&buffer, &eof);
        if (eof) break;
        char* next_word = strtok(line, DELIMS);
        while (next_word != NULL)
        {
            printf("word: %s\n", next_word);
            next_word = strtok(NULL, DELIMS);
        }
        free(line);
    }
}

Key points:

The example uses strbuffer_getline() so line length is not fixed by a small stack buffer.
Each returned line is tokenized with strtok().
The loop exits when the input function reports end of file.
Each heap-allocated line is freed after it is processed.

Regular Expression Example#

The systems-code-examples/regex-demo example reads text and extracts matches using the POSIX regular expression API.

/* Credit: https://www.lemoda.net/c/unix-regex/ */
/* Credit: GNU C docs */
/* Credit to self for fixing some problems with handling on OS X and Linux */


#define WORD_REGEX "(\\w+)"
#define ALT_WORD_REGEX "([[:digit:]]+)[^[:digit:]]+([[:digit:]]+)" // not used (yet)

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <regex.h>

#include "strbuffer.h"
#include "regex-match.h"

void process_regex(const char* find_text, const char* regex_text);

int main (int argc, char ** argv)
{
    int eof;
    const char * regex_text = WORD_REGEX;

    // if no arguments read text from stdin; else use argv[1] as text
    if (argc < 2)
    {
        strbuffer_t buffer;
        strbuffer_init(&buffer, 80, 10);
        char* line = strbuffer_getline(&buffer, &eof);
        process_regex(line, regex_text);
        if (line) free(line);
    }
    else
    {
        process_regex(argv[1], regex_text);
    }
    return 0;
}

void process_regex(const char* find_text, const char* regex_text)
{
    regex_t regex;
    basic_matchlist_t match_list;
    regex_compile (&regex, regex_text);
    basic_matchlist_init(&match_list);
    regex_match (&regex, find_text, &match_list);
    basic_matchlist_print(&match_list);
    basic_matchlist_delete(&match_list);
    regfree (&regex);
}

Key points:

If the program receives an argument, it processes that argument as the input text.
If there is no argument, it reads a line from standard input with the string buffer helper.
regcomp() and related POSIX regex calls are wrapped in helper functions from the example.
The match list is explicitly initialized, printed, and deleted.
regfree() releases resources owned by the compiled regular expression.

Character List Case Study#

The systems-code-examples/charlist_t example stores characters in a tail queue. It is a small data-structure case study for text processing.

#include <stdio.h>
#include <stdlib.h>

#include "charlist.h"

void charlist_add_char(struct charlist_t* head_ptr, char c)
{
    struct charentry_t* my_entry = malloc(sizeof(struct charentry_t));
    my_entry->c = c;
    TAILQ_INSERT_TAIL(head_ptr, my_entry, entries);
}

void charlist_add_string(struct charlist_t* head_ptr, char* text)
{
    for (char* next_char=text; *next_char != '\0'; next_char++)
        charlist_add_char(head_ptr, *next_char);
}

void charlist_print(struct charlist_t* head_ptr)
{
    struct charentry_t* np;
    TAILQ_FOREACH(np, head_ptr, entries)
    {
        printf("%c\n", np->c);
    }
}

char* charlist_tostring(struct charlist_t* head_ptr)
{
    /* will cache size later */
    int size = charlist_size_slow(head_ptr);
    char* text = (char*)malloc(size * sizeof(char));
    struct charentry_t* np;
    int i=0;
    TAILQ_FOREACH(np, head_ptr, entries)
    {
        text[i++] = np->c;
    }
    text[i] = '\0';
    return text;
}

int charlist_size_slow(struct charlist_t* head_ptr)
{
    struct charentry_t* np;
    int size = 0;
    TAILQ_FOREACH(np, head_ptr, entries)
    {
        size++;
    }
    return size;
}

void charlist_delete(struct charlist_t* head_ptr)
{
    struct charentry_t* n1, *n2;
    n1 = TAILQ_FIRST(head_ptr);
    while (n1 != NULL)
    {
        n2 = TAILQ_NEXT(n1, entries);
        free(n1);
        n1 = n2;
    }
}

void charlist_init(struct charlist_t* head_ptr)
{
    struct charlist_t head = TAILQ_HEAD_INITIALIZER(head);
    *head_ptr = head;
    /* create a tail queue */
    TAILQ_INIT(head_ptr);
}

Key points:

Each character is stored in a heap-allocated list entry.
TAILQ_INSERT_TAIL appends entries while preserving order.
charlist_tostring() converts the list back into a normal C string.
charlist_delete() walks the list and frees each entry.
This representation is flexible but much heavier than a contiguous string buffer.

Word Counting with Hash Tables#

The systems-code-examples/std-hsearch example uses the standard hsearch family to maintain word counts.

#include <stdio.h>
#include <stdlib.h>

#include "wordtable.h"

/*
 * This example is adapted from the man page for hsearch.
 * Annoyingly, that example has a single haswtable that made reuse of that code nearly impossible.
 * This one uses the _r() functions, which not only are able to work on different haswtables
 * but are also reentrant. Sadly, these are a bit GNU specific, but who doesn't use gcc? LOL
 */

static char *data[] = { "alpha", "bravo", "charlie", "delta",
                        "echo", "foxtrot", "golf", "hotel", "india", "juliet",
                        "kilo", "lima", "mike", "november", "oscar", "papa",
                        "quebec", "quebec", "romeo", "sierra", "tango", "uniform",
                        "victor", "whisky", "whisky", "x-ray", "yankee", "zulu", "zulu"
                      };


int main(void)
{
    wordtable_t wtable;
    wordtable_init(&wtable, 50);
    int data_size = sizeof(data) / sizeof(char*);
    int i;

    /* insert all words to get word counts */
    for (int i = 0; i < data_size; i ++)
    {
        wordentry_t* wp = wordtable_increment(&wtable, data[i]);
        printf("new word i=%d, data=%s, count=%ld, id=%p\n", i, data[i], wp->count, wp);
    }
    printf("\n");

    /* delete every 4th word */
    for (int i = 0; i < data_size; i += 4)
    {
        wordentry_t* wp = wordtable_decrement(&wtable, data[i]);
        printf("dec word i=%d, data=%s, count=%ld, id=%p\n", i, data[i], wp->count, wp);
    }
    printf("\n");

    for (int i = 0; i < data_size; i++)
    {
        wordentry_t* wp = wordtable_lookup(&wtable, data[i]);
        if (wp == NULL || wp->count == 0) continue;
        printf("non-zero word= %s, count=%ld, id=%p\n", data[i], wp->count, wp);
    }

    printf("\n");
    for (int i = 0; i < data_size; i++)
    {
        wordentry_t* wp = wordtable_lookup(&wtable, data[i]);
        if (wp == NULL || wp->count > 0) continue;
        printf("zero word= %s, count=%ld, id=%p\n", data[i], wp->count, wp);
    }

    for (int i = 0; i < data_size; i ++)
    {
        int result = wordtable_delete_entry(&wtable, data[i]);
        printf("deleting word %s, result=%d\n", data[i], result);
    }
    wordtable_delete(&wtable);
    exit(EXIT_SUCCESS);
}

Key points:

wordtable_init() prepares a hash table with a fixed capacity.
Repeated words increment an existing count rather than creating a new logical word.
The example looks up words after insertion and after decrementing counts.
Cleanup is explicit because the table owns allocated entries.
Hash tables are useful when fast lookup matters more than sorted order.

Word Counting with Trees#

The systems-code-examples/std-tsearch example stores word information in a binary search tree.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define __USE_GNU

#define _GNU_SOURCE     /* Expose declaration of tdestroy() */
#include <search.h>
#include <time.h>


static char *data[] = { "alpha", "bravo", "charlie", "delta",
                        "echo", "foxtrot", "golf", "hotel", "india", "juliet",
                        "kilo", "lima", "mike", "november", "oscar", "papa",
                        "quebec", "quebec", "romeo", "sierra", "tango", "uniform",
                        "victor", "whisky", "whisky", "x-ray", "yankee", "zulu", "zulu"
                      };


typedef struct _wordinfo_t
{
    char* word;
    int count;
} wordinfo_t;


static int compare_by_word(const void *pa, const void *pb)
{
    const wordinfo_t* wi_a = pa, *wi_b = pb;

    //printf("comparing %s to %s\n", wi_a->word, wi_b->word);
    return strcmp(wi_a->word, wi_b->word);
}

static void action(const void *nodep, VISIT which, int depth)
{
    //int *datap;
    wordinfo_t* datap;

    switch (which)
    {
    case preorder:
        break;
    case postorder:
        datap = *(wordinfo_t **) nodep;
        printf("word %s count %d depth %d\n", datap->word, datap->count, depth);
        break;
    case endorder:
        break;
    case leaf:
        datap = *(wordinfo_t **) nodep;
        printf("word %s count %d depth %d\n", datap->word, datap->count, depth);
        break;
    }
}

int main(void)
{
    int i;
    wordinfo_t* ptr;
    wordinfo_t** val;
    int data_size = sizeof(data) / sizeof(char*);
    void *root = NULL;

    for (i = 0; i < data_size; i++)
    {
        printf("insert word %s\n", data[i]);
        ptr = (wordinfo_t*) malloc(sizeof(wordinfo_t));
        ptr->word = data[i];
        ptr->count = 0;
        if (ptr == NULL) break;
        val = (wordinfo_t**) tsearch((void *) ptr, &root, compare_by_word);
        if (val != NULL)
        {
            wordinfo_t* entry = *val;
            entry->count++;
        }
    }
    twalk(root, action);
    tdestroy(root, free);
    exit(EXIT_SUCCESS);
}

Key points:

tsearch() inserts or finds an entry using the comparison function.
The comparison function orders entries by word text.
Repeated words find the existing node and increment its count.
twalk() traverses the tree and calls a callback for each node.
tdestroy() releases the tree at the end.

Looking Ahead: I/O Performance#

Good I/O performance depends on choosing the right buffering strategy. Small buffers increase the number of system calls and lower throughput. Very large buffers may cause longer waits before processing can continue. Programs often need a balance.

Producer-consumer designs can improve throughput. One thread or process reads data while another processes it. Memory-mapped I/O is another approach, and we return to it in the IPC chapter.

Simple I/O Performance Experiment#

The dd command gives a quick view of how block size can affect throughput.

dd if=/dev/zero of=tmp.dat bs=1 count=1000000 - 671 kB/s
dd if=/dev/zero of=tmp.dat bs=10 count=100000 - 5.9 MB/s
dd if=/dev/zero of=tmp.dat bs=100 count=10000 - 38.9 MB/s
dd if=/dev/zero of=tmp.dat bs=1000 count=1000 - 244 MB/s
dd if=/dev/zero of=tmp.dat bs=10000 count=100 - 537 MB/s
dd if=/dev/zero of=tmp.dat bs=100000 count=10 - 834 MB/s
dd if=/dev/zero of=tmp.dat bs=1000000 count=1 - 461 MB/s

Increasing block size usually improves performance until some other limit dominates. These are single-run measurements, so system load and cache state can affect the numbers.

Reading/Writing Performance#

Vectored I/O, also called gather-scatter I/O, combines multiple buffers into one read or write operation. This reduces the number of system calls when a program has data split across several buffers.

Vectored I/O Example#

The systems-code-examples/vectored_io example writes three separate buffers with one writev() call.

#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <sys/uio.h>

char *file_data1 = "1234567890";
char *file_data2 = "abcdefghijk";
char *file_data3 = "lmnopqrstuvwxyz";
const char *file_name = "temp.dat";

int main(int argc, char* argv[])
{

    int fd = open(file_name, O_CREAT | O_TRUNC | O_RDWR, 0666);
    if(fd == (-1))
    {
        printf("open returned (-1)\n");
        return (-1);
    }

    struct iovec buffers[3];
    buffers[0].iov_base = file_data1;
    buffers[0].iov_len = strlen(file_data1);
    buffers[1].iov_base = file_data2;
    buffers[1].iov_len = strlen(file_data2);
    buffers[2].iov_base = file_data3;
    buffers[2].iov_len = strlen(file_data3);

    int written = writev(fd, buffers, 3);
    if(written == (-1))
    {
        printf("writev returned (-1)\n");
        return (-1);
    }
    printf("wrote %d bytes\n", written);

    close(fd);
    return 0;
}

Key points:

Each struct iovec entry describes one buffer and its length.
writev() writes the buffers in order as one logical operation.
This avoids three separate write() system calls.
Vectored I/O is useful for data with a header, body, and trailer stored in separate buffers.
The file descriptor is still opened and closed with the same calls used for ordinary file I/O.

Files and I/O

Contents

Files and I/O#

Common attributes of all (UNIX) files#

Types of Files in Unix#

Regular Files#

Folders#

Symbolic Links#

Block Device Files#

Character Device Files#

Named Pipes/FIFOs#

Unix Domain Sockets#

Filesystem System Calls#

Filesystem System Calls#

More Filesystem System Calls#

Opening Files with open()#

Closing files with close()#

Writing to a File#

Writing Example#

Reading from a File#

Reading Example#

Complete File I/O Example#

Seeking within a File#

Standard File Descriptors#

Duplicating File Descriptors#

Redirecting File Descriptors#

Reading Folders#

Directory Traversal in C#

Directory Traversal in C++#

Text Input#

Bounded Line Input Example#

Tokenizing Input with strtok()#

Dynamic String Buffer#

String Buffer Input Example#

Regular Expression Example#

Character List Case Study#

Word Counting with Hash Tables#

Word Counting with Trees#

Looking Ahead: I/O Performance#

Simple I/O Performance Experiment#

Reading/Writing Performance#

Vectored I/O Example#