IPC Topics

Interprocess communication, or IPC, is the set of mechanisms processes use to exchange data or coordinate activity. UNIX systems provide many IPC mechanisms because different programs need different tradeoffs among speed, structure, persistence, and portability.

The main IPC mechanisms covered here are pipes, named pipes, signals, shared memory, memory mapped files, files, UNIX domain sockets, Solaris doors, and TCP/IP sockets.

IPC Performance Hierarchy

IPC mechanisms differ mostly in how much kernel involvement they need.

Shared memory and locks in shared memory are usually fastest after setup because ordinary reads and writes do not require a system call. Memory mapped files are also fast because they use the virtual memory system. Pipes, FIFOs, signals, and domain sockets require system calls and therefore context switches. Files and network sockets can be slower because they may involve filesystem or network activity.

Pipes

A pipe is a half-duplex byte stream between related processes. Data flows in one direction, from the write end to the read end.

Pipes are among the oldest UNIX IPC mechanisms. A parent process creates a pipe and then child processes inherit the pipe file descriptors across fork(). Passing data through a pipe requires system calls, but pipes are still fast enough for many command-line pipelines and process coordination tasks.

Pipe Descriptors

A pipe has two file descriptors: one for reading and one for writing.

On Linux, the call is:

int pipe(int pipefd[2]);

pipefd[0] is the read-only end. pipefd[1] is the write-only end.

On Windows, the comparable call is:

BOOL CreatePipe(
   HANDLE readHandle,
   HANDLE writeHandle,
   LPSECURITY_ATTRIBUTES attr,
   DWORD nSize);

The handles represent the read and write ends. The security attributes control inheritance and permissions, and nSize gives a suggested buffer size.

Pipe Example

This local pipe example creates a pipe, forks, and redirects standard input and output through the pipe.

#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdlib.h>

int main(int argc, char* argv[]) {

        int pipes[2];
        pipe(pipes);

        int inputPartOfPipe = pipes[0];
        int outputPartOfPipe = pipes[1];

        int pid = fork();

        if(pid > 0) {   //parent process
                dup2(inputPartOfPipe, 0); // redirect STDIN
                close(outputPartOfPipe);  // close unused half of pipe
                int value;
                scanf("%d\n", &value);
                printf("child sent value = %d\n", value);
        } else if(pid == 0) {   //child process
                dup2(outputPartOfPipe, 1); // redirect STDOUT
                close(inputPartOfPipe);     // close unused half of pipe
                printf("%d\n", 5000);
        } else {
                printf("fork failed!\n");
        }

        //don't worry about closing remaining pipes,
        //process exit does this for us

        return 0;
}

Key points:

• pipe(pipes) creates the read and write file descriptors.

• The parent redirects standard input to the read end with dup2().

• The child redirects standard output to the write end with dup2().

• Each process closes the pipe end it does not use.

• The parent reads a value that the child wrote through standard output.

Fork and Pipe Example

The systems-code-examples/fork_pipe example shows the same basic pattern as a standalone submodule example.

#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdlib.h>

int main(int argc, char* argv[])
{

    int pipes[2];
    pipe(pipes);

    int inputPartOfPipe = pipes[0];
    int outputPartOfPipe = pipes[1];

    int pid = fork();

    if(pid > 0)  	//parent process
    {
        dup2(inputPartOfPipe, 0); // redirect STDIN
        close(outputPartOfPipe);  // close unused half of pipe
        int value;
        scanf("%d\n", &value);
        printf("child sent value = %d\n", value);
    }
    else if(pid == 0)  	//child process
    {
        dup2(outputPartOfPipe, 1); // redirect STDOUT
        close(inputPartOfPipe);     // close unused half of pipe
        printf("%d\n", 5000);
    }
    else
    {
        printf("fork failed!\n");
    }

    //don't worry about closing remaining pipes,
    //process exit does this for us

    return 0;
}

Key points:

• The pipe is created before fork() so both processes inherit it.

• The parent uses the read end of the pipe.

• The child uses the write end of the pipe.

• dup2() lets ordinary scanf() and printf() operate through the pipe.

• The example depends on the parent and child closing the unused pipe halves.

Pipes and Context Switches

Pipe I/O crosses the user-kernel boundary. A write enters the kernel, the kernel buffers the data, and a read later copies the data into the receiving process.

This extra copying and scheduling work is the cost of getting a simple kernel-managed byte stream.

Named Pipes and FIFOs

A named pipe, or FIFO, is a pipe that has a name in the filesystem.

Named pipes differ from ordinary pipes in three useful ways. They live in the filesystem namespace, they can outlive the processes that use them, and normal filesystem permissions can control access to them.

Regular and Named Pipe Usage

Ordinary shell pipelines use anonymous pipes:

$ cat file | gzip -c9 > file.gz

Named pipes make the pipe visible as a filesystem entry:

$ mkfifo pipe_file
$ gzip -c9 < pipe_file > file.gz
$ cat file > pipe_file
$ rm -f pipe_file

This pattern can adapt programs that expect files to a streaming data source.

Named Pipe Uses

Named pipes are commonly used for local client-server designs and for adapting file-oriented programs to streaming input.

For example, a program that can only read from a local file can sometimes read from a FIFO while another process fetches, decrypts, decompresses, or generates the data. This works best for sequential I/O. It does not work well when the program needs random access.

Named Pipe Atomicity

Writes to a pipe are atomic only up to a system-defined size.

If each write() call is no larger than PIPE_BUF, the writes are atomic and are not interleaved with writes from other processes. If a write is larger than PIPE_BUF, the kernel may split the write and interleave it with other writers. A safe rule is to keep each logical message at or below PIPE_BUF or provide a separate synchronization mechanism.

Signals

A signal is a software interrupt delivered to a process.

Signals are used for asynchronous events such as child termination, terminal interrupts, timers, broken pipes, and invalid memory accesses. Each signal has a disposition: ignore it, catch it with a handler, or use the default action.

Important Signals

Many UNIX systems define more than thirty signals. The following table lists common ones used in systems programming.

List of Important Signals

Signal	Description
SIGABRT	abnormal termination, generated by abort(). Default action terminates and may dump core.
SIGALRM	generated when a timer set by alarm() expires.
SIGBUS	hardware fault or memory access fault.
SIGCHLD	sent to a parent process when a child process terminates.
SIGCONT	sent to a stopped process when it is continued.
SIGEMT	general hardware fault on systems that define it.
SIGFPE	arithmetic exception, such as divide by zero.
SIGHUP	terminal disconnected or controlling session ended.
SIGILL	illegal instruction.
SIGINT	terminal interrupt, usually Ctrl-C.
SIGKILL	cannot be caught or ignored; kills the process.
SIGPIPE	write to a pipe whose reader has closed.
SIGSEGV	segmentation violation.
SIGSTOP	cannot be caught or ignored; stops the process.

Handling Signals

A process can register a handler for a catchable signal.

static void handler(int);

int main(int argv, char* argv[]) {
  signal(SIGUSR1, handler);
  signal(SIGINT, handler);
  while(1) { pause(); }
}

static void handler(int signalNum) {
  if(signalNum == SIGUSR1) {
    printf("received SIGUSR1\n");
  } else if(signalNum == SIGINT) {
    printf("received SIGHUP\n");
  }
}

Key points:

• signal() registers the handler for SIGUSR1 and SIGINT.

• pause() blocks until a signal is caught.

• The handler receives the signal number and branches on it.

• A real handler should only call async-signal-safe functions.

Sending Signals

Signals can be sent from the shell with kill.

$ firefox &
[1] 5050
$ kill -USR1 5050
$ kill 5050

In C, kill(pid, signo) sends a signal to another process, raise(signo) sends a signal to the current process, alarm(seconds) sets a timer for SIGALRM, and pause() waits for a signal.

Interrupted System Calls

A signal can interrupt a blocking system call.

Older UNIX systems could interrupt almost any blocking system call. Modern systems still treat reads and writes on pipes, terminals, and network devices as slow calls because they can block indefinitely. If a signal handler runs while such a call is blocked, the call may return with errno set to EINTR. Code that catches signals must be ready to retry interrupted operations, unless sigaction() was configured with SA_RESTART and the system call is restartable.

Reentrant Signal Handlers

A signal handler interrupts whatever the process was doing at that moment. The interrupted code might have been inside malloc(), printf(), or another function with internal state.

For that reason, signal handlers must only call reentrant, async-signal safe functions. System calls and low-level wrappers such as read(), write(), close(), fork(), execve(), kill(), and waitpid() are generally safe. Library functions that allocate memory or use mutable static data are not safe in a signal handler.

Sleep with Alarm

Early UNIX implementations used SIGALRM to implement sleep-like behavior.

    void sig_alrm(int signo) 
    {
       /* ... */
    }

    unsigned int sleepFor(int numSeconds) {
       signal(SIGALRM, sig_alrm);    //set signal handler
       alarm(numSeconds);            //set alarm for n-seconds
       pause();                      //wait for signal
       return alarm(0);              //turn off alarm
       handler;
    }

Key points:

• The function installs a SIGALRM handler.

• alarm(numSeconds) schedules a signal in the future.

• pause() waits until a signal is delivered.

• alarm(0) cancels any remaining alarm and returns unslept time.

• The race between alarm() and pause() makes this style fragile.

Better Sleep with Longjmp

One attempted improvement is to use setjmp() and longjmp() so the signal handler can escape the interrupted wait.

jmpbuf env_alrm;

void sig_alrm(int signo) {
    longjmp(env_alrm, 1);
}

unsinged int sleepFor(int numSeconds) {
    signal(SIGALRM, sig_alrm);
    if(setjmp(env_alrm) == 0) {
        alarm(nsecs);
        pause();
    }
    return alarm(0);
}

Key points:

• setjmp() records a return point before the process blocks.

• The signal handler calls longjmp() to return to that point.

• This avoids some forms of blocking after the alarm fires.

• Jumping out of a signal handler is still delicate.

• sigsetjmp() and siglongjmp() are safer choices when signal masks matter.

Signal Timing Problem

Signal-based sleep code is difficult because one signal can interrupt another handler or arrive between two operations that looked adjacent in the source.

void sig_int(int signo) {
    volatile int j = 0;
    printf("sig_int enter\n");
    for(int i = 0; i < 10000000; i++) {
        j += i * i;
    }
    printf("sig_int done\n");
}

int main(int argc, char* argv[]) {
    signal(SIGINT, sig_int);
    unsigned int unslept = sleepFor(1);
    printf("sleepFor returned: %u\n", unslept);
    return 0;
}

Key points:

• SIGINT runs a handler while the program is also using sleepFor().

• The handler performs a long computation to make timing effects easier to observe.

• The example shows why signal handlers should do as little work as possible.

• Complicated signal interactions are a common source of subtle bugs.

Using Alarm for Time Limits

alarm() can also place an upper time limit on a blocking operation.

For example, a process reading from a named pipe can set an alarm before the read. If no writer appears, the signal interrupts the read and gives the program a chance to recover. This pattern is useful, but it requires careful handling of EINTR and careful signal-handler design.

Memory Mapped Files

A memory mapped file maps a file into a process’s virtual address space. After setup, the program can read and write the mapped region with normal memory operations.

The file is the backing store for the mapped memory. If two processes map the same file with shared mappings, they can communicate by reading and writing the mapped bytes. Their virtual addresses may be different, so shared data structures must not depend on process-local pointer values.

Virtual Addressing in Mapped Files

Memory mapped regions should use position-independent data layouts.

Pointers store virtual addresses, and the same mapped file can appear at different virtual addresses in different processes. Data stored in a mapped region should use fixed-size arrays, offsets, or indexes instead of raw pointers to process-local memory.

Absolute Memory Addressing

This example stores pointers inside a structure.

typedef struct {
    char[50] Brand;
    char[2] TractionRating;
    char[2] SpeedRating;
} Wheel;

typedef struct {
    Wheel*[4] Wheels;
    char* Make;
    char* Model;
} Car;

Car* car = (Car*)malloc(sizeof(Car));
car.Make = (char*)malloc(sizeof(char)*50);
car.Model = (char*)malloc(sizeof(char)*50);
car.Wheels[0] = (Wheel*)malloc(sizeof(Wheel));
car.Wheels[1] = (Wheel*)malloc(sizeof(Wheel));
car.Wheels[2] = (Wheel*)malloc(sizeof(Wheel));
car.Wheels[3] = (Wheel*)malloc(sizeof(Wheel));

car.Wheels[1]->Brand

Key points:

• Car stores pointers to Wheel objects.

• The wheel objects are allocated separately from the Car object.

• The pointer values are meaningful only in the process that created them.

• This layout is not suitable for sharing through a mapped file.

Relative Memory Addressing

This example stores the related objects inside one contiguous structure.

typedef struct {
    char[50] Brand;
    char[2] TractionRating;
    char[2] SpeedRating;
} Wheel;

typedef struct {
    Wheel[4] Wheels;
    char[50] Make;
    char[50] Model;
} Car;

Car* car = (Car*)malloc(sizeof(Car));

Key points:

• Car contains the wheel array directly.

• Make and Model are fixed-size arrays inside the same object.

• The data can be interpreted relative to the start of the mapped region.

• This layout is much more suitable for memory mapped IPC.

Using mmap

The basic mmap() pattern is to create a file, size it, map it, and then treat the mapped region as memory.

The producer process creates and writes the mapped file.

int main(int argc, char* argv[]) {
    const char *data = "Hello World";
    const int dummyValue = 0;
    int dataSize = sizeof(char) * strlen(data) + sizeof(char);

    int fd = open("shared.dat", O_CREAT|O_TRUNC|O_RDWR, 0666);
    lseek(fd, dataSize, SEEK_SET);
    write(fd, (char*)&dummyValue, sizeof(char));
    
    void* map = mmap(NULL,dataSize,PROT_READ|PROT_WRITE,MAP_SHARED,fd,0);

    memcpy(map, data, dataSize);

    getchar();

    munmap(map, dataSize);
    close(fd);

    return 0;
}

Key points:

• The file is opened with O_CREAT, O_TRUNC, and O_RDWR.

• lseek() and write() extend the file to the desired size.

• mmap() maps the file with read and write permissions.

• memcpy() writes data into the mapped region.

• munmap() and close() clean up the mapping and descriptor.

The consumer process maps the same file and copies the data out.

int main(argc, char* argv[]) {
    struct stat fileStat;
    stat("shared.dat", &fileStat);
    int fileSize = fileStat.st_size;

    int fd = open("shared.dat", O_RDWR);

    void* map = mmap(NULL, fileSize, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);

    char* data = (char*)calloc(1, fileSize);

    memcpy(data, map, fileSize);

    printf("%s\n", data);

    free(data);
    munmap(map);
    close(fd);

    return 0;
}

Key points:

• stat() obtains the file size.

• The file is opened with read-write access.

• mmap() maps the existing file into memory.

• memcpy() copies the mapped bytes into process-local memory.

• The process unmaps and closes the file after reading it.

Memory Mapped Example

The systems-code-examples/mmap example provides complete producer and consumer programs for a shared mapped file.

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
#include <strings.h>

int main(int argc, char* argv[])
{

    const char *data = "Hello World";
    int dataSize = sizeof(char) * strlen(data) + sizeof(char);

    const char *sharedFileName = "shared.dat";
    const mode_t mode = 0666;
    const int openFlags = (O_CREAT | O_TRUNC | O_RDWR);
    const int dummyValue = 0;
    int fd = open(sharedFileName, openFlags, mode);

    if(fd == (-1))
    {
        printf("open returned (-1)\n");
        return (-1);
    }

    if(lseek(fd, dataSize, SEEK_SET) == (-1))
    {
        printf("error in lseek\n");
        close(fd);
        return (-1);
    }
    if(write(fd, (char*)&dummyValue, sizeof(char)) == (-1))
    {
        printf("error in write\n");
        close(fd);
        return (-1);
    }

    int protection = (PROT_READ | PROT_WRITE);
    int mapFlags   = MAP_SHARED;
    void* map = mmap(NULL, dataSize, protection, mapFlags, fd, 0);

    if(map == (void*)(-1))
    {
        printf("mmap returned -1\n");
        close(fd);
        return (-1);
    }

    memcpy(map, data, dataSize);

    printf("memory mapped. press any key to exit...\n");
    getchar();

    if(munmap(map, dataSize) == (-1))
    {
        printf("munmap returned -1\n");
        close(fd);
        return (-1);
    }

    close(fd);
}

Key points:

• The producer creates shared.dat and sizes it for the string.

• MAP_SHARED makes updates visible through the backing file.

• memcpy() writes Hello World into the mapped region.

• The program waits for input before unmapping so another process can inspect the file.

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
#include <strings.h>

int getFileSize(const char *fileName)
{
    struct stat fileStat;
    stat(fileName, &fileStat);
    return fileStat.st_size;
}

int main(int argc, char* argv[])
{

    const mode_t mode = 0666;
    const int openFlag = (O_RDWR);
    const char *fileName = "shared.dat";
    int fileSize = getFileSize(fileName);

    int fd = open(fileName, openFlag, mode);
    if(fd == (-1))
    {
        printf("error in open\n");
        return (-1);
    }

    int protection = (PROT_READ | PROT_WRITE);
    int mapFlags = MAP_SHARED;
    void* map = mmap(NULL, fileSize, protection, mapFlags, fd, 0);

    if(map == (void*)(-1))
    {
        printf("mmap() returned -1\n");
        close(fd);
    }

    char *data = (char*)calloc(1, fileSize);

    memcpy(data, map, fileSize);

    printf("%s\n", data);

    free(data);

    if(munmap(map, fileSize) == (-1))
    {
        printf("munmap() failed\n");
        close(fd);
        return (-1);
    }

    close(fd);

    return 0;
}

Key points:

• The consumer opens the existing shared.dat file.

• stat() determines how much memory to map.

• The mapped bytes are copied into a local buffer.

• The consumer prints the string produced by the other process.

Memory Mapped Atomicity

Memory mapped I/O does not make reads and writes atomic.

The system calls mmap() and munmap() set up and tear down the mapping. Ordinary reads and writes to mapped memory do not enter the kernel. That is why mapped memory is fast, but it also means the program must provide its own synchronization when multiple processes write the same region.

Shared Memory Locking

Locks used inside shared memory must also be safe to share between processes.

POSIX semaphores can be placed in shared memory when initialized with the process-shared argument enabled. Data structures in mapped memory must also avoid raw process-local pointers, because each process may map the file at a different address.

Memory Mapped Queue Example

The local bounded-buffer snippets show the structure of a queue stored in mapped memory.

#include <semaphore.h>

const int MessageQueueSize = (5);
const int MaxMessageSize = (20);

class Message {
public:
  ~Message();
  void EnqueueMessage(const char *msg);
  char* DequeueMessage();
  static Message *CopyToMemoryMappedFile(int fd);
  static Message *GetFromMemoryMappedFile(int fd);
static void ReleaseFile(Message *msg, int fd);
private:
  Message();
  sem_t _lock;
  sem_t _empty;
  sem_t _full;
  int _current;
  char _messages[MessageQueueSize][MaxMessageSize];
};

Key points:

• Message stores semaphores directly in the shared object.

• _lock protects the queue state.

• _empty counts available messages.

• _full counts available queue slots.

• The messages are stored as fixed-size arrays rather than pointers.

Message::Message() {
  sem_init(&_lock,  1, 1);                                //passing 1 as the 2nd parameter allows the
  sem_init(&_empty, 1, 0);                              //semaphore work in a memory mapped region
  sem_init(&_full,  1, MessageQueueSize);
  _current = 0;
}
Message::~Message() { }
Message *Message::CopyToMemoryMappedFile(int fd) {
  int datasize = sizeof(Message);
  printf("message size = %d\n", datasize);
  if(lseek(fd, sizeof(Message), SEEK_SET) == (-1)) {
     fprintf(stderr, "error in lseek\n");
  }
  int dummyVal = 0;
  if(write(fd, (char*)&dummyVal, sizeof(char)) == (-1)) {
     fprintf(stderr, "error in write\n");
  }
  void *map = mmap(NULL, sizeof(Message), (PROT_READ|PROT_WRITE), MAP_SHARED, fd, 0);
  if(map == (void*)(-1)) {
     fprintf(stderr, "mmap() returned -1\n");
  }
  Message *msg = new Message();
  memcpy(map, (void*)msg, sizeof(Message));
  delete msg;
  return (Message*)map;
}

Key points:

• The constructor initializes process-shared semaphores.

• CopyToMemoryMappedFile() sizes the file and maps it.

• A temporary Message object is copied into the mapped region.

• The returned pointer refers to the mapped shared object.

Message *Message::GetFromMemoryMappedFile(int fd) {
  void *map = mmap(
     NULL, sizeof(Message), 
     (PROT_READ|PROT_WRITE), MAP_SHARED, fd, 0);
  if(map == (void*)(-1)) {
     fprintf(stderr, "mmap() returned -1\n");
  }
  Message* msg = (Message*)map;
  return msg;
}

void Message::ReleaseFile(Message *msg, int fd) {
  if(munmap((void*)msg, sizeof(Message)) == (-1)) {
     fprintf(stderr, "munmap() failed\n");
  }
}

Key points:

• GetFromMemoryMappedFile() maps an existing shared file.

• The mapped memory is interpreted as a Message object.

• ReleaseFile() unmaps the shared object.

• The file descriptor remains the caller’s responsibility.

void Message::EnqueueMessage(const char *msg) {
  sem_wait(&_full);
  sem_wait(&_lock);
  _current += 1;
  bzero(&_messages[_current], MaxMessageSize*sizeof(char));
  memcpy(&_messages[_current], msg, strlen(msg)*sizeof(char));
  sem_post(&_lock);
  sem_post(&_empty);
}

char* Message::DequeueMessage() {
  char *msg = new char[MaxMessageSize];
  sem_wait(&_empty);
  sem_wait(&_lock);
  memcpy(msg, &_messages[_current], MaxMessageSize*sizeof(char));
  _current -= 1;
  sem_post(&_lock);
  sem_post(&_full);
  return msg;
}

Key points:

• EnqueueMessage() waits for a free slot and then locks the queue.

• DequeueMessage() waits for an available message and then locks the queue.

• The lock semaphore protects _current and the message array.

• The counting semaphores coordinate producer and consumer progress.

Memory Mapped Queue Programs

The local producer and consumer snippets show how separate programs use the mapped queue.

int main(int argc, char* argv[]) {
  const char *sharedFileName = "shared.dat";
  const mode_t mode = 0666;
  const int openFlags = (O_CREAT | O_TRUNC | O_RDWR);
  int fd = open(sharedFileName, openFlags, mode);

  if(fd == (-1)) {
     printf("open returned (-1)\n");
     return (-1);
  }

  Message* msg = Message::CopyToMemoryMappedFile(fd);

  for(int i = 0; i < 100; i++) {
     char message[10];
     sprintf(message, "%d\n", i);
     msg->EnqueueMessage(&message[0]);
     printf("enqueued %d\n", i);
  }
  printf("message queue written\n");
  getchar();
  Message::ReleaseFile(msg, fd);
  close(fd);
}

Key points:

• The producer creates and truncates shared.dat.

• CopyToMemoryMappedFile() initializes the queue in the mapped file.

• The producer enqueues one hundred messages.

• The process waits before releasing the mapping so a consumer can run.

int main(int argc, char* argv[]) {
  const char *sharedFileName = "shared.dat";
  const mode_t mode = 0666;
  const int openFlags = (O_RDWR);
  int fd = open(sharedFileName, openFlags, mode);

  if(fd == (-1)) {
     printf("open returned (-1)\n");
     return (-1);
  }

  Message* msg = Message::GetFromMemoryMappedFile(fd);

  int count = 0;

  while(1) {
     char *message = msg->DequeueMessage();
     printf("%d: %s", ++count, message);
     fflush(stdout);
  }

  Message::ReleaseFile(msg, fd);

  close(fd);
}

Key points:

• The consumer opens the existing mapped file.

• GetFromMemoryMappedFile() attaches to the shared queue.

• DequeueMessage() blocks until a message is available.

• The loop prints messages as they are removed.

Submodule Memory Mapped Lock Example

The systems-code-examples/mmap_locks example is the complete version of the mapped queue design.

#ifndef MESSAGE_HH
#define MESSAGE_HH

#include <semaphore.h>

const int MessageQueueSize = (5);
const int MaxMessageSize = (20);

class Message {
public:
    ~Message();
    void EnqueueMessage(const char *msg);
    char* DequeueMessage();
    static Message *CopyToMemoryMappedFile(int fd);
    static Message *GetFromMemoryMappedFile(int fd);
    static void ReleaseFile(Message *msg, int fd);
private:
    Message();
    sem_t _lock;
    sem_t _empty;
    sem_t _full;
    int _current;
    char _messages[MessageQueueSize][MaxMessageSize];
};

#endif

Key points:

• The shared object contains semaphores, queue state, and message storage.

• Fixed-size arrays keep the object self-contained inside the mapped file.

• Static helper functions create, attach to, and release the mapping.

#include "message.hh"
#include <semaphore.h>
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <sys/mman.h>

#define SEMA_TYPE (1)
#define SEMA_MUTEX (1)
#define SEMA_EVENT (0)

Message::Message()
{
    sem_init(&_lock,  SEMA_TYPE, 1);
    sem_init(&_empty, SEMA_TYPE, 0);
    sem_init(&_full,  SEMA_TYPE, MessageQueueSize);
    _current = 0;
}

Message::~Message()
{
}

Message *Message::CopyToMemoryMappedFile(int fd)
{
    int datasize = sizeof(Message);
    printf("message size = %d\n", datasize);
    if(lseek(fd, sizeof(Message), SEEK_SET) == (-1))
    {
        fprintf(stderr, "error in lseek\n");
    }
    int dummyVal = 0;
    if(write(fd, (char*)&dummyVal, sizeof(char)) == (-1))
    {
        fprintf(stderr, "error in write\n");
    }
    void *map = mmap(NULL, sizeof(Message), (PROT_READ|PROT_WRITE), MAP_SHARED, fd, 0);
    if(map == (void*)(-1))
    {
        fprintf(stderr, "mmap() returned -1\n");
    }
    Message *msg = new Message();
    memcpy(map, (void*)msg, sizeof(Message));
    delete msg;
    return (Message*)map;
}

Message *Message::GetFromMemoryMappedFile(int fd)
{
    void *map = mmap(NULL, sizeof(Message), (PROT_READ|PROT_WRITE), MAP_SHARED, fd, 0);
    if(map == (void*)(-1))
    {
        fprintf(stderr, "mmap() returned -1\n");
    }
    Message* msg = (Message*)map;
    return msg;
}

void Message::ReleaseFile(Message *msg, int fd)
{
    if(munmap((void*)msg, sizeof(Message)) == (-1))
    {
        fprintf(stderr, "munmap() failed\n");
    }
}

void Message::EnqueueMessage(const char *msg)
{
    sem_wait(&_full);
    sem_wait(&_lock);
    _current += 1;
    bzero(&_messages[_current], MaxMessageSize*sizeof(char));
    memcpy(&_messages[_current], msg, strlen(msg)*sizeof(char));
    sem_post(&_lock);
    sem_post(&_empty);
}

char* Message::DequeueMessage()
{
    char *msg = new char[MaxMessageSize];
    sem_wait(&_empty);
    sem_wait(&_lock);
    memcpy(msg, &_messages[_current], MaxMessageSize*sizeof(char));
    _current -= 1;
    sem_post(&_lock);
    sem_post(&_full);
    return msg;
}

Key points:

• sem_init() uses process-shared semaphores.

• CopyToMemoryMappedFile() creates and initializes the shared object.

• GetFromMemoryMappedFile() maps an existing object.

• EnqueueMessage() and DequeueMessage() use semaphores to coordinate access across processes.

• The queue data itself lives in the mapped file.

#include "message.hh"
#include <stdio.h>
#include <sys/types.h>
#include <fcntl.h>
#include <unistd.h>

int main(int argc, char* argv[])
{

    const char *sharedFileName = "shared.dat";
    const mode_t mode = 0666;
    const int openFlags = (O_CREAT | O_TRUNC | O_RDWR);
    int fd = open(sharedFileName, openFlags, mode);

    if(fd == (-1))
    {
        printf("open returned (-1)\n");
        return (-1);
    }

    Message* msg = Message::CopyToMemoryMappedFile(fd);

    for(int i = 0; i < 100; i++)
    {
        char message[10];
        sprintf(message, "%d\n", i);
        msg->EnqueueMessage(&message[0]);
        printf("enqueued %d\n", i);
    }

    printf("message queue written\n");
    getchar();

    Message::ReleaseFile(msg, fd);

    close(fd);
}

Key points:

• The producer creates the shared backing file.

• It initializes the mapped Message queue.

• It enqueues a sequence of numbered messages.

• The program waits before releasing the file so the consumer can read.

#include "message.hh"
#include <stdio.h>
#include <sys/types.h>
#include <fcntl.h>
#include <unistd.h>

int main(int argc, char* argv[])
{

    const char *sharedFileName = "shared.dat";
    const mode_t mode = 0666;
    const int openFlags = (O_RDWR);
    int fd = open(sharedFileName, openFlags, mode);

    if(fd == (-1))
    {
        printf("open returned (-1)\n");
        return (-1);
    }

    Message* msg = Message::GetFromMemoryMappedFile(fd);

    int count = 0;

    while(1)
    {
        char *message = msg->DequeueMessage();
        printf("%d: %s", ++count, message);
        fflush(stdout);
    }

    Message::ReleaseFile(msg, fd);

    close(fd);
}

Key points:

• The consumer opens the existing shared file.

• It maps the Message queue created by the producer.

• It repeatedly dequeues and prints messages.

• Synchronization occurs through semaphores stored in the mapped region.

Memory Mapped I/O Performance

Memory mapped I/O can be faster than ordinary file I/O because it avoids an explicit copy between a user buffer and a kernel buffer.

With write(), the process enters the kernel and the kernel copies data from the user buffer into filesystem buffers. With mmap(), the process writes into mapped memory and the virtual memory system writes dirty pages back to the file later. This works especially well when the file format already matches the program’s in-memory data model.

Files as IPC

Files are the oldest and most portable form of IPC.

File-based IPC appears in several patterns: persistent state, exposing current state, spool directories, lock files, and simple queues. Files are slower and less structured than specialized IPC mechanisms, but they are durable and easy to inspect.

Exposing Current State

The /proc filesystem exposes kernel and process state as files.

This gives tools and developers a file-oriented interface to information that would otherwise require custom system calls.

List of /proc-style Filesystems

Filesystem	Description
/proc/vmstat	virtual memory statistics and configuration
/proc/cpuinfo	CPU information
/proc/<PID>	individual process information
/proc/loadavg	moving average of ready process load

Spool Folders

A spool folder is a directory used as a persistent work queue.

Mail daemons, print systems, and scheduled job systems often use spool folders. A daemon watches the directory, processes new files, and then moves or deletes completed work. If the daemon crashes, the queued work remains on disk.

Cron Spool Folders

Cron uses spool-like directories for scheduled tasks.

Common examples include /etc/cron.daily, /etc/cron.hourly, /etc/cron.monthly, and /etc/cron.weekly. At the scheduled time, cron runs executable files in the appropriate directory.

CUPS Spool Folders

CUPS uses spool files to represent print jobs.

The spool directory is typically /var/spool/cups. Data files contain the bytes to print, while control files describe the job. The naming convention gives the print system a simple domain model for queue processing.

Lock Files

A lock file uses atomic filesystem operations to represent ownership of a resource.

For example, a server might create /var/lock/http_80 before binding to port 80. If another process tries to create the same lock file at the same time, only one creation succeeds. The lock file should be removed when the owning process exits normally.

Doors

Doors are a Solaris IPC mechanism that exposes procedure calls through filesystem entries.

They are not portable, but they are useful historically because they show one way to build RPC-like behavior into a filesystem namespace.

Door Server

The server creates a door and attaches it to a filesystem path.

void server(void* cookie, char* argp, size_t arg_size, door_desc_t* dp, uint_t n_desc) {

    /*server code goes here*/

}

int doorfd = door_create(server, 0, 0);
int fd = creat("/tmp/door", 0666);
fdetach("/tmp/door");
fattach(doorfd,"/tmp/door");
pause();

Key points:

• door_create() registers the server function.

• creat() creates the filesystem entry.

• fdetach() removes any previous attachment at the path.

• fattach() attaches the door to the path.

• pause() keeps the server process alive.

Door Client

The client opens the door path and calls through it.

typdef struct myArg {
    int id
} myArg_t;

door_arg_t d_arg;
int doorfd = open("/tmp/door", O_RDONLY);
myArg_t* arg = (myArg_t*)malloc(sizeof(myArg_t));
arg->id = 12;
d_arg.data_ptr = (char*)arg;
d_arg.data_size = sizeof(*arg);
d_arg.desc_ptr = NULL;
d_arg.desc_num = 0;
door_call(doorfd, &d_arg);

Key points:

• The client opens the door path like a file.

• door_arg_t packages the argument data.

• door_call() invokes the server through the door.

• The filesystem path names the IPC endpoint.

Domain Sockets

A UNIX domain socket is a local socket addressed through the filesystem namespace.

Domain sockets are similar to named pipes, but they can be full-duplex, support multiple clients, and support stream or datagram communication. They use familiar socket calls such as socket(), bind(), listen(), accept(), and connect().

Domain Socket Example

The local domain socket snippets show the server, handler, and client pieces separately.

int server_listen(const char *fileName) {
  unlink(fileName);
  int socket_fd = socket(PF_UNIX, SOCK_STREAM, 0);
  struct sockaddr_un address;
  memset(&address, 0, sizeof(struct sockaddr_un));
  address.sun_family = AF_UNIX;
  sprintf(address.sun_path, fileName);

  bind(socket_fd, (struct sockaddr*)&address, sizeof(struct sockaddr_un));
  listen(socket_fd, 5);
  int connection_fd;
  socklen_t address_length;
  while((connection_fd = accept(socket_fd, (struct sockaddr*)&address, &address_length)) > (-1)) {
     int child = fork();
     if(child == 0) {
        return connection_handler(connection_fd);
     } else {
        close(connection_fd);
     } 
  }

  close(socket_fd);
  unlink(fileName);
  return 0;
}

Key points:

• socket(PF_UNIX, SOCK_STREAM, 0) creates a local stream socket.

• unlink() removes any stale socket file.

• bind() attaches the socket to a filesystem path.

• listen() marks the socket as a server endpoint.

• accept() receives client connections.

int connection_handler(int socket_fd) {
  char buff[256];
  int nBytes = read(socket_fd, buff, 256);
  buff[nBytes] = 0;
  printf("message from client: %s\n", buff);
  nBytes = snprintf(buff, 256, "hello from server");
  write(socket_fd, buff, nBytes);

  close(socket_fd);
  return 0;
}

Key points:

• The handler reads a request from the connected socket.

• It prints the client message.

• It writes a response back to the client.

• It closes the connected socket when finished.

int client_connect(const char* fileName) {
  int socket_fd = socket(PF_UNIX, SOCK_STREAM, 0);

  struct sockaddr_un address;
  memset(&address, 0, sizeof(struct sockaddr_un));
  address.sun_family = AF_UNIX;
  sprintf(address.sun_path, fileName);

  connect(socket_fd, (struct sockaddr*)&address, sizeof(struct sockaddr_un));
  char buffer[256];
  int nBytes = snprintf(buffer, 256, "hello from a client");
  write(socket_fd, buffer, nBytes);

  nBytes = read(socket_fd, buffer, 256);
  buffer[nBytes] = 0;

  printf("message from server: %s\n", buffer);

  close(socket_fd);

  return 0;
}

Key points:

• The client creates a UNIX stream socket.

• connect() attaches the client to the server’s socket path.

• The client writes a message and then reads a response.

• The same connected socket supports both directions of communication.

Submodule Domain Socket Example

The systems-code-examples/domain_sock directory contains a complete domain socket client and server.

#include <stdio.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <sys/types.h>
#include <unistd.h>
#include <string.h>

int connection_handler(int socket_fd)
{
    char buff[256];
    int nBytes = read(socket_fd, buff, 256);
    buff[nBytes] = 0;
    printf("message from client: %s\n", buff);
    nBytes = snprintf(buff, 256, "hello from server");
    write(socket_fd, buff, nBytes);

    close(socket_fd);
    return 0;
}

int server_listen(const char *fileName)
{
    int socket_fd = socket(PF_UNIX, SOCK_STREAM, 0);
    if(socket_fd == (-1))
    {
        printf("socket() failed\n");
        return (-1);
    }

    unlink(fileName);

    struct sockaddr_un address;
    memset(&address, 0, sizeof(struct sockaddr_un));
    address.sun_family = AF_UNIX;
    sprintf(address.sun_path, fileName);

    if(bind(socket_fd, (struct sockaddr*)&address, sizeof(struct sockaddr_un)) != 0)
    {
        printf("bind() failed\n");
        return (-1);
    }
    if(listen(socket_fd, 5) != 0)
    {
        printf("listen() failed\n");
        return (-1);
    }

    int connection_fd;
    socklen_t address_length;
    while((connection_fd = accept(socket_fd, (struct sockaddr*)&address, &address_length)) > (-1))
    {
        int child = fork();
        if(child == 0)
        {
            return connection_handler(connection_fd);
        }
        else
        {
            close(connection_fd);
        }
    }

    close(socket_fd);
    unlink(fileName);
    return 0;
}

int main(int argc, char* argv[])
{
    if(argc != 2)
    {
        printf("usage:");
        printf("server [socket file]\n");
        return (-1);
    }
    return server_listen(argv[1]);
}

Key points:

• The server binds a PF_UNIX socket to the path provided on the command line.

• Each accepted connection is handled in a child process.

• The child reads the client message and writes a response.

• The parent closes the connected socket and continues accepting clients.

• The socket file is unlinked when the server exits normally.

#include <stdio.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
#include <string.h>

int client_connect(const char* fileName)
{
    int socket_fd = socket(PF_UNIX, SOCK_STREAM, 0);
    if(socket_fd < 0)
    {
        printf("socket() failed\n");
        return (-1);
    }

    struct sockaddr_un address;
    memset(&address, 0, sizeof(struct sockaddr_un));
    address.sun_family = AF_UNIX;
    sprintf(address.sun_path, fileName);

    if(connect(socket_fd, (struct sockaddr*)&address, sizeof(struct sockaddr_un)) != 0)
    {
        printf("connect() failed\n");
        return (-1);
    }

    char buffer[256];
    int nBytes = snprintf(buffer, 256, "hello from a client");
    write(socket_fd, buffer, nBytes);

    nBytes = read(socket_fd, buffer, 256);
    buffer[nBytes] = 0;

    printf("message from server: %s\n", buffer);

    close(socket_fd);

    return 0;
}

int main(int argc, char* argv[])
{
    if(argc != 2)
    {
        printf("usage:\n");
        printf("client [socket file]\n");
        return (-1);
    }
    return client_connect(argv[1]);
}

Key points:

• The client receives the socket path on the command line.

• It connects to the server with connect().

• It sends a short request string.

• It reads and prints the server response.

• The client closes the socket before exiting.

TCP/IP

TCP/IP sockets are the networked counterpart to local socket IPC.

They use many of the same programming ideas as domain sockets, but the address is a network address and port rather than a local filesystem path. TCP/IP sockets are usually the right choice when communication must cross machine boundaries.