The Linux Programming interface

Basic Terms and Definitions

  • SUS: Single UNIX Spec and the minimal requirement for UNIX branding - UNIX is a trademark

  • MULTICS: Predecessor of UNIX which inspired some UNIX concepts

    • Tree Structured FileSystem

    • A separate program for interpretting commands (the Shell)

    • Files as unstructured stream of bytes

  • POSIX: Promotes Source Code Compatibility across UNIX systems. It defines API for a set of services available to a program, irrespective of implementation. Doesn’t have to be SUS compliant.

  • LSB: Linux Standard Base, Promotes Binary Compatibility across Linux Distros

Important

Linux in general is not fully compliant with SUS and POSIX

File Types:

  • regular files: plain files, text or binary

  • Directory: a table of filenames coupled with references (links) to the corresponding files. directories may contain links to files and other directories

  • link: A filename plus its reference is called a link. a single file can have multiple links (and thus multiple names) in one or more directories

  • Sym link: A file that references another file (the target), and thus provides an alternate name for that other file

  • devices

  • pipes

  • sockets

Process Perspective and Kernel Perspective

Process Perspective

  1. For a process, all system events are happening asyncronously, including Signals and IPC events, Networking events, etc.

  2. A process doesn’t know when it will next time out, and later when it will next be given back the CPU (pre-emptive scheduling)

  3. A process doesn’t know which other processes are running, and which are blocked, and which are waiting

  4. A process doesn’t know where it is physically located in RAM, and if parts of its runtime memory is currently being swapped

  5. A process doesn’t know physical location of files - it can only refer to files by name

  6. A Process is isolated from other processes, and cannot directly communicate with other processes beyond a kernel managed IPC protocol

  7. A Process cannot create a new process by itself, and cannot terminate itself by itself

  8. A process cannot directly interface with IO Devices attached to the system

The Kernel Perspective

  1. A system has one running Kernel that knows and controls everything

  2. Facilitates the operation of all processes on the system

  3. The Kernel decides who gets CPU time, when it will do so, and for how long

  4. The kernel maintains data structures containing information about all running processes and updates these structures as processes are created, change state, and terminate.

  5. The kernel maintains all of the low-level data structures that enable the filenames used by programs to be translated into physical locations on the disk

  6. The kernel also maintains data structures that map the virtual memory of each process into the physical memory of the computer and the swap area(s) on disk

  7. Communication among processes take place via IPC mechanisms managed by the kernel

  8. In response to requests from processes, the kernel creates new processes and terminates existing processes

  9. kernel device drivers performs all direct communication with input and output devices, transferring information to and from user processes as required

Linux Process

A program exists in one of three forms:

  • Source: generally architecture independant Text ready to be compiled to an Arch specific binary executable

  • Binaries: Arch specific executable

  • Script: text to be interpreted and run by an executable program, like Bash or Python

A Process:

An instance of a running program, where a single program can have multiple processes running simultanouly each isolated and with their own RAM space

Process Memory:

  • Text: the binary instructions of the program, and are read only

  • Data: Initialized and non-initialized Static variables and globals

  • Heap: Dynamically allocated virtual memory

  • Stack: function local variables, function parameters and function return adresses

Process Creation:

A process (the parent) can start a new process (the child) by calling the fork() system call. The kernel copies the Data, Heap and Stack of the parent to the child. The child process can either execute functions defined in the parents Text, or to execute a new program with the execve() system call. A call to exceve() destroys the Text, Data, Heap and Stack of the child and replaces them with new segments based on the new program. However, the PID, and other aspects of the child remains the same. Therefore, execve does not create a new process, and hence the need to call fork()

Memory Mapping:

mmap() system call can be used to create a new memory mapping in the calling process virtual memory space.

  1. File Mapping: map an existing file to the process virtual memory, and file contents can be accessed as bytes and loaded from the file as required

  2. Anonymous Mapping: Doesn’t have a corresponding file, and memory pages are initialized to byte-data 0

  • Mapping can either be private, where modifications made by one process are not visible to other processes and are not carried through to an underlying file

  • shared File mapping, where changes are visible across two or more processes, and are carried through to an underlying file

Processes with shared mapping are either a parent and its children, or peer processes sharing the same file mapping

Typical Usages of mmap()

  • initializing of a process text segment when first loaded from binary executable

  • allocation of new heap memory during the course of the program

  • File I/O

  • IPC via shared mapping

Process Groups, Jobs, and Sessions

ls -lh | sort -k5n | less

This pipeline launches 3 processes in parallel, with FIFO Pipes between them, which are placed in a process group or job, sharing the same Process Group ID The kernel allows the delivery of a signal to all processes in a job.

A Session is a collection of jobs, all sharing the same Session ID Within a Terminal Session, only one job can be in the foreground, which reads input from the terminal and writes output to it. Within a Terminal Session, zero or more jobs can be running on the background, which are created by terminating a pipeline with &

Interproces Communication (IPC) and Syncronization

IPC Mechanisms include:

  1. Sharing Information Via Disk Files, which is generally slow and inflexible

  2. Signals: which are used to indicate when events have occured

  3. Pipes and FIFOs: Can be used to effiently transfer data concurrently between running processes

  4. Sockets: Similar to pipes, but can enable IPC across processes in different hosts

  5. File Locking: allows a process to lock regions of a file to block other processes from reading or updating contents

  6. Message Queues: used to exchange messages, or atomic packets of data accross processes

  7. Semaphores: used to sync Actions between processes

  8. Shared Memory

System Calls APIs

A system call allows a process to request that the kernal perform some action on behalf of the process.

man syscalls # list of all system calls and notes
man syscall # library function to indirectly invoke syscall based on assembly lang interface *number*

Steps on invoking system calls

Important

System calls are a relatively heavy operation, since the kernel needs to validate the call and the arguments passed.

  1. User program invokes a clib wrapper around a given syscall, such as getppid()

  2. User program passes arguments normally on the stack to the wrapper function

  3. The Wrapper places the arguments in specific CPU registers

  4. The Wrapper identifies the unique ID of the syscall, and places that ID in the %eax CPU Register

  5. The Wrapper executes the legacy trap (kinda like an exception) machine instruction 0x80, or the more modern instruction SYSENTER with OpCode 0x0F34 to enter Kernel mode

  6. The Kernel invokes a handling routine (system_call()) that:

    • save the register values (arguments) to the kernel stack

    • Check the validity of the system call number, and look it up in sys_call_table

    • If the syscall has arguments, first check their validity (for example, valid pointers to user virtual memory)

    • Invoke the appropriate System call routine, which may involve modifying values at addresses, and/or transferring data across kernel memory and user memory

    • Return a result status to the system_call() routine

    • Restore register values from the kernel stack and places the system call resturn value on the stack

    • Returns to the wrapper function simultenously switching the CPU back to User Mode

  7. The Wrapper function returns to the caller, returning an integer status code (negative for failed, and positive for success)