![\"Three](\"/assets/struct-file.png\")
Can you imagine a computer that takes no input and produces no output?\nPretty useless, right? So I/O is one of the most important and fundamental\npieces of a computer.
\nAbove the interesting machinery that makes your CPU able to communicate with\ndevices, there's an API for you, the programmer, to perform the desired operations,\nand that's what we're going to focus on today. Specifically: what a file actually\nis, how directories fit in, and the handful of system calls you use to work\nwith them, like open, read, write, lseek and fsync.
\nI took inspiration for this text mainly from OSTEP (Operating Systems:\nThree Easy Pieces), one of my favorite books on this subject.
\nYou must have already heard that everything in Linux is a file, that used to\nconfuse me a little, because what is everything? everything seems like a lot\nof things. What everything refers to, though, comes from the UNIX definition\nof what's a file: a linear array of bytes, which you can read or write.\nThinking of it, what is I/O besides writting and reading bytes from somewhere?
\nSo everything means: the keyboard you're typing on, the terminal printing\nthese bytes back at you, a disk partition, a network socket, even the\nkernel's own state in /proc. All of them are things you open, read bytes\nfrom, write bytes to, and close.
\nWith this, you get a common interface across any kind of device. You just\nneed a driver for it and the superpower of abstractions does the job: you\ndon't need a HARD_DRIVE_SPECIFIC_BRAND_open() call.
So a file is a linear array of bytes. But that's what a file is, not where\nit is. If you have a couple hundred thousand of these arrays sitting on a\ndisk, and you ask for one of them by typing main.c, how does the kernel\nknow which one you mean?
The answer is that every file has a second name, one you never type: a\nnumber, called the inode number, unique within its filesystem. That number\nis what the kernel actually uses. You can see it:
\n$ ls -i main.c\n1310742 main.cThe number identifies a small record on disk, the inode, holding everything\nthe kernel knows about the file: its size, its owner, its permissions, its\ntimestamps, and where its data blocks actually live.
\nNotice what's missing from that list. The inode doesn't know the file's\nname. You have a name, the kernel has a number, and nothing so far connects\nthe two. That's what a directory is for.
\nBelieve it or not, a directory is a file too.
\nIt's a linear array of bytes, with an inode and the same everything we\njust talked about. What makes it a directory is a type field in its\ninode saying so, which tells the kernel how to interpret those bytes.\nAnd the interpretation is simple, a dir is basically a list:
\n. 131074\n.. 2\nmain.c 1310742\nnotes.md 1310744A user-readable and the file unique number. This way you can see that your\nfile's name doesn't live in the file, it lives in the directory that points\nto it.
\nSo when you ask for /home/user/main.c, the kernel walks it:
start at the root\n look up \"home\" -> 131074\n look up \"user\" -> 262401\n look up \"main.c\" -> 1310742If you're curious about where the walk starts, the root can't be looked up\nby name, since there's no directory above it to look in. So it just gets a\nfixed number, decided by the filesystem format.
\nThe first operation is creating a file. That's done with the open syscall,\nfrom #include <fcntl.h>, passing it the O_CREAT flag. The name of the\nheader is short for file control, and it's also where the O_* flags live.
Note that open both creates and opens. There's no separate call for it.\nWell, there was the creat syscall, but since open is already able to\ndo such thing, it became kinda obsolete.
Here's an example:
\nint fd = open(\"my_file\", O_CREAT | O_WRONLY | O_TRUNC, S_IRUSR | S_IWUSR);The first argument is the path, resolved the way we just saw. Every\ncomponent except the last has to exist already: open creates a file, not\na path.
The second is a single int with flags OR'd together, doing three different\njobs. One access mode, exactly one, either O_RDONLY, O_WRONLY or\nO_RDWR. Then flags for open time, like O_CREAT to make the file if it\nisn't there and O_TRUNC to empty it if it is. So the line above means\n"give me this file, empty and ready to write". The rest change how later\nreads and writes behave, and can wait.
The third is the permissions, and it only matters with O_CREAT, since an\nexisting file already has them. S_IRUSR | S_IWUSR is owner read and write,\nalso written 0600.
See what's returned from the create operation, an fd, short for file\ndescriptor. This deserves attention because that's how your process\nmaintains a list of opened files.
\nThe fd is an index into a list of open files. Just an int, and a signed one,\nwhich is what leaves room for -1 to mean failure. "I want to operate on\nthe fourth entry in my table of open files."
That table is per process. Here's xv6's version, which is about as small as\nit gets:
\nstruct proc {\n struct file *ofile[NOFILE]; // open files\n};Being per process is the important part: fd 3 here and fd 3 in the process\nnext to you are unrelated entries in different tables. And since the array\nis fixed size, there's a cap on how many files you can have open at once,\nwhich you can see with ulimit -n. In my machine it is 2560.
The first three entries are already taken before your code runs:
\n0 stdin\n1 stdout\n2 stderrThat's why your first open usually comes back as 3, since the kernel hands\nyou the lowest free index. And there's nothing special about these three,\nthey're ordinary entries in the same table:
write(1, \"hello\\n\", 6);That's roughly what printf ends up doing.
Now that you've acquired an fd, what can you do with it?
\nThe most fundamental operations, of course, are reading and writing. And\nto see both in action, there's no better example than cat, a program\nyou've almost certainly used, and one that fits in a few lines:
#include <fcntl.h>\n#include <stdio.h>\n#include <unistd.h>\n\nint main(int argc, char *argv[]) {\n int fd = open(argv[1], O_RDONLY);\n if (fd < 0) {\n perror(\"open\");\n return 1;\n }\n\n char buf[4096];\n ssize_t n;\n while ((n = read(fd, buf, sizeof(buf))) > 0) {\n write(1, buf, n);\n }\n\n close(fd);\n return 0;\n}The open part you already know. We ask for the file read only, and if\nwe get -1 back, we print what went wrong and quit.
The interesting part is the loop:
\nwhile ((n = read(fd, buf, sizeof(buf))) > 0)The file's bytes don't live in our process. They're on the disk, and once\nread, cached in the kernel's own memory, in what's called the page cache.\nread is the way to bring them over, the kernel copies them from the page\ncache into memory we own, with copy_to_user, essentially a memcpy that\nchecks the destination really belongs to our process. That's why we pass\nbuf, it's the destination for that copy.
Which means something interesting. If the block isn't in the page cache\nyet, the kernel first reads it from disk into the cache, then copies it to\nus. Either way, after a read those bytes exist twice in memory: once in\nthe kernel, once in our buffer. mmap avoids this by mapping the page\ncache straight into your address space, but that's an interface for\nanother day. I suggest this video by Tsoding if you want to know more:\nhttps://www.youtube.com/watch?v=sFYFuBzu9Ow
The kernel has no idea how big buf is. In C, an array passed to a\nfunction decays into a pointer, so all read gets is an address.\nsizeof(buf) is how we tell it the ceiling, so it never writes past the\nend of our array.
What comes back is the number of bytes it actually copied. We set the\nceiling at 4096, but a file with "hello" in it gives us 6: five letters\nand a newline (\\n). When there's nothing left, it gives us 0, which is\nhow cat knows it's done. It never checks the file size, it just keeps\nreading until the kernel says there's nothing more.
Then we hand those bytes to fd 1:
\nwrite(1, buf, n);This is the same copy in the other direction, from our buffer back into\nthe kernel. We pass n because only the first n bytes of buf hold\nwhat we just read. Fd 1, as stated before, is stdout, the entry that\nis already opened in our process by default. As far as cat is\nconcerned, it's writing bytes to an fd, and whether that ends up on a\nterminal is none of its business.
Notice that our buffer only holds 4096 bytes. What happens with a file\nbigger than that? Nothing special: the loop reads the first 4096 bytes,\nwrites them out, reads the next 4096, writes those, and so on. We never\nhold the whole file at once, only one chunk at a time, passing each one\nalong before fetching the next.
\nThe buffer size is a tradeoff. A bigger buffer costs more memory but\nmeans fewer calls to read and write. Remember that every syscall has\na price: crossing into the kernel and back. A smaller one saves memory and pays\nthat price more often. Reading a 1MB file takes 256 round trips with\n4096 bytes, and 16 with 64KB.
Finally, close gives the table entry back.
There's one thing left you may have noticed. If not, you can pretend you\ndid. We call read over and over with the exact same arguments, and each\ntime we get the next chunk of the file. So something is remembering where\nthe last read stopped. How does a function called with identical\narguments hold on to that state?
That's our next subject.
\nRemember the table from the file descriptors section:
\nstruct proc {\n struct file *ofile[NOFILE]; // open files\n};The answer is behind those pointers. Each entry points to a struct file, a small object the kernel creates every time you call open.\nHere's xv6's, trimmed:
struct file {\n int ref;\n char readable;\n char writable;\n struct inode *ip;\n uint off;\n};ip points to the inode, the file itself, and off how far into the\nfile we are, that answers our previous question. Each read starts copying at\noff and moves it forward by the number of bytes it copied. So when\ncat calls read again with the same arguments, it picks up right\nafter the last chunk.
xv6 keeps these in a fixed global array, since it has no allocator for\nsmall objects, while Linux allocates them dynamically. Either way, your\ntable only holds a pointer:
\n
Process A opened the file twice and got two of them, each with its own\noffset. Process B opened it once and got a third. All three lead to the\nsame inode.
\nSince the table only holds pointers, two entries can point to the same\nstruct file, dup does it within a process, fork across parent and\nchild. ref counts those entries, and close only frees the struct file when it reaches zero.
You can see the independent offsets in a few lines:
\nint a = open(\"main.c\", O_RDONLY);\nint b = open(\"main.c\", O_RDONLY);\n\nread(a, buf, 100); // reads bytes 0 to 99, a's offset is now 100\nread(b, buf, 100); // reads bytes 0 to 99 again, b starts at 0\nread(a, buf, 100); // reads bytes 100 to 199Reading through a doesn't touch b's offset at all.
Reading front to back is fine for cat. But say you need the bytes at\noffset 5000, and only those. For that there's lseek:
lseek(fd, 5000, SEEK_SET);\nread(fd, buf, 100);lseek does no I/O at all. It just sets the offset, and the next read\nstarts from there. The third argument says what the offset is relative\nto, as the man page puts it:
If whence is SEEK_SET, the offset is set to offset bytes.\nIf whence is SEEK_CUR, the offset is set to its current location\nplus offset bytes.\nIf whence is SEEK_END, the offset is set to the size of the file\nplus offset bytes.This gets tricky with threads. Say you open a file once and hand the\nsame fd to several threads. Since threads share the fd table, they all\nend up using the same struct file, and so the same off.
You could open a new fd per thread instead, and each would get its own\noffset. But every open is a new path lookup, which may land on a\ndifferent file if it was replaced in between. Sharing one fd means\neveryone reads the same file, at the price of a shared offset.
\nAnd that price shows up when two threads each do a seek followed by a\nread. They can interleave like this:
\nthread A: lseek(fd, 5000, SEEK_SET)\nthread B: lseek(fd, 9000, SEEK_SET)\nthread A: read(fd, buf, 100) // reads from 9000\nthread B: read(fd, buf, 100) // reads from 9100Thread A ends up with bytes from offset 9000, and no error tells it so.
\nI hit exactly this while building an append only storage engine, with\nseveral threads reading records at random offsets. The fix was pread:
pread(fd, buf, 100, 5000);It takes the offset as an argument and reads from there. Since it never\ntouches the shared offset, threads can call it on the same fd without\nstepping on each other.
\nRemember the page cache from the reading section? write copies your\nbytes into it, the same cache read copies from, and marks the page\ndirty, which means it has changes the disk doesn't have yet. Since every process\nshares that cache, anyone reading the file sees your changes right\naway, before they reach the disk.
The kernel does this on purpose, and it's why these writes are called\nbuffered. Disks are slow, and many small writes are much cheaper as a\nfew big ones. So it holds dirty pages for a while, letting writes pile\nup, and if you write to the same page twice, only the final version\never goes to disk. Meanwhile write returns as soon as the copy is\ndone.
Those dirty pages get flushed eventually, for one of two reasons. Either\nthey've been dirty for too long, around 30 seconds by default on Linux,\nor too much of memory is dirty and the kernel starts writing back to\nmake room.
\nMost of the time that's exactly what you want. But if the machine loses\npower before the flush, those bytes are gone, even though write told\nyou everything went fine.
For a database, that's unacceptable. When I was building my append only\nstorage engine, a record being acknowledged had to mean it was actually\non disk, and a successful write doesn't promise that. That's what\nfsync is for:
write(fd, buf, n);\nif (fsync(fd) < 0) {\n perror(\"fsync\");\n}It blocks until all of the file's dirty pages are on disk, along with\nthe inode's metadata, like its size.
\nThere's a catch if the file is new. Its name isn't stored in the file,\nit's an entry in the directory, and the directory is a file with its own\ndirty pages. fsync on your file leaves those alone. So after a crash,\nthe data could be on disk with no name pointing to it. To avoid that,\nopen the directory and fsync it too.
There's a lot I left out: hard and symbolic links, permissions, stat,\nmmap, and plenty more. Most of what I covered here comes from OSTEP's\nchapter on files and directories, and if you want to keep going, I\nreally recommend it. The rest of the book is just as good.
If you are not reading this on a carved stone, every character you see on this screen arrived through a system call. Every keystroke you typed triggered one. They run invisibly, underneath everything.
\nWhen learning some language, one of the first things you have contact with is how to print stuff on the terminal, perhaps a "Hello World". Let's take C as an example, to print this classic opener we can make use of a procedure called printf(), but underneath that one line, the CPU is switching privilege levels, your register state is being frozen, and the kernel is running on an entirely separate stack, all before the printf returns. By the end, I hope the kernel will feel a little less like a black box.
One of the most important abstractions the OS gives you is a process. When you create a binary, it is just a file stuffed with instructions (and maybe some static data) sitting on the disk. It is the operating system that gives you a running program. That running program is what we call a process.
\nWhen you run a program, a lot happens before your first line of code executes. The OS creates an entry in the process list, allocates memory for the program, loads program from disk to memory, set up the stack with argc and argv, clears the registers, and finally hands control to your entry point, main(). From there, the CPU fetches your instructions directly from memory and executes them one by one.
Pretty simple, right? But if your code runs directly on the CPU with no middleman, what stops it from accessing the disk, reading another process's memory, or turning on your camera? You need to ensure all this security while still maintaining efficiency. Sounds like a hell of a problem.
\nThankfully, the CPU itself has a built-in solution. It does not treat all code equally. It runs in two distinct modes, each with a different level of privilege. This mode is not an OS policy, it is hardwired into the processor itself.
\nThe CPU tracks the current mode with a two-bit value called the privilege level. Kernel mode means full access to hardware, memory, and every instruction the CPU can execute. User mode means restricted. If you tried to issue an I/O request while running in user mode, the processor would raise an exception and the OS would likely kill your process.
\nYour operating system runs in kernel mode. Every program you write runs in user mode.
\nBut every real program needs to do some sort of I/O at some point: read a file, write to the terminal, open a socket. If user mode code cannot do any of that directly, how does it get anything done?
\nAgain, the CPU comes to our rescue. It gives us a controlled way to cross that boundary through a mechanism called a trap. A trap is how the CPU transfers control from user mode to the kernel: it stops what it was doing, switches to kernel mode, and jumps to a kernel handler. This happens both when something goes wrong, like a divide by zero or an illegal memory access, and when your program deliberately asks the kernel for help.
\nBut for any of this to work, the kernel has to tell the CPU where to jump when a trap fires. It does this during boot, in a function called trap_init() (the following code is directly from the Linux codebase, at linux/arch/x86/kernel/traps.c):
void __init trap_init(void)\n{\n\t/* Init cpu_entry_area before IST entries are set up */\n\tsetup_cpu_entry_areas();\n\n\t/* Init GHCB memory pages when running as an SEV-ES guest */\n\tsev_es_init_vc_handling();\n\n\t/* Initialize TSS before setting up traps so ISTs work */\n\tcpu_init_exception_handling(true);\n\n\t/* Setup traps as cpu_init() might #GP */\n\tif (!cpu_feature_enabled(X86_FEATURE_FRED))\n\t\tidt_setup_traps();\n\n\tcpu_init();\n}Before explaining what matters in this code, let's state something first: beyond the general purpose registers like rax and rdi, the CPU has a set of special registers that hold its own internal configuration state. These are definitely kernel mode only, your program can never read or write them directly. Some of them are relevant to how traps work, and the kernel writes to them during boot so the CPU knows what to do the moment a trap fires.
"How does the CPU know where to jump when an exception like a divide by zero happens?" Good question indeed. The answer is a table the kernel builds during boot called the IDT, Interrupt Descriptor Table. Each entry maps an exception type to a handler address. Divide by zero gets one handler, page fault gets another, illegal instruction gets another. The kernel fills this table with idt_setup_traps() and writes its address into a special CPU register. On newer Intel CPUs that support a feature called FRED, this whole mechanism is replaced with a cleaner design, but that is not our subject today.
Those are the unwanted cases, the errors your program did not ask for. Returning to our question, how can a user mode program do any sort of I/O? Remember that a trap switches the CPU to kernel mode. The CPU gives us a way to trigger one deliberately. Instead of waiting for something to go wrong, your program can intentionally fire a trap, cross the boundary into kernel mode, and ask the kernel to perform a privileged operation on its behalf: write to a file, read from a socket, open a device. That deliberate request is what we call a system call.
\nAnd just like exceptions, the CPU needs to know where to jump when a system call fires. Back to trap_init(), the line that handles this is cpu_init(). It looks like generic initialization but it eventually calls syscall_init() inside arch/x86/kernel/cpu/common.c, which writes the system call entry point address into a special CPU register called MSR_LSTAR. When your program fires a system call, the CPU reads that register and jumps there instantly. No table lookup, just one register read and a jump.
Here is the chain, directly from the kernel source:
\ncpu_init() calls syscall_init():
// arch/x86/kernel/cpu/common.c\nvoid cpu_init(void)\n{\n // ...\n syscall_init();\n // ...\n}syscall_init() calls idt_syscall_init():
void syscall_init(void)\n{\n\t// ...\n if (!cpu_feature_enabled(X86_FEATURE_FRED))\n idt_syscall_init();\n}And finally, idt_syscall_init() is where it happens. The kernel writes the address of its system call entry point into MSR_LSTAR:
static inline void idt_syscall_init(void)\n{\n wrmsrq(MSR_LSTAR, (unsigned long)entry_SYSCALL_64);\n // ...\n}From this point on, every syscall instruction executed anywhere on the system will cause the CPU to read MSR_LSTAR and jump to entry_SYSCALL_64. That is where your program lands when it triggers a system call.
Given this mechanism the CPU hands us, how does a program actually trigger a system call? Remember printf() from the beginning? One might think "but it just writes some text to the terminal, why does that need the kernel?"
Because in Linux, writing to the terminal is just like writing to a file. If you have read my networking post, you already know why: every form of I/O in Unix is represented as a file descriptor, a handle to an open resource that lives inside the kernel. Writing to a file requires privileged access.
\nYour program cannot just reach into the filesystem and put bytes somewhere. That is what write() is for: a system call that says "take these bytes and write them to this file descriptor."
Don't take my word for it. Let's use a tool called strace, which intercepts and prints every system call a program makes as it runs. Take this simple C program:
#include <stdio.h>\nint main() {\n printf(\"hello world\\n\");\n return 0;\n}Running it under strace gives us:
$ strace ./main\nexecve(\"./main\", [\"./main\"], ...) = 0\nmmap(...) = ...\nopenat(..., \"/usr/lib/aarch64-linux-gnu/libc.so.6\", ...) = 3\n...\nwrite(1, \"hello world\\n\", 12) = 12\nexit_group(0) = ?Two things stand out here. First, all those calls before write are not your code at all. That is the dynamic linker loading the C standard library before your main even runs. Second, see that there is no printf in the output? Just write. That is because printf lives in libc and eventually calls write(), which is also a libc wrapper, a thin assembly function dressed up as a nice C API.
To trigger a system call, the CPU provides a dedicated instruction called syscall. When executed, it switches the CPU to kernel mode and jumps to entry_SYSCALL_64. But before firing it, your program has to tell the kernel what it wants and pass the arguments. This is done through registers, following a contract called the x86-64 syscall ABI. rax gets the syscall number, and the arguments follow in rdi, rsi, rdx, r10, r8, and r9 in order. The kernel will read exactly those registers when it takes over.
For write(1, "hello world\\n", 12) that looks like this:
mov rax, 1 ; syscall number for write\nmov rdi, 1 ; fd = stdout\nmov rsi, buf ; pointer to \"hello world\\n\"\nmov rdx, 12 ; length\nsyscall ; switch to kernel mode, jump to entry_SYSCALL_64Each system call has a unique number. write is 1, read is 0, open is 2. That number goes into rax before firing syscall, and the kernel uses it to look up the right handler in a table. You can see the full table directly in the kernel source here, or look up any syscall and its arguments by running:
$ man 2 writeThe 2 refers to section 2 of the manual, which is system calls. Every syscall has a page there.
The kernel maintains a table where each syscall number maps to a handler function. That 1 you put in rax is literally just an index into this:
long x64_sys_call(const struct pt_regs *regs, unsigned int nr)\n{\n switch (nr) {\n #include <asm/syscalls_64.h>\n default: return __x64_sys_ni_syscall(regs);\n }\n}The #include <asm/syscalls_64.h> inside the switch is interesting. At build time the kernel runs a script that reads syscall_64.tbl and generates a header that expands to a case for every single syscall. What looks like one line is actually hundreds of cases: case 0: return __x64_sys_read(regs); case 1: return __x64_sys_write(regs); and so on. So nr == 1 calls write, nr == 0 calls read. Do not worry about pt_regs for now, we will get there.
The point is that the syscall number is nothing magical, it is just an index. The kernel looks it up, calls the right function, and does the work.
\nBefore diving into the entry point assembly, let me clarify a few things.
\nEach CPU has one set of registers: rax, rdi, rsi, rbx, and the rest. When your program is running, those registers hold your data: local variables, pointers, return values, things your program is actively using. When syscall fires, the kernel takes over on that same CPU using those same registers. If it just starts using them, your program's state is gone.
In a normal C function call, the x86-64 ABI handles this through a split between caller-saved and callee-saved registers. Caller-saved registers like rax, rcx, rdi can be trashed by any function you call. Callee-saved registers like rbx, rbp, r12 through r15 must be restored before returning. Both sides agree on this contract and everyone is happy.\nBut a syscall is not a normal function call. The kernel does not follow your ABI. It is completely separate code running at a different privilege level. It cannot promise to restore anything. So the kernel has to save everything before it touches a single register. That is the first thing that happens after syscall fires.
But save them where? The kernel cannot use your user stack. Your stack pointer might be corrupted or pointing at garbage. Even if it is valid, the kernel would be writing sensitive data into memory your program can read and modify. So the kernel has its own stack, one per thread, allocated when the thread is created. It lives in kernel virtual memory, above 0xffff800000000000, a region your process can never touch. It does not appear in /proc/PID/maps. It is completely outside your process memory structure, invisible to your program. It sits empty, waiting for a trap to fire.
Now, for the kernel to switch to that stack the moment entry_SYSCALL_64 runs, it needs to know where it is. That is what the TSS (Task State Segment) is for. It is a structure the kernel maintains per CPU core. Its sp0 field always holds the kernel stack address of whichever thread is currently running on that CPU:
struct x86_hw_tss {\n unsigned int reserved1;\n u64 sp0; /* kernel stack pointer for the current thread */\n u64 sp1;\n u64 sp2;\n // ...\n} __attribute__((packed));Every thread has its own kernel stack, allocated at thread creation and fixed for its lifetime. The stack itself never moves. What changes is sp0. Every time the scheduler picks a new thread to run on a CPU, it updates sp0 and a per-CPU variable called cpu_current_top_of_stack to point at that thread's kernel stack:
/* called on every context switch */\nvoid update_task_stack(struct task_struct *task)\n{\n load_sp0(task_top_of_stack(task));\n}This updates both sp0 in the TSS and a per-CPU variable called cpu_current_top_of_stack, always keeping them in sync. By the time entry_SYSCALL_64 runs, that variable already holds the right kernel stack address for the current thread. So when syscall fires, the kernel already knows exactly which stack to switch to. It was prepared by the last context switch and it is waiting to be used.
Remember cpu_init_exception_handling() from trap_init() earlier? That is where the TSS gets registered with the CPU at boot:
void cpu_init_exception_handling(bool boot_cpu)\n{\n struct tss_struct *tss = this_cpu_ptr(&cpu_tss_rw);\n // ...\n set_tss_desc(cpu, &get_cpu_entry_area(cpu)->tss.x86_tss);\n load_TR_desc();\n // ...\n}This is a two step process. set_tss_desc() registers the TSS in the GDT, the global table where the CPU looks up system structures. Think of it as adding the TSS to a directory so the CPU can find it. Then load_TR_desc() executes the ltr instruction, which loads that GDT entry into the Task Register, a dedicated CPU register that holds the currently active TSS. After ltr runs, the CPU no longer needs to search the GDT. It already knows where the TSS is, and the moment a trap fires, it reads sp0 from it instantly.
You do not need to memorize any of this. I certainly do not. The important thing is just to understand that all this preparation means the kernel knows exactly which stack to switch to the moment it needs it. It was all wired up at boot, waiting to be used.
\nTo make this concrete, here is the full chain from boot to the moment your syscall lands on the kernel stack:
\n![\"Boot-time](\"/assets/kernel_stack_setup.png\")
And with all that wired up at boot, your one-line printf can finally make it to the kernel. Remember MSR_LSTAR, the special CPU register the kernel wrote at boot to tell the CPU where to jump when a syscall fires? That address is entry_SYSCALL_64, and we can finally handle the syscall. And if all this complexity just to cross a privilege boundary feels overwhelming, well, now you know why there are not that many operating systems in the world.
The following is real kernel assembly, don't be scared.
\nThe CPU jumped to entry_SYSCALL_64. Everything we just covered, the TSS wired up at boot, the kernel stack allocated per thread, cpu_current_top_of_stack updated on every context switch, it all comes together in the very first two instructions.
The first thing entry_SYSCALL_64 does is switch to the kernel stack. The kernel does it itself in the very first instructions, reading from cpu_current_top_of_stack, the per-CPU variable that was updated on the last context switch:
movq %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)\nmovq PER_CPU_VAR(cpu_current_top_of_stack), %rspYour rsp is stashed into a scratch slot in the TSS and replaced with the kernel stack pointer. The stack is now switched.
Remember when we said the kernel has to save everything before it touches a single register? This is where that actually happens. It builds a pt_regs struct by pushing your registers one by one onto the kernel stack:
pushq $__USER_DS\npushq PER_CPU_VAR(cpu_tss_rw + TSS_sp2) /* your rsp */\npushq %r11 /* rflags */\npushq $__USER_CS\npushq %rcx /* your return address */\npushq %rax /* syscall number */\n\nPUSH_AND_CLEAR_REGSThose pushes map directly onto this struct:
\nstruct pt_regs {\n unsigned long r15;\n unsigned long r14;\n unsigned long r13;\n unsigned long r12;\n unsigned long bp;\n unsigned long bx;\n unsigned long r11;\n unsigned long r10;\n unsigned long r9;\n unsigned long r8;\n unsigned long ax;\n unsigned long cx;\n unsigned long dx;\n unsigned long si;\n unsigned long di;\n unsigned long orig_ax; /* syscall number */\n unsigned long ip; /* your return address */\n unsigned long cs;\n unsigned long flags;\n unsigned long sp; /* your rsp */\n unsigned long ss;\n};Every register your program was using is now sitting safely on the kernel stack. PUSH_AND_CLEAR_REGS saves the remaining general purpose registers and zeroes them after. The kernel runs with full privileges, and leaving user controlled values in registers is an attack surface. A clean slate before kernel execution is not optional.
With all of that done, the kernel calls into C:
\nmovq %rsp, %rdi\nmovslq %eax, %rsi\n\ncall do_syscall_64That assembly is literally calling the function below, passing pt_regs as the first argument and the syscall number as the second. Inside it, do_syscall_x64 dispatches to x64_sys_call, the giant switch statement we saw earlier that maps your rax number to the actual handler:
__visible noinstr bool do_syscall_64(struct pt_regs *regs, int nr)\n{\n nr = syscall_enter_from_user_mode(regs, nr); /* seccomp, audit */\n\n add_random_kstack_offset(); /* security mitigation */\n\n do_syscall_x64(regs, nr);\n\n syscall_exit_to_user_mode(regs); /* check pending signals */\n\n return true; /* use SYSRET */\n}\n\nstatic __always_inline bool do_syscall_x64(struct pt_regs *regs, int nr)\n{\n unsigned int unr = nr;\n if (likely(unr < NR_syscalls)) {\n unr = array_index_nospec(unr, NR_syscalls);\n regs->ax = x64_sys_call(regs, unr); /* the switch we saw earlier */\n return true;\n }\n return false;\n}Whatever the handler returns lands in regs->ax. That is the return value of your syscall, the number of bytes written, the file descriptor, the error code. It will be sitting in rax when your program gets control back.\nOnce the handler is done, POP_REGS mirrors what PUSH_AND_CLEAR_REGS did, restoring every register from pt_regs back into the CPU. Then the kernel exits with:
sysretqsysretq is the symmetric counterpart to syscall. When syscall fired, the CPU saved your return address into rcx and your flags into r11. sysretq reads them back, restores rip from rcx and rflags from r11, drops the privilege level back to user mode, and jumps to where you left off. One instruction undoes the crossing.\nYour program continues as if nothing happened. write returned its byte count, libc handed it back to printf, printf returned, and your main kept running.\nTo make this concrete, here is the full path from your printf call down to the kernel and back:\n![\"End-to-end](\"/assets/printf_flux.png\")
We started with a single printf("hello world\\n") and ended up reading kernel assembly. Along the way we saw how the CPU enforces privilege levels in hardware, how the kernel sets up the IDT and MSR_LSTAR at boot so it always knows where to jump, how every thread has a kernel stack sitting invisible in memory waiting for a trap to fire, and how entry_SYSCALL_64 orchestrates the entire crossing: switching stacks, freezing your registers, dispatching to the right handler, and returning as if nothing happened.
The kernel is not magic. It is just code that runs on the same CPU as yours, with a lot of careful setup done at boot time to make the handoff work. I hope it feels a little less like a black box now.
\nIf you want to go deeper, the next natural steps are signal handling, which happens right before sysretq returns, virtual memory and how the page tables get switched on entry, and the scheduler, which is what keeps cpu_current_top_of_stack always pointing at the right place. Each one of those is its own rabbit hole. But now you have the foundation to read them.
If you come from C, Rust strings will probably feel unfamiliar at first. The underlying concepts are the same: bytes in memory, pointers, lengths. What's different is that Rust wasn't built under the same constraints, so it made different tradeoffs. Let's start from the ground up.
\nIn C, a string is a char*. A pointer to a sequence of characters that ends when you hit a null byte. That's the entire contract.
char *s = \"hello\";![\"C](\"/assets/c-string-in-memory.png\")
There's no length stored anywhere. If you want to know how long the string is, you call strlen, which walks every byte until it finds \\0. That's O(n). Every time.
size_t strlen(const char *s) {\n size_t len = 0;\n while (s[len] != '\\0') {\n len++;\n }\n return len;\n}This also means you can't have a null byte in the middle of a string in any meaningful way. The moment strlen or any standard function hits \\0, the string is over as far as they're concerned.
It works, and it's simple, but you're trading a lot for that simplicity. C was designed in the early 70s under real constraints: memory was scarce, hardware was limited, and storing an extra integer alongside every string was not a free decision. And it wouldn't even be a single byte: 1 byte caps you at 255 characters, so you'd need at least 2 or 4 bytes to store anything useful, which starts to feel expensive on machines with kilobytes of RAM. The null terminator was a pragmatic call for its time.
\nBut it also means every time you allocate or copy a string, you have to remember to account for it. That extra + 1 for the null byte is easy to forget:
char *s = \"hello\";\nchar *copy = malloc(strlen(s)); // wrong, should be strlen(s) + 1\nstrcpy(copy, s); // writes '\\0' past the allocationThis class of bug has caused more security vulnerabilities than most people care to count. The null terminator is not just a performance tradeoff, it's an ongoing source of correctness problems that you carry with you every time you touch a string.
\nTo understand why Rust does it differently, the easiest way is to understand memory itself, without a bunch of abstractions.
\nWhen a program runs, the OS loads it into RAM and gives it a structured chunk of memory to work with. There's a lot going on in there, but we only need to care about three regions today.
\n![\"Process](\"/assets/process-memory-layout.png\")
The stack is where local variables live. Every time you call a function, a new frame is pushed onto the stack with space for that function's variables. The size of everything on the stack must be known at compile time, because the compiler has to calculate how much space each frame needs before the program runs. When the function returns, the frame is popped and that memory is gone.
\nThe heap is a pool of memory you can request chunks from at runtime, in sizes you don't need to know ahead of time. The allocation lives until you explicitly free it, or in Rust's case, until the owner goes out of scope.
\nThe rodata segment is where read-only data lives, things baked into the binary at compile time that never change.
\nWith that in mind, strings start to make a lot more sense.
\nWhat if you just stored the length? Then you wouldn't need a sentinel value at the end. You wouldn't need to walk the bytes to find the size. A null byte would just be a byte. No more + 1, no more walking until you find something that isn't the string.
That's what Rust does. The tradeoff is a few extra bytes per string. The gain is that an entire class of bugs simply stops existing. You can't forget to account for a null terminator that isn't there. You can't read past the end of a string because the length is always known. The size is part of the type, not a convention you have to remember to follow.
\nBut where exactly that length lives depends on where the string lives. There are three cases worth understanding.
\nlet s: [u8; 5] = *b\"hello\";![\"Stack](\"/assets/stack-string-rust.png\")
The bytes live directly in the stack frame. There is still a pointer involved when you take a reference to it, but that pointer points back into the stack itself, not somewhere else. Nothing is allocated on the heap, nothing is looked up elsewhere.
\nThe limitation is the same as anything else on the stack: the size must be known at compile time. [u8; 5] and [u8; 10] are completely different types. It is closer to char s[5] in C than to anything you would normally call a string, but the memory layout is the same idea.
The most common compile-time case is the string literal. Its bytes are placed in the rodata segment before your main even runs, and they stay there for the entire lifetime of the program. You can't modify them and you don't need to free them.
let s = \"hello\";![\"String](\"/assets/string-literal-rust.png\")
No heap, no capacity. Just two fields on the stack: a pointer to the bytes and their length. This is what is generally called a fat pointer, a pointer that carries extra metadata alongside the address. A regular pointer is just an address. A fat pointer is an address plus something else, in this case the length. The type is &'static str. Static because the data lives for the entire duration of the program.
The heap case is what you get when the string needs to be dynamic: it can grow, shrink, be built at runtime from user input or a file or a network response. You don't know the content at compile time, so the bytes can't live on the stack or in rodata. They go on the heap.
\nWhen you write String::from("hello"), Rust puts the bytes on the heap and creates a struct on the stack to manage them. In Rust, the ownership system handles freeing for you: when the String goes out of scope, the heap allocation is freed automatically.
![\"Heap](\"/assets/heap-string-rust.png\")
Three fields on the stack: a pointer to the heap allocation, the current length in bytes, and the total capacity. The heap side is just raw bytes. No null terminator. No length prefix sitting next to the data. Nothing but the characters themselves.
\nThe length is not near the data. It lives in the struct, on the stack, separated from what it describes. This matters and will come back later.
\nYou can verify there's nothing after the bytes:
\nlet s = String::from(\"hello\");\nlet ptr = s.as_ptr();\n\nunsafe {\n for i in 0..5 {\n print!(\"{} \", *ptr.add(i)); // 104 101 108 108 111\n }\n println!(\"{}\", *ptr.add(5)); // whatever was in memory before, not a null byte\n}len() in Rust is O(1). You read the field. Done. And a \\0 in the middle of a String is just another byte:
let s = String::from(\"hel\\0lo\");\nprintln!(\"{}\", s.len()); // 6, not 3You'll see String and &str all over Rust code. But they are not the same thing, and the reason why gets at something fundamental.
\nstr is the type that represents a sequence of UTF-8 bytes. Not &str. Not String. Just str. And you almost never use it directly, because you can't.
let s: str = \"hello\"; // doesn't compileThe compiler refuses because str is a dynamically sized type, a DST. Its size is not known at compile time. Every value you put on the stack needs a known size so the compiler can figure out how much space to reserve. str has no fixed size. It could be 3 bytes, it could be 300. The compiler cannot reserve stack space for something whose size only exists at runtime.
![\"Sized](\"/assets/sized-vs-unsized-types.png\")
Think about what a slice actually is. [u8] is not a container, it's not a struct, it has no metadata attached to it. It is literally just the bytes themselves, sitting somewhere in memory. A pointer to a [u8] is a pointer to the first byte and nothing else. There is no way to know where it ends without extra information. The size is not in the type, it's in the data, and the data is only knowable at runtime.
str is exactly that: a slice of bytes. Same problem. This isn't specific to strings either. Every slice type in Rust has it. [T] is a DST for the exact same reason, and you can't have a bare [i32] on the stack either.
So you can't have a bare str. You always need to go through a reference, and that reference needs to carry the length with it. Which brings us back to the fat pointer.
A normal pointer would not be enough. If &str were just a pointer to the bytes, you'd lose the length, and you'd be back to C territory, walking until you hit a sentinel.
So &str is a fat pointer. Two fields: a pointer to the bytes and the length. The size of &str itself is fixed: two machine words. The size of the str it points to is carried inside it. That's how you hold a reference to an unsized type without losing information. Now the compiler is happy. You can pass &str around, put it on the stack, store it in structs.
So you have a String. A struct on the stack with ptr, len, and cap, pointing at bytes on the heap. You want a &str: a fat pointer directly to those bytes.
These are different things. How do you get from one to the other?
\nManually you'd do this:
\nlet s = String::from(\"hello\");\nlet ptr = s.as_ptr();\nlet len = s.len();\n// now you have what a &str needsYou pull the pointer and the length out of the struct and construct the fat pointer yourself. Capacity is irrelevant, you leave it behind.
\n![\"From](\"/assets/string-to-str.png\")
This is exactly what happens when you write:
\nlet s = String::from(\"hello\");\nlet slice: &str = &s;But it doesn't happen by accident. Something has to automate that extraction. That something is Deref.
String implements Deref with Target = str. That impl is what makes &String coerce into &str transparently.
Here's a simplified version of what it looks like:
\nuse std::ops::Deref;\n\nstruct MyString {\n ptr: *const u8,\n len: usize,\n cap: usize,\n}\n\nimpl Deref for MyString {\n type Target = str;\n\n fn deref(&self) -> &str {\n unsafe {\n let slice = std::slice::from_raw_parts(self.ptr, self.len);\n std::str::from_utf8_unchecked(slice)\n }\n }\n}Look at what deref uses: self.ptr and self.len. That's it. self.cap is never touched. The struct on the stack is the entry point, but what comes out the other side is a fat pointer aimed directly at the heap bytes, with the struct left behind.
This is why &str feels like it skips the struct. It does. Deref extracts exactly the two pieces a &str needs and constructs one, discarding the rest. Coming from C this feels like the compiler is doing something behind your back. But it's not hidden, it's just automated. The real String impl does the same thing, just with safe wrappers around the raw pointer construction.
Everything above works because Rust controls both sides. The length is always there, the null terminator is never needed.
\nBut C doesn't know any of this. When you need to pass a string to a C function, you need to go back to the old contract: a pointer to bytes ending in \\0.
That's what CString is for:
use std::ffi::CString;\n\nlet s = CString::new(\"hello\").unwrap();![\"FFI](\"/assets/ffi-boundary.png\")
It adds the null terminator back and gives you a pointer you can hand to C. The convention never went away, it just got pushed to the edges of the program, behind an explicit type that makes the cost visible.
\nThe full picture: C strings are bare pointers with an implicit length encoded as a sentinel, a pragmatic decision made under the constraints of its time. Rust stores length explicitly, keeps metadata separate from data, and uses a DST to make slices work uniformly whether the bytes are on the heap, the stack, or baked into the rodata segment. &str exists because str can't. And Deref is the mechanism that bridges the owned heap type with the borrowed slice type, skipping the struct and handing you a fat pointer straight to the bytes.
A socket is a way to speak to other programs using standard Unix file descriptors.\nLet's start with a famous catch: everything in Unix is a file.
\nThe "everything is a file" idea comes down to one observation: every I/O operation is fundamentally about moving bytes from one place to another. Whether you're reading from a keyboard, writing to a screen, or reading a file off disk, the operation is structurally identical. That flow of bytes has a name: a stream. An ordered sequence of bytes moving between two endpoints, with no built-in separators. Just bytes from A to B.
\nFor every kind of I/O in Unix, it uses a file descriptor. A file descriptor is simply an integer associated to an open file. These open files can be a network connection, a FIFO, a terminal, or obviously a disk file. They all use the same API: open(), read(), write(), close().
A socket is just another file descriptor, but instead of pointing at a file on disk, it points at a communication endpoint. Behind the hood, it is two kernel data structures layered on top of each other:
\n// include/linux/net.h\n// The VFS layer — what the file descriptor actually points to.\nstruct socket {\n socket_state state; // connected, listening, etc.\n short type; // SOCK_STREAM, SOCK_DGRAM, etc.\n struct file *file; // back-pointer to the fd\n struct sock *sk; // pointer to the protocol layer below\n const struct proto_ops *ops; // send, recv, bind, connect, etc.\n};\n\n// include/net/sock.h\n// The protocol layer — where the actual data and state live.\nstruct sock {\n struct sock_common __sk_common; // family, addresses, port\n int sk_rcvbuf; // receive buffer size\n int sk_sndbuf; // send buffer size\n struct sk_buff_head sk_receive_queue; // incoming packets\n struct sk_buff_head sk_write_queue; // outgoing packets\n int sk_state; // TCP_ESTABLISHED, TCP_LISTEN, etc.\n struct proto *sk_prot; // protocol impl (TCP, UDP...)\n};struct socket is the thin wrapper the kernel exposes through the file descriptor, it is what syscalls like send() and recv() operate on. struct sock is where the real work happens: the protocol family, addresses, and connection state, plus the two queues that do the actual data movement. The comments point to what each field is for. When you send() data, the kernel copies bytes from your userspace buffer into sk_write_queue, the network stack wraps them in headers and pushes them out. On the receiving end, incoming bytes land in sk_receive_queue and wait until you call recv().
The file descriptor you hold is just a handle. The real thing lives in kernel space.
\nTo get a socket file descriptor you run the syscall socket(). You communicate through this socket using send() and recv(). Although you can still use read() and write() for sockets, send() and recv() offer greater control over the data being sent and received.
There are more than one group of sockets: Internet Sockets, Raw Sockets, Unix Sockets (for IPC), etc. The two types of Internet Sockets you'll be most in contact with are Stream sockets (SOCK_STREAM) and Datagram sockets (SOCK_DGRAM), sometimes called "connectionless sockets".
Don't worry if TCP, UDP, IP, or ports don't mean anything yet. We'll cover all of them in the sections below.
\nStream sockets provide a reliable, two-way, connection-oriented communication stream. If you send data in the order "a, b", it will arrive in that same order, without errors or duplication.
\nIf you have used telnet or ssh you have used stream sockets, because all your input needs to be delivered in the same order. If you're reading this in a browser, you're already using a stream socket. Web browsers use HTTP (Hypertext Transfer Protocol) which uses streams to get pages.
What ensures the reliability of stream sockets? It is reliable due a protocol called the Transmission Control Protocol, or TCP. TCP's job is to make sure those bytes actually arrive, in order, without loss or duplication. The stream itself is just bytes moving from A to B. TCP is what makes that journey reliable.
\nYou might have seen TCP referred to as part of TCP/IP. That's because TCP doesn't work alone, it runs on top of IP (Internet Protocol). You'll read more about IP later.
\nWhen using stream sockets with TCP you need to maintain an open connection between both sides, whereas with Datagram sockets you don't, and that's why they are called "connectionless".
\nDatagram sockets use a protocol called UDP (User Datagram Protocol). With this type of socket, you don't need a connection, you just build the packet and throw it at the receiver. It doesn't keep track if everything went right.
\nHave you ever heard of FTP (File Transfer Protocol)? Its simpler cousin, tftp, does the same thing but over UDP. But wait, how can you reliably transfer a file over an unreliable protocol? The answer is that tftp builds its own reliability on top of UDP: for every packet received, the receiver sends back a small "I got it" reply, called an ACK (acknowledgment). If the sender doesn't hear back in time, it throws the packet again.
\nThe reason why such an unreliable protocol is used is, drumroll please... Speed! It is way faster to fire-and-forget than it is to track the connection. When you're implementing something that receives a ton of packets per second, it doesn't really matter if you lose one or two.
\nBefore we go further, let's define something we've been throwing around: a packet. When you send data over a network, it doesn't travel as one continuous stream. It gets chopped up into small chunks, each one wrapped with some header information about where it came from and where it's going. Each chunk is a packet. Keep this definition for now, we'll elaborate more later.
\nOk, we have packets, sockets, TCP, UDP... but how does any of this know where to go?
\nAn IP address is a unique identifier for a machine on a network. Just like a house has a postal address so letters can find it, every machine on the internet has an IP address so packets can find it.
\nUntil recently we only had IPv4, the classic format that looks like 192.168.1.1. Each of those four numbers is separated by a dot, goes from 0 to 255, which is a byte, so an IPv4 address is really just a 32-bit integer under the hood. That gives us about 4 billion possible addresses (2^32), which sounded like plenty at the time. It wasn't.
\n![\"IPv4](\"/assets/ipv4-address-format.png\")
With the rapid and unpredicted expansion of the internet, someone realized that 4 billion addresses wouldn't be enough and came up with IPv6. Instead of 32 bits, IPv6 uses 128 bits, giving us 2^128 possible addresses. That's a number so astronomically large it won't run out in any foreseeable future.
\nIt looks quite different from IPv4. Eight groups of four hexadecimal digits (remember that 2 hex digits = 1 byte), separated by colons:
\n2001:0db8:c9d2:0012:0000:0000:0000:0051
That's a lot to type. IPv6 allows you to compress it by dropping leading zeros and collapsing consecutive groups of zeros into :::
2001:db8:c9d2:12::51
Both represent the exact same address.
\n![\"IPv6](\"/assets/ipv6-address-format.png\")
You'll also see this a lot:
\n::1
That's the loopback address, the equivalent of IPv4's 127.0.0.1. It means "this machine itself". If you ever see it, it means the traffic never left your computer.
An IP address gets a packet to the right machine, but your machine is running dozens of programs that all want to use the network. How does the OS know which packet belongs to which program? That's what ports are for.
\nA port is a 16-bit number, from 0 to 65535. When a packet arrives, the OS looks at three things to decide where to send it: the protocol (TCP or UDP), the destination IP, and the destination port. It then hands the packet to whichever program has a socket open on that combination.
\nThis applies even locally: when you run a server on your own machine and open localhost:3000 in your browser, the 3000 is the port the OS uses to find the right program.
Ports exist on both sides of a connection. Servers listen on well-known ports so clients know where to connect. Clients also get a port, called an ephemeral port, automatically assigned by the OS for the duration of the connection so replies know where to come back to.
\nSome ports have well-known conventional mappings: port 80 for HTTP, port 22 for SSH, port 23 for telnet. Note that port 80 on TCP and port 80 on UDP are separate things. You can see the conventional mappings in /etc/services on any Unix machine, though it's just a reference, programs aren't forced to follow it.
Ports 0 to 1023 are privileged ports. On Unix, binding to them requires root. This is an OS-level convention, not a property of TCP/IP itself, so a privileged process can still take port 80, and a server will fail to start if something else already claimed it.
\nTo complicate your life, CPU architectures don't all agree on the order bytes should be stored in memory. Take the two-byte hex number a1b2 as an example. Some machines store it as a1 followed by b2, the natural reading order. This is called Big-Endian. Others store it reversed, b2 followed by a1. This is called Little-Endian. There's a good chance you're on an Intel or AMD machine, which is Little-Endian. The internet agreed on Big-Endian as the standard, which is why it's also called Network Byte Order.
![\"Big-Endian](\"/assets/network-byte-order.png\")
The reason this matters: when you send a number over the network, the bytes have to be in the right order or the other side will read them backwards and get a completely wrong value. Before sending, you convert to Network Byte Order. When receiving, you convert back to whatever your machine uses.
\nThe C standard library gives you four functions for this:
\nhtons() // host to network short (2 bytes)\nhtonl() // host to network long (4 bytes)\nntohs() // network to host short (2 bytes)\nntohl() // network to host long (4 bytes)You'll use these when filling out socket structs. As for why different hardware stores bytes in different orders in the first place, I'll spare you that rabbit hole. You're probably bored enough already.
\nWe said IP addresses are how machines find each other. But when you type google.com in your browser, you're not typing an IP address. So how does your machine know where to go? Someone has to translate that name into an IP. But who keeps track of all of this? Is there a giant notebook somewhere mapping every domain name to an IP address?
Not quite. The internet only understands IP addresses, but humans only remember names, so DNS (Domain Name System) exists as the bridge between the two. Every time you ping google.com, ssh user@myserver.com, or open a website, something is resolving that name to an IP before a single packet is sent. That translation involves a chain of requests bouncing between servers around the world, but from your perspective it just happens, invisibly, in milliseconds.
A common misconception is that DNS is a browser thing. That resolution happens at the OS level. The browser just asks the OS "what's the IP for this name?" and the OS handles the rest. ping, curl, ssh, any program that uses a hostname goes through the same process without knowing or caring how it works.
For that to work, your computer needs to know which DNS server to ask. That's just a setting, an IP address pointing to a resolver. By default your router hands you this automatically, usually pointing to your ISP's (Internet Service Provider, the company that provides your internet connection) DNS servers. But you can override it and point to any resolver you want, like 8.8.8.8 (Google) or 1.1.1.1 (Cloudflare).
You can also bypass DNS entirely for specific names by editing /etc/hosts on your own machine. Before asking any DNS server, your OS checks that file first. That's how localhost resolves to 127.0.0.1 without any network request involved.
Well, if you've reached here (without scrolling all the way 🧐), that was a lot of theory. You now have the basic vocabulary of networking. I plan to write a follow-up going deeper into TCP and UDP, the network layers they sit on, and actual code to make all of it concrete.
\n","date_published":"2026-04-15T00:00:00.000Z"},{"id":"what-is-a-tar-gz","url":"https://viniciusvidal.com/what-is-a-tar-gz","title":"After all, what is a .tar.gz file?","summary":"What a .tar.gz actually is, where its shape comes from, and what the flags in tar -xzvf mean.","content_html":"If you use Linux, you have downloaded a .tar.gz file. For most of my time using Linux I assumed the format was a Linux thing, the same way .exe feels like a Windows thing or .dmg feels like a Mac thing. You see it everywhere on Linux, so you just assume it is what it is and that's it.
I was studying git, trying to understand how it stores objects internally, and somewhere along the way I ran into the fact that before git existed, the way people shared code was by packaging the source into a .tar.gz and putting it on a server for other people to download. That had been the standard way Unix software got distributed for decades before GitHub or anything like it.
tar is short for tape archive, and the name is literal. It was written in 1979, on early Unix, to take files from a filesystem and lay them out as a single flat stream that could be written onto a magnetic tape and read back later.
Networking was rare and slow, and disks were small and expensive. If you needed to back something up or move it to another machine, the usual option was to write it onto a reel of magnetic tape and physically carry the tape over. Tar was the tool that turned a directory into something a tape drive could write.
\n![\"1970s](\"https://live.staticflickr.com/6220/6293910323_29f8b5c27e_b.jpg\")
A lot of the format makes more sense once you know it was aimed at tape:
\nBy the late 80s and through the 90s, the more common reason to encounter a .tar file was downloading software over the internet, and that brought a new constraint: bandwidth.
A 56k modem gave you about 5 kilobytes per second of real-world throughput. A one-megabyte download took a few minutes, a ten-megabyte one was something you started before going to do something else. Compression made a real, visible difference, and Unix software started getting distributed as tar archives passed through a compressor on the way out the door.
\nThe first common combination was .tar.Z, which was tar piped through a program called compress. When gzip was released in 1992 as a free, patent-free replacement, .tar.gz quickly took over and became the default shape of "downloadable Unix source code" for the next couple of decades.
tar and gzip are not one tool. They are two completely separate programs, written years apart, that get used together so often we write the two extensions back to back and treat the combination as if it were a single format.
tar is an archiver. It takes files and directories and packs them into a single file, preserving Unix metadata like permissions, ownership, timestamps, and symlink targets. It doesn't compress anything. A .tar file is usually a little larger than the sum of its inputs because tar adds a 512-byte header in front of every file.
gzip is a compressor. It takes a single file and makes it smaller using an algorithm called DEFLATE (the same one used inside zip and PNG). It only works on one file at a time, and it doesn't know anything about directories at all. If you run gzip mydir/ on a directory, it refuses and exits. If you run gzip report.txt, it replaces the file with report.txt.gz. That's the whole tool. This is exactly why pairing it with tar works so well: tar first flattens a whole directory into a single file, and then gzip has something it can actually compress.
So .tar.gz describes, in order, what happened to the data: bundled with tar, then compressed with gzip. .tar.bz2 is tar plus bzip2. .tar.xz is tar plus xz. .tar.zst is tar plus zstd. Same bundling step, different compressor.
![\"tar](\"https://commons.wikimedia.org/wiki/Special:FilePath/Targzip.svg\")
This is also why you basically never see .tar.gz on Windows, where .zip is the default. Zip came out of DOS in 1989, a few years before gzip even existed, and made the opposite design choice: bundling and compression are combined into a single format, handled by a single tool. That fits Windows, where the OS hides the shell and "right-click, extract" needs to just work. Unix went the other way because the whole point of Unix was that small composable tools beat one big format.
You can write the Unix pipeline out by hand:
\ntar -cf - mydir/ | gzip > mydir.tar.gzThe - tells tar to write to stdout instead of a file. Gzip reads from stdin, compresses what it gets, and the shell redirects the result into mydir.tar.gz. Tar doesn't know gzip exists, gzip doesn't know tar exists, and they only meet inside the pipe.
The benefit of this design is that you can swap the compressor without touching the archiver. When bzip2 came along and compressed better than gzip, nothing about tar had to change. Same with xz and zstd. The archive format stayed the same across decades of new compressors.
\nThe z in tar -xzvf (the other letters do unrelated things, which I'll get to in a bit) is a later convenience: at some point tar gained flags for calling the compressor on your behalf. z for gzip, j for bzip2, J for xz. Tar is still shelling out to a separate program, the flag just hides the pipe.
You still see .tar.gz pretty commonly. GitHub generates one on the fly when you click "Source code (tar.gz)" on a release page, every Docker image layer is a compressed tar archive, npm packages are .tgz, Python source distributions are .tar.gz. And gzip on its own (without tar) is even more everywhere. HTTP servers compress responses before sending them, and gzip is one of the options they can use, alongside others like brotli or zstd. This page actually got to your browser compressed with the same gzip from the same 1992 program. No tar in that case, because there's only one file in flight, just the compression half of .tar.gz doing its job on its own.
After all this context, the answer is a lot less mysterious. A .tar.gz is a tar archive (a directory flattened into a single stream, in a format originally shaped by magnetic tape) that has then been passed through gzip and compressed with an algorithm called DEFLATE. The format exists in two parts because Unix split bundling and compression into separate tools, so the compressor could be swapped out independently of the archiver.
tar -xzvf actually meanI learned tar -xzvf as a single chunk long before I ever read the flags:
x means extract. The other common one is c for create, which you'd use to make an archive instead of unpacking one.z means run it through gzip. For .tar.bz2 you'd use j, for .tar.xz you'd use J.v means verbose, so tar prints the name of each file as it processes it. Purely cosmetic, you can drop it.f means the next argument is the filename. Without f, tar would try to read from or write to a tape device, which is another sign of where the tool came from. The default destination really is a tape drive.So tar -xzvf something.tar.gz reads as: extract, decompress with gzip, be verbose, the file is something.tar.gz.