Simple-OS is a (rather basic) hobbyist operating system for x86-32bit platform. The target is to create a self-hosting operating system, i.e. one can compile Simple-OS in itself.
Although the final goal is to make the system self-hosting, we have planned for several practical milestone:
-
Milestone One: Bootloader Many of the existing tutorials rely on GRUB to boot the system and do some initialize (like entering Protected Mode), which we don't like. For educational purpose, we want to fully control everything, starting from booting the system. Therefore, we choose to implement a standalone bootloader as our first step.
- Enter 32-bit protected mode and do various initialize before passing control to the kernel
- Provide disk I/O routines and a (read-only) file system to host the kernel
- Able to parse and load a ELF kernel written in C
- Provide basic printing utilities for debugging
- Code should be self-contained, i.e. no external dependency.
- Finished: Implemented under the folder
bootloader
-
Milestone Two: Kernel with Input and Output
- Write in C, compile to a Multiboot ELF kernel file
- Initialize IDT and ISR
- Enable Paging and Higher Half Kernel
- Support
printf
to write various type of data as string to the screen - Support reading user input from keyboard
- After initialization, run a program that user can enter some text to the screen and edit/delete it
- Finished: Implemented under the folder
kernel
-
Milestone Three: Memory Management, User Space and Multiprocessing
- Implement a heap to do dynamic memory allocation
- Allow switch to user space and run ELF program under user/ring3 privilege
- Enable multi-processing with independent kernel stack and page directory, starting with Cooperative Scheduling and then Preemptive Scheduling
- Implement
fork
,exec
andyield
system calls on a simple read-only file system (potentially reuse the USTAR code from the bootloader) - Finished:
- Kernel heap
- ELF loader
- Cooperative and preemptive multi-tasking (single CPU, synchronized by disabling interrupt when doing system call)
- Exec system call based on the USTAR read-only file system
-
Milestone Four: Filesystem, Libc and Hosted Tool-chain
- Provide a readable & writable file system (FAT or Ext) and corresponding system calls
- Build user space standard C library with system calls, e.g. printf, malloc and open/read/write. Potentially port Newlib as our standard C library.
- Build our OS specific/hosted compiling tool-chain (Binutils and GCC)
- Finished:
- Simple-FS: A fully functional FAT32 file system
- Simple-newlib: Ported Newlib 4.1.0 with all system calls implemented
- Simple-gcc and Simple-binutils: Hosted Binutils 2.35.1 / GCC 10.2.0 tool-chain established (See HostedToolchain.md)
-
Milestone Five: Shell and User Space Applications
- Write a shell to allow navigating through the file system and execute applications
- Write an editor to show file content, allowing editing and saving to disk
- Port a simplified C compiler to the system
- Finished
- A shell with
cd
and program execution was implemented - Implemented various file management utilities including
ls
,mkdir
,rmdir
,mv
,cp
andrm
- Ported the simple text editor Kilo, with basic editing functionalities and C syntax highlighting
- A shell with
-
Milestone Six: Locks and IPC
- Provide synchronization mechanism for multi-tasking environment, like locks
- Provide inter process communication (IPC) mechanism like pipe and signal
- Finished
- Pipe is implemented with memory circular buffer as an file system
- Mutex lock (yield when waiting) and Readers–writer lock implemented
-
Milestone Seven: Graphics
- Switch to VESA Video Modes using the VBE (VESA BIOS Extensions Standard) functions (Ref: VESA Tutorial)
- Provide basic graphical drawing routines, like drawing a line, a rectangle
- Enable higher resolution text console (showing more than 80*25 characters), including text drawing (VGA Font), screen scrolling and text cursor
- Write an image viewer
- Finished
- Bootloader can now switch to VESA video modes and pass the VBE information to the kernel through multi-boot structure
- Implemented a video driver supporting double buffering which only redraw changed pixel by comparing with a third video buffer
- TTY now support emulated text mode under video modes. Text cursor is also available
- Image viewer
image
implemented with the support of stb_image.h
-
Milestone Eight: Networking
- Allow connecting to the Internet
- Implement DNS query and ping command
- Implement Ethernet, ARP, IPv4, ICMP/ping and UDP protocol
- Implement a subset of POSIX socket API
- In Progress
RTL8139
NIC driver implemented- Ethernet layer implemented (no caching)
- ARP probe/announcement implemented
- IPv4 layer implemented
ping
utility implemented (ICMP protocol)- Socket API supporting ICMP protocol implemented
-
Milestone Nine: Compiler
- Port an assembler, e.g. FASM
- Port a simplified C compiler, e.g. SmallerC
- Compile our text editor within our OS
- In Progress
- FASM ported
- SmallerC hosted-toolchain built (libc ported)
-
Milestone Ten: Window Manager and GUI
- Allow switching between two consoles at the same time
- Split screen to show two consoles
- Allow dynamically open and close new consoles
- Enable mouse and allow simple button interaction
-
Final Goal: Self-hosting
- Port a sophisticate enough compiler to the system
- Compile the source code of the system inside the system (self-compiling)
- Compile the compiling toolchain in the system and use them to compile the system (self-hosting)
-
QEMU Emulator: We will use QEMU to emulate our system, avoiding restarting computer again and again just to test the system.
-
System wide GCC & Binutils toolchain: We will need them to compile a cross-compiler and build our OS specific toolchain at next step.
Installing 1-3 on
Ubuntu
:sudo apt -y update sudo apt -y install build-essential autoconf automake git sudo apt -y install bison flex libgmp3-dev libmpc-dev libmpfr-dev texinfo libisl-dev curl sudo apt -y install gcc-multilib sudo apt -y install qemu-system-x86 nasm
Installing 1-3 on
Arch Linux
:sudo pacman -Syy sudo pacman -S --needed base-devel gmp libmpc mpfr sudo pacman -S --needed git qemu qemu-arch-extra nasm
-
Hosted GCC & Binutils and Newlib for Simple-OS: We will need this specialized tool-chain and Newlib to compile those user space programs including init and shell.
To build the cross-compiler and our OS specific tool-chain, you can use the script
build-toolchain.sh
. It will run for 30 - 45 minutes depending on your machine's spec. Based on our experience, at least 4GB ram is required in building GCC and 8GB is recommended. Also, please prepare around 10GB of free disk space to hold all the intermediate files generated in the building process. -
(Optional) VSCode: We provide some integration of the building/debugging process with VSCode, but it is optional. Some extension may also be needed as described in the
Debug
section below.
If your tool-chain is built by the build-toolchain.sh
script, you can skip this section. The default values should work automatically. Otherwise, please still use build-toolchain.sh
as a reference.
Build environment variables are set up in config.sh
:
CROSSCOMPILERBIN
: Point it to the folder containing the cross-compiling GCC/Binutils binaries (see Dependencies section).AS
: Point to the system wide NASM assembler, if not set,nasm
is used.TOOL_CHAIN_ROOT
: Point to the location holding the Simple-OS hosted tool-chain and Newlib
Once you have the cross-compiler and hosted tool-chain ready, you can build Simple-OS by running:
./clean.sh
./build.sh
If compile finish successfully, bootable_kernel.bin
will be generated under the source root dir.
You can test run the compiled kernel by QEMU:
./qemu.sh
Anything written to the serial port (all outputs to the screen will be copied to the serial port by the kernel) will be logged into serial_port_output.txt
. Kernel debug info will also be available through the the file.
The script also check if there is a testfs.fat
image file under root dir, if so, it will mount it through the -hdb
argument when starting QEMU. This file should be a hard disk image of a FAT-32 file system. Simple-OS will try to mount it under /home
.
For testing purpose, you can use the FAT32 image provided by the FAT32-FS-Driver project. Or simply generate an empty FAT32 image (512MiB) by:
dd if=/dev/zero of=testfs.fat bs=1024 count=$(expr 512 \* 1024)
mkfs.vfat testfs.fat
You can mount the disk/partition image in Linux to manage the files in it:
mkdir -p mnt
sudo mount -t vfat testfs.fat mnt
# manage the files, once done, umount
sudo umount mnt
It is possible to debug the kernel by GDB. See QEMU GDB Usage.
First start QEMU in debug mode by running:
./qemu.sh debug
And then attach GDB to the QEMU instance by:
gdb -ex "target remote localhost:1234" -ex "symbol-file sysroot/boot/simple_os.kernel"
The above loads debug symbols for the kernel, to debug our own bootloader:
gdb -ex "target remote localhost:1234" -ex "symbol-file bootloader/bootloader.elf"
Note that bootloader/bootloader.elf
is generated solely to provide the debug symbols. The real bootloader binary is bootloader/bootloader_padded.bin
.
This debugging trick is also describe in the os-tutorial for macOS.
Better still, it is possible to do all the above graphically in VSCode.
.vscode/tasks.json
provide integrated Clean/Build/Emulate task.
You can trigger them in Command Palette (Ctrl+Shift+P).
.vscode/launch.json
provides the debug gdb debug profiles.
Assuming you are running Simple-OS in a remote (virtual) machine, to setup the remote debugging:
- How to SSH to a system inside Virtual Box
- Remote - SSH extension
- Native Debug extension for remote debug through SSH.
- VSCode extensions Remote X11 and Remote X11 (SSH) to run QEMU (GUI application) through SSH
- If your host machine is Windows, then a local X-window server is also needed. You can install vcxsrv.
- If you SSH into a virtual machine by port forwarding, change
remoteX11.SSH.host
andremoteX11.SSH.port
in "Remote X11 (SSH)" remote setting (e.g. change to 127.0.0.1 and 5679). Note that the extension does not support password authentication to the remote host, so please add your public key to the serve's~/.ssh/authorized_keys
(Ref: How To Set Up SSH Keys) - In Archlinux, X11 forwarding needs to be enabled explicitly, please see OpenSSH - X11 Forwarding
- GDB Init Script
- Copy/Append
.gdbinit
to~/.gdbinit
. The utility GDB functions defined there will be used in.vscode/launch.json
.
- Copy/Append
With all of the setup, the debugging process is streamlined to:
- Set break points in any source file
- Trigger
Build
task in Command Palette, make sure it runs successfully - Go to VSCode debugger, run
Debug All
- A QEMU session will be fired up and the system will run until a break point
- After finishing debugging, click detach debugger and close the QEMU window
Note: Some code in bootloader/arch/i386/bootloader.asm
and kernel/arch/i386/boot/boot.asm
can not be debug in this way, please see the comments there.
Familiarity with x86 assembly language and C will help you understand the code.
Tutorialspoint Assembly Programming Tutorial provides a good tutorial for NASM assembly, which is the assembly dialect we use in this project.
We also use Makefile to automate the compile process. The Tutorialspoint Unix Makefile Tutorial is a good introduction.
A one-stop shop for OS development knowledge is the OsDev Wiki. You can find various useful resources there, and there are bunch of tutorials to follow. It is highly recommended to take a look at the Bare Bones tutorial at OsDev Wiki to get some feeling on how to get started with OS development.
The bootloader/
folder contains a standalone bootloader that can load any ELF format kernel binary into memory and kick it started.
Files in the bootloader/
folder are mostly a combination of pieces scattered across many topics on the OsDev Wiki.
The entry point is at the boot
label in bootloader/arch/i386/bootloader.asm
.
The bootloader/arch/i386/
folder contains various initialization subroutines written in assembly.
- Entering 32-bit Protected Mode:
switch_pm.asm
: Switch to Protected Mode.a20.asm
: Enabling A20 address line.gdt.asm
: Initialize the GDT.
- Disk I/O:
disk.asm
: Read data from disk using BIOS interrupt (works in Real Mode).
- Printing:
print.asm
: Print using BIOS interrupt.print_hex.asm
: Print data as hex numbers.print_pm.asm
: Print in protected mode, using VGA video buffer.
linker.ld
: Linker script
Once we finish switching to protected mode, we start to write the remaining part in C.
bootloader/main.c
: The main C entry point, load kernel and pass control to it.elf/elf.*
: Parse ELF format binary and loader it to memory.tar/tar.*
: Provide basic utility to read a USTAR "file system".arch/i386/port_io.h
: Provide port I/O using C inline assembly.arch/i386/ata.*
: Read data from disk by ATA PIO mode (works in Protected Mode), referenced this tutorial- Printing utilities:
arch/i386/tty.*
,arch/i386/vga.h
: Write string to screen using VGA Text Modestring/string.*
: String processing utilities
The kernel/
and libc/
folders are build upon a clone from the Meaty Skeleton tutorial.
-
kernel/
block_io/
- Abstract layer for block storage
console/
- A console/terminal emulator implemented a subset of VT-100 ASCII escape control sequence.
elf/
- ELF binary parsing and loading
fat/
- A full functioning FAT32 filesystem. See Simple-FS
heap/
- Heap manager, allowing dynamic memory allocation and virtual memory manager, implemented as a sorted linked list of free memory blocks with memory header and footer inspired by INWOX
include/
- Headers will be copied by
kernel
'smake install
tosysroot/
under project root folder, which will ultimately be packed into the disk imagebootable_kernel.bin
bybootloader
's Makefile, i.e. the headers will be shipped along with the kernel binary executable. arch/i386/
: Architecture specific headersport_io/
: Inline assembly for port I/O, e.g. inb/outb (Examples)
- Headers will be copied by
kernel/
kernel.c
: Kernel main function
memory_bitmap/
- A physical memory manager / page frame allocator implemented as a bitmap.
panic/
- Utility functions to raise kernel panic
tar/
- A simple US-TAR (tar ball) based read-only filesystem
vfs/
- Virtual file system. An abstract layer to support mounting multiple device under a single rooted file tree.
arch/i386/
: Architecture specific implementationsarch_init/
: Collection of architecture specific initializationsata/
: 28 bit ATA PIO disk driverboot
:boot.asm
:- Intel/NASM syntax version of Meaty skeleton's boot.S
- Plus initializing paging and higher half kernel
gdt.asm
: Same as bootloader's flat mode GDT
cpu/
: CPU related operations lick enable/disable interruptscrt/
: C program initialization components for the kernel- Interrupt related
idt/
: IDT initialization and interrupt gate installationisr/
: CPU exceptions and IRQs handlers (ISRs)pic/
: PIC utility for IRQs
keyboard/
: Keyboard driverpaging/
: Virtual memory management and pagingprocess/
:process.c
: Process management (Ref: xv6/proc)start_init.asm
: First usage mode program to run, will call SYS_EXEC to replace itself withinit.elf
switch_kernel_context.asm
: Used to switch between CPU context between scheduler (context which we initially boot into) and individual process's kernel code (e.g. system call handler)
serial/
: Serial port I/O utilities. We output all printing though serial port so QEMU can export the printed content to a fileserial_port_output.txt
.syscall/
: Wrapper for various system calls, mostly just recovering the argument from the stack.time/
: RTC clocktimer/
: PIT timer (used for process switching)tty/
andvga.h
: VGA Text Mode driver
-
libc/
: Progressive implementation of the standard C library- Mostly from Meaty Skeleton, and some are from Xv6.
- Will be minimum in terms of feature set, since more complicated user space programs will be supported by ported external C library, like newlib.
User space applications to be compiled by os specific tool-chain and use Newlib as their standard C library.
applications
:init/
- Compile to the first user space program to be ran (
init.elf
), perform user space initialization (like preparing for stdin/stdout/stderr) and then fork-exec shell. Currently also be used to test several user space features.
- Compile to the first user space program to be ran (
shell/
- Shell, supporting
cd
commands and program execution with argc/argv- e.g. if user type
ls
and hit enter, the shell will try to findls.elf
in/usr/bin/
,/home/bin/
and the current working directory. If found, the shell will fork and execute it.
- e.g. if user type
- Shell, supporting
editor/
- A port from the Kilo project
- A simple editor with basic editing feature and C syntax highlighting
ls
,mv/
,rm/
,mkdir/
,rmdir/
- Various file management utilities
Multiple tutorials have been referenced in the development of this project.
-
OsDev WikiTutorial Series: Very useful starting point of learning OS development. You can start with Bare Bones and then look into Meaty Skeleton, which we actual use in this project.
-
For tutorial in Chinese, I would recommend the INWOX project.
-
os-tutorial: A very user friendly step by step OS dev tutorial. Many files of the bootloader or the kernel are modified from this repo. This tutorial also combines a lot of pieces from some of the following tutorials. In addition, this tutorial assumes a macOS environment.
-
Roll your own toy UNIX-clone OS a well known tutorial, very concise and easy to understand. You shall read it with the errata at hand.
-
The little book about OS development and Writing a Simple Operating System - from Scratch. They provide explanation to exemplary code for many OS concepts. They can be used as a complement to the OSDev Wiki.
-
MIT 6.S081 Operating System Engineering Course. This course provides lecture video this year so it can be seen as a good OS MOOC. It is based on a very popular educational system Xv6, which is a full functioning multi-process OS implemented within 10K lines of code. It will serve as our reference implementation. The course switched to RISC-V architecture in 2019 (Xv6-RISC), however for our purpose, the x86 version will be more useful. It also provides a functional classification for each file in the x86 version repo.
-
Build Your Own Text Editor A very good step by step tutorial on how to build a text application like an editor from scratch. The final product, Kilo, is around 1000 lines of code, and includes features like basic text editing, searching and even C syntax highlighting.