theseus-os / theseus

Theseus is a modern OS written from scratch in Rust that explores 𝐢𝐧𝐭𝐫𝐚𝐥𝐢𝐧𝐠𝐮𝐚𝐥 𝐝𝐞𝐬𝐢𝐠𝐧: closing the semantic gap between compiler and hardware by maximally leveraging the power of language safety and affine types. Theseus aims to shift OS responsibilities like resource management into the compiler.

Theseus OS

Theseus is a new OS written from scratch in Rust to experiment with novel OS structure, better state management, and how to leverage intralingual design principles to shift OS responsibilities like resource management into the compiler.

For more info, check out Theseus's documentation or our published academic papers, which describe Theseus's design and implementation.

Theseus is under active development, and although it is not yet mature, we envision that Theseus will be useful in high-end embedded systems or edge datacenter environments. We are continually working to improve the OS, including its fault recovery abilities for higher system availability without redundancy, as well as easier and more arbitrary live evolution and runtime flexibility.

Quick start

On Linux (Debian-like distros), do the following:

1. Obtain the Theseus repository (with all submodules):

   git clone --recurse-submodules --depth 1 https://github.com/theseus-os/Theseus.git
   

2. Install Rust:

   curl https://sh.rustup.rs -sSf | sh
   

3. Install dependencies:

   sudo apt-get install make gcc nasm pkg-config grub-pc-bin mtools xorriso qemu qemu-kvm wget
   

4. Build and run (in QEMU):

   cd Theseus
   make run

   To exit QEMU, press Ctrl + A, then X.

See below for more detailed instructions.

Building and Running Theseus

Note: when you first check out the project, be sure to get all the submodule repositories too:

git submodule update --init --recursive

Currently, we support building Theseus on the following platforms:

• Linux, 64-bit Debian-based distributions like Ubuntu, tested on Ubuntu 16.04, 18.04, 20.04.
  • Arch Linux and Fedora have also been reported to work correctly.
• Windows, using the Windows Subsystem for Linux (WSL), tested on the Ubuntu version of WSL and WSL2.
• MacOS, tested on versions High Sierra (v10.13), Catalina (v10.15), and Ventura (v13.5).
• Docker, atop any host OS that can run a Docker container.

Setting up the build environment

First, install Rust by following the setup instructions here. On Linux, just run:

curl https://sh.rustup.rs -sSf | sh

Building on Linux or WSL (Windows Subsystem for Linux)

Install the following dependencies using your package manager:

sudo apt-get install make gcc nasm pkg-config grub-pc-bin mtools xorriso qemu qemu-kvm wget

• Or:

  # Arch Linux
  sudo pacman -S make gcc nasm pkg-config grub mtools xorriso qemu wget
  
  # Fedora
  sudo dnf install make gcc nasm pkg-config grub2 mtools xorriso qemu wget

If you're on WSL, also do the following steps:

• Install an X Server for Windows; we suggest using Xming or VcXsvr.

  • You'll likely need to invoke those X servers with the -ac argument (or use the GUI to disable access control).

• Setup an X display as follows:

  • on original WSL (version 1), run:

    export DISPLAY=:0

  • on WSL2 (version 2), run:

    export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}'):0

  You'll need to do this each time you open up a new WSL terminal, so it's best to add it to the end of your .bashrc or .profile file in your $HOME directory.

• If you get an error like Could not initialize SDL (No available video device) ... or any type of GTK or video device error, then make sure that your X Server is running and that you have set the DISPLAY environment variable above.

• NOTE: WSL and WSL2 do not currently support using KVM.

Building on MacOS

• Install HomeBrew, then run the MacOS build setup script:

  sh ./scripts/mac_os_build_setup.sh

  If things go wrong, remove the following build directories and try to run the script again.

  rm -rf /tmp/theseus_tools_src

• NOTE: on MacOS, you need to run gmake instead of make for build commands (or you can simply create a shell alias).

  • This is because HomeBrew installs its binaries in a way that doesn't conflict with built-in versions of system utilities.

• (This is typically not necessary): if you're building Theseus on older Apple Silicon (M1 chips), you may need to use bash with x86 emulation:

  arch -x86_64 bash   # or another shell of your choice

  and possibly adjust your system PATH if both x86 and ARM homebrew binaries are installed:

  export PATH=/usr/local/Homebrew/bin:$PATH

Building using Docker

Note: building and running Theseus within a Docker container may be slower than on a native host OS.

1. Ensure docker scripts are executable:

   chmod +x docker/*.sh
   

2. (Skip if docker is already installed.) Install Docker Engine. We provide a convenience script for this on Ubuntu:

   ./docker/install_docker_ubuntu.sh
   

   • After docker installs, enable your user account to run docker without root privileges:
     sudo groupadd docker; sudo usermod -aG docker $USER
     Then, log out and log back in (or restart your computer) for the user/group changes to take effect.

3. Build the docker image:

   ./docker/build_docker.sh
   

   This does not build Theseus, but rather only creates a docker image that contains all the necessary dependencies to build and run Theseus.

4. Run the new docker image locally as a container:

   ./docker/run_docker.sh
   

   Now you can run make run or other Theseus-specific build/run commands from within the docker container's shell.

Notes on Docker usage:

• The docker-based workflow should only require you to re-run the run_docker.sh script multiple times when re-building or running Theseus after modifying its code. You shouldn't need to re-run build_docker.sh multiple times, though it won't hurt.
• KVM doesn't currently work in docker. To run Theseus in QEMU using KVM, you can build Theseus within docker, exit the container (via Ctrl + D), and then run make orun host=yes` on your host machine.

Building and Running

Build the default Theseus OS image and run it in QEMU:

make run

Or, build a full Theseus OS image with all features and crates enabled:

make full   ## or `make all`

Run make help to see other make targets and the various command-line options.

Using the Limine bootloader instead of GRUB

To use Limine instead of GRUB, clone pre-built limine and pass bootloader=limine to make:

git clone https://github.com/limine-bootloader/limine.git limine-prebuilt
git -C limine-prebuilt reset --hard 3f6a330
make run bootloader=limine

Feel free to try newer versions, however they may not work.

Targeting ARMv8 (aarch64)

Support for Theseus on aarch64 is an ongoing effort, but most of the core subsystems are complete.

To build and run Theseus on aarch64, first install the required dependencies:

• On Debian-like Linux (Ubuntu, etc):

  sudo apt-get install qemu-system-arm gcc-aarch64-linux-gnu

• On Arch Linux:

  sudo pacman -S aarch64-linux-gnu-gcc qemu-system-aarch64

• On macOS, the mac_os_build_setup script should have already installed this for you, but if not:

  brew install aarch64-elf-gcc aarch64-elf-binutils
  

Then, build Theseus and run it in QEMU:

make ARCH=aarch64 run

Doing a "full" build of all Theseus crates isn't yet supported on aarch64, as our aarch64 support in Theseus doesn't yet cover all crates in the entire repo.

Using QEMU

QEMU allows us to run Theseus quickly and easily in its own virtual machine. To release your keyboard and mouse focus from the QEMU window, press Ctrl + Alt + G, or Ctrl + Alt on some systems, or just Cmd + Tab out to another app on macOS. To exit QEMU, in the terminal window that you originally ran make run, press Ctrl + A then X, or you can also click the GUI ⓧ button on the title bar if running QEMU in graphical mode.

To investigate the hardware/machine state of the running QEMU VM, you can switch to the QEMU console by pressing Ctrl + Alt + 2. Switch back to the main window with Ctrl + Alt + 1. On Mac, manually select VGA or compact_monitor0 under View from the QEMU menu bar.

To access/expose a PCI device in QEMU using PCI passthrough via VFIO, see these instructions.

Linux does not support the ICMP protocol (for ping) for guest OSes in QEMU by default, so to allow ping to work on Theseus, you may need to run the following in your Linux host machine:

sudo sh -c "echo \"0 2147483647\" > /proc/sys/net/ipv4/ping_group_range"

KVM Support

While not strictly required, KVM will speed up the execution of QEMU. To install KVM, run the following command:

sudo apt-get install kvm

To enable KVM support, add host=yes to your make command, e.g.,

make run host=yes

Note that KVM acceleration is only supported on native Linux hosts.

Documentation

Theseus includes two forms of documentation:

1. The source-level documentation, generated from code and inline comments (via rustdoc).
   • Intended for Theseus developers and contributors, or those who want low-level details.
2. The book-style documentation, written in Markdown.
   • Useful for high-level descriptions of design concepts and key components.

To build the documentation yourself, set up your local build environment and then run the following:

make view-doc   ## for the source-level docs
make view-book  ## for the Theseus book

Other

Booting on Real Hardware

We have tested Theseus on a variety of real machines, including Intel NUC devices, various Thinkpad laptops, and Supermicro servers. Currently, we have only tested booting Theseus via USB or PXE using a traditional BIOS bootloader rather than UEFI, but UEFI is fully supported so it should work.

To boot over USB, simply run make usb drive=sdc, in which sdc is the device node for the USB disk itself (not a partition like sdc2) to which you want to write the OS image. On WSL or other host environments where /dev device nodes don't exist, you can simply run make iso and burn the .iso file in the build/ directory to a USB, e.g., using Rufus on Windows.

To boot Theseus over PXE (network boot), see this set of separate instructions.

Debugging Theseus on QEMU

GDB has built-in support for QEMU, but it doesn't play nicely with OSes that run in 64-bit long mode. In order to get it working properly with our OS in Rust, we need to patch it and build it locally. The hard part has already been done for us (details here), so we can just quickly set it up with the following commands.

1. Install the following packages:

   sudo apt-get install texinfo flex bison python-dev ncurses-dev
   

2. From the base Theseus directory, run this script to download and build GDB from an existing patched repo:

   curl -sf https://raw.githubusercontent.com/phil-opp/binutils-gdb/rust-os/build-rust-os-gdb.sh | sh
   

   After that, you should have a rust-os-gdb directory that contains the gdb executables and scripts.

3. Run Theseus in QEMU using make run (or make run_pause to pause QEMU until we attach GDB).

4. In another terminal window, run the following to start GDB and attach it to the running QEMU instance:

   make gdb 
   

   QEMU will be paused until we move the debugger forward, with standard GDB commands like n to step through the next instruction or c to continue execution. Any standard GDB commands will now work.

Connecting GDB to aarch64 Theseus on QEMU

We don't yet have a patched version of GDB for aarch64 targets, but we can use the existing gdb-multiarch package to

1. Install the required package:

   sudo apt-get install gdb-multiarch

2. Build Theseus for aarch64 and run it in QEMU:

   make ARCH=aarch64 run ## or use `run_pause`

3. In another terminal window, run the following to start GDB and attach it to the running QEMU instance:

   gdb-multiarch -ex "target remote :1234"

4. Within GDB, symbols aren't yet supported, but you can view assembly code with layout asm and set breakpoints on virtual addresses.

IDE Setup

Our personal preference is to use VS Code, which has excellent cross-platform support for Rust. Other options are available here.

For VS Code, recommended plugins are:

• rust-analyzer, by matklad
• Better TOML, by bungcip
• x86 and x86_64 Assembly, by 13xforever

Acknowledgements

We would like to express our thanks to the OS Dev wiki and its community and to Philipp Oppermann's blog_os for serving as excellent starting points for Theseus. The early days of Theseus's development progress are indebted to these resources.

License

Theseus's source code is licensed under the MIT License. See the LICENSE-MIT file for more.

Contributing

We adhere to similar development and code style guidelines as the core Rust language project. See more here.

PRs and issues are welcome from anyone; because Theseus is an experimental OS, certain features may be deprioritized or excluded from the main branch. Don't hesitate to ask or mention something though! 😄

theseus's Issues

Incorrect Terminal Prompt Position When mirror_to_vga is Enabled

When mirror_log_to_vga is enabled, the log messages (usually the Warn log message that the task exited with some value) will overshadow the most recent terminal prompt after a command is run, and the user has to type a character for a new prompt to show up

Byte size mismatch in ap_realmode assembly

@snoword there's a potentially major bug in ap_realmode, line 134. The linker throws a warning (it should probably be an error) because you're comparing a byte to the value 600, which is too large to fit in a byte. You should use the proper type, not byte but maybe a word? I dunno how big it is.

Clean up all Cargo.toml files

There is residue from Kevin's original version like: # "lib" does the same thing I think

Add Rust log support via serial console

Originally reported by: Kevin Boos (Bitbucket: kevinaboos, GitHub: kevinaboos)

---

https://doc.rust-lang.org/log/log/index.html#use-with-no_std

Rust's log crate supports no_std, we should utilize it and use set_raw_logger() to write to serial port.

---

• Bitbucket: https://bitbucket.org/kevinaboos/theseus-os/issue/1

Terminal code is sloppy

the task_handler function could use a rewrite, it's a mess and a bit hard to read.
Some points for improvement:

• way too many nested conditionals.
• current_task_id is dangerous, and unnecessarily slow. Instead of keeping current_task_id as an integer, simply keep a current_task_ref or something, which is of type Option<TaskRef>. That's much cleaner, and cannot fail when getting the current task.
• line spacing is clustered, hard to read. Indentation isn't proper.

Add interrupt-safe heap

Create another heap that will immediately (atomically) allocate and return memory. This is basically similar to the Linux kernel's notion of kmalloc GFP_ATOMIC.

Fix unsafe usage of AtomicMap

The key type of AtomicMap is fine (but could probably be changed to a reference for get() and get_mut()).

However, the value type of AtomicMap should be required to be marked Sync + Send, because it's not protected by anything. That is, the value should be wrapped in a thread-safe wrapper like Mutex or RwLock or Once, or should be another atomic type itself.

This applies to the TSS, GDT, and LOCAL_APICS lists. Here are the proposed fixes:

• TSS: modified on every context switch, so it should be wrapped in Mutex.

• GDT: created just one time and then never changed, so it should be wrapped in Once.

• LOCAL_APICS: modified sometimes, but read often, so probably should be wrapped in RwLock? Unsure, needs further investigation/consideration.

syscall_handler entry point is wrong, causes page fault in release mode

the page fault occurs in switch_to(). Haven't yet found why it occurs, but it is very consistent. Here is the output:

EXCEPTION: PAGE FAULT while accessing 0x8
error code: CAUSED_BY_WRITE
ExceptionStackFrame {
    instruction_pointer: 0xffffffff80108a01,
    code_segment: 8,
    cpu_flags: 0x2,
    stack_pointer: 0xffffffff80154018,
    stack_segment: 16
}

We should also find out which commit caused this to occur, and then diff it with the next one. Who wants to do some differential debugging?

Terminal overtype mode doesn't actually change the entered command string

When typing a command in the terminal, if you make a typo and need to change it, the backspace key works, but using the arrows to move the cursor left and right doesn't work. The cursor will indeed move, and the text on the screen will change, but the recorded command string does not change.

Example sequence to reproduce problem:

1. Type helto (typo)
2. Press the left arrow twice, cursor is now on t
3. Type l to overwrite the t
4. Press enter
5. it tries to run the command helto anyways, and fails.

VFS improvements

✔️ The ls app has redundant code that has been copy-pasted for two identical cases: when the path is specified, and when there is no path specified. Condense those into one case.

✔️ All file creation should accept a &DirRef rather than a WeakDirRef. This eliminates the potential error of not being able to upgrade the weak ref to the parent.

✔️ Don't use the return keyword at the end of a function, that violates Rust's style conventions.

✔️ TaskFiles are never actually created. Mmi files are pretty worthless at the moment.

folders for crates should be better organized

I understand Theseus has a flat modular architecture but that does not mean we should have a flat organization of the code. Can we create a hierarchy to organize the modules?

Terminal task using a lot of resources

terminal task ("default_terminal") is adversely affecting the performance on whatever core it spawns on, preventing other tasks from running. possibly from holding irqsafe locks

Check if the usb option is a device

In the Makefile, if the usb option is a partition and sdc4, the image will be written to it and the booting will fail.
Check if the option is a usb device directory.

Reserve memory below 0x1000000 for DMA usage

DMA address is according to this: http://wiki.osdev.org/Brendan%27s_Memory_Management_Guide

Basically in the AreaFrameAllocator, we need to exclude physical memory areas below 0x1000000 from being given to a frame.

Note: this is likely only necessary for ISA DMA, not the PCI Busmastering DMA that we will use for the ATA IDE disk.

Virtual Address Allocator leaves an allocated chunk in the beginning of the FREE_PAGE_LIST.

The allocator places fragmented (free) chunks in the back of the FREE_PAGE_LIST. However, it leaves allocated chucks (allocated == true) in the same place where the chunks are found.

This makes memory allocation slow because the allocator needs to skip the allocated chunks to find a free chunk.

Specifically, memory::allocate_pages() is related.

Use hashbrown instead of hashmap_core

It's been deprecated, see here: Amanieu/hashmap_core#8

Implement physical FS support

ext2 outline in Rust: https://github.com/gz/rust-ext2/blob/master/src/lib.rs

Use bitflags const hack

Example here: bitflags/bitflags#180

This can primarily be used for EntryFlags in places where we need a const-defined flag bitfield.

Move memory mapping from spawn_userspace to userspace_wrapper

Currently, spawn_userspace does the bulk of the work in creating the address space and loading/mapping the modules/programs into the new space. That means that the kernel thread calling spawn_userspace does a lot of the work and incurs huge delays calling that function; instead, it should be a very quick call. Then, in the userspace_wrapper, most of the work should be done by that actual task, not the task that invoked spawn_userspace.

This needs to be done for spawn_kthread and kthread_wrapper as well, though those are less critical because they don't result in an address table switch.

Terminal doesn't work with `mirror_serial` feature

When building with the mirror_serial feature enabled, the entering a command at the Terminal prompt locks up the entire machine!

To enable this feature for the make run target, add this line to your makefile:

run : export RUST_FEATURES = --manifest-path "captain/Cargo.toml" --features "mirror_serial"

right above the run target:

run: $(iso) 
	@qemu-img resize random_data2.img 100K
	qemu-system-x86_64 $(QEMU_FLAGS)

Then run make clean and make run again.

Use `cargo:rustc-cfg=` inside a build.rs script instead of using `--features`

Previously we have been using rustc --features (RUSTC_FEATURES) from the Makefile to specify which conditional compilation options we want. However, this has the massive downside of only allowing ONE single feature to be used across the entire OS, which is obviously insufficient for any amount of complicated configuration beyond setting one option in one crate.

Instead, we should use custom config tokens, e.g., cargo:rustc-cfg=kevin in the build.rs script. This allows any crate to use any number of config options, and isn't private to one crate. We can have one build.rs script that is used by all crates in the entire workspace.
Then, in the actual rust crate code, we can specify something like:

#[cfg(kevin)]
debug!("'kevin' cfg option enabled");

In this way, the build.rs script basically acts as a configuration choice tool.

Add symbolic links for files and directories

As discussed here: #93 (comment)

Use `unwrap_or_else` instead of `unwrap_or`, and `ok_or_else` instead of `ok_or`

The unwrap_or(default) and ok_or(default) functions are used ubiquitously, but they eagerly evaluate the default value, even when the actual value is Some or Ok. That's bad, and really wasteful.

Instead, we should convert them to unwrap_or_else(|| closure) and ok_or_else(|| closure), because the closures that provide the default value are not eagerly evaluated; they are only lazily evaluated if the original value was None or Err.

Improve Heap management

Add deallocation, reallocation in place, etc.

Or, adopt an existing no_std allocator, such as alloc_buddy_simple

Fix makefiles to depend on subdirectories

Right now, the makefiles that recursively invoke other makefiles nested in subdirectories do so without each subdirectory target being a dependency for the parent target. This causes every makefile to run every time, which is slow when cross-compiling with Xargo, or with the cargo build --release option.

Instead, we should make each subdirectory a dependency of a target, such that if the subdirectory hasn't changed, it won't be rebuilt.

MappedPages::as_func() should check for executable flags

the as_*_mut() functions check for writable flags, so the as_func() function should check that the underlying mapping is executable (on x86, that'd consist of checking that the NO_EXEC flag is not set).

Terminal does not display all text when scrolled

Split MMI vmas into kernel_vmas and user_vmas

They must be split, because like the kernel_stack_allocator, the kernel_vmas should have a single instance created once and then shared globally via an Arc<Mutex<vmas_list>>. User vmas are separate because they are not shared among all tasks.

Although, considering future usage, to allow multiple "thread" Tasks to share an address space, it might be a good idea to put each Task's MMI's user_stack_allocator and user_vmas in an Option<Arc<Mutex< >>>.

On real hardware, system does not issue a warning when size of heap is set greater than available memory

If I set
KERNEL_HEAP_INITIAL_SIZE = KERNEL_HEAP_MAX_SIZE (512 GiB)

on Qemu, I get an error reading "FATAL ERROR: AreaFrameAllocator: out of physical memory!!!", but when I run this on the server a few of the beginning debug messages are printed and then the system restarts. No error message is printed out.

Improve heuristic for finding a missing section's containing crate

Currently, to find missing symbols, we look for a containing crate name at the beginning of the symbol. This is done in the get_symbol_or_load() function and uses the get_containing_crate_name() function to determine the containing crate.

However, this doesn't always work, especially for generic functions whose specialized implementation will exist in another crate rather than in the crate that provides the generic definition.
For example, when running in debug mode (BUILD_MODE = debug), any application that relies on the getopts crate doesn't work. This is because the symbol "<core::slice::Iter<'a, T> as core::iter::traits::iterator::Iterator>::next::h6515c23f1fa80efe" actually exists in the getopts crate as a monomorphized generic implementation, but our get_containing_crate_name()-based heuristic looks for it in the core crate instead, and thus fails to find it.

We can solve this by using dependency information to narrow the search space of potential crates that may contain the symbol (I think Min Hong's compiler analysis tool already produces similar information). For example, the ls application depends on that above symbol in getopts, so we can pass that dependency info into Theseus at runtime in order to tell the crate manager to load getopts instead of core to fulfill the dependency on that symbol. This dependency info could either be crate-level or it could be symbol-level, in which we generate at build time a map of symbols to containing crate. This could be done for all symbols (which would be a huge huge map) or for only generic symbols, or in general, symbols that do not start with the name of their containing crate.

Panics in idle tasks cause the idle task to be killed, and no error message to display

For example, if there's an error in the captain::init function, then a panic is returned and the idle task is killed, meaning that the shutdown error message will not be shown.

We should avoid killing idle tasks for now. Not sure of a long-term solution.

Replace log with slog

The log crate is overly simple and has reached its limit within Theseus. We need a more robust logging facility, and I think the slog crate is a better choice.

Though not necessary, it'd be nice to preserve compatibility with the existing log crate, since some crates have an optional feature that can use the log crate to print debugging info (such as smoltcp).

Use lock-free heap allocator

Heap allocations are horribly expensive right now because the Heap acquires an Irq-safe Mutex lock, which disables interrupts for the duration of a heap allocation.

We should switch to a lock-free implementation of the heap, such that interrupts can remain enabled.

Add special message for stack overflow/underflow in the page_fault_handler

Currently if a function accesses data beyond its thread's stack, it will result in a page fault, which is good. However, for less experienced users, these page faults are hard to debug and result in frustration.

Thus, we should keep a list of all kernel stacks allocated so we can know if the failing address access (cr2 value) is just outside of a stack, perhaps by a single page. So if a stack goes from 0xB_0000 to 0xC_0000 and we get a page fault trying to access 0xC0100 or something, we know it's a stack problem and can give a better exception message in the page fault handler.

adding support for compiling on macOS.

On Lin's Mac, it failed at the linking stage. See below. Might be quite easy to resolve?

ld -n -T kernel/nano_core/src/boot/arch_x86_64/linker_higher_half.ld -o build/nano_core/nano_core-x86_64.bin build/nano_core/boot/x86_64/ap_boot.o build/nano_core/boot/x86_64/ap_realmode.o build/nano_core/boot/x86_64/boot.o build/nano_core/boot/x86_64/common.o build/nano_core/boot/x86_64/multiboot_header.o target/x86_64-theseus/release/libnano_core.a
ld: unknown option: -n

Avoid dependency hell by pushing Cargo.lock files to the repo

Relevant transcript of an IRC chat I had with the cargo team:

[13:49:54] Hi all, I know that you can set exact version dependencies with something like spin = "=0.4.5". Is there a way to globally specify that all dependencies should be exact, even if the Cargo.toml file doesn't state that they are?
[13:56:41] kevinaboos: Basically, you want spin = "0.4.5" to resolve to the minimum compatible version instead of the maximum compatible version by default?
[13:57:29] If you have one dependency that requires spin = "0.4.5" and another that requires spin = "0.4.4" do you want it to fail, or choose the minumum version that satisfies both?
[13:58:52] kevinaboos: Anyway, on nightly you can try the experimental cargo -Z minimal-versions update
[13:59:10] kevinaboos: Out of curiousity, what's your use case?
[13:59:39] mbrubeck: thanks, I had previously ran the minimal-versions command, and while that did work, I was wondering if there was a way to specify it as part of the build command.
[13:59:42] <-- pcwalton ([email protected]) has quit (Quit: My MacBook has gone to sleep. ZZZzzz…)
[14:00:49] kevinaboos: cargo -Z minimal-versions build should also work, but note that build will only do version resolution if there isn't already a Cargo.lock
[14:01:08] If I have a dependency of 0.4.4 and 0.4.5, I'd just want both of them to behave as if I specified "=0.4.4" and "=0.4.5"
[14:01:22] kevinaboos: Okay, so then it would fail to build
[14:01:32] There's no built-in way to do that, as far as I know.
[14:01:34] why would that fail to build?
[14:01:57] I'm able to use multiple versions of different dependencies across different crates, so why would that cause a problem?
[14:02:22] You can't link spin 0.4.4 and spin 0.4.5 into the same Cargo project. You can only link multiple versions if they are "incompatible" (i.e. the first non-zero component is unequal)
[14:02:49] Yes, these are in different crates though, sorry if I misspoke there.
[14:03:24] Okay, but if your crate depends on "A" which depends on spin "0.4.4" and also on "B" which depends on spin "0.4.5"
[14:03:35] Cargo has to choose a single version to satisfy both of those dependencies
[14:03:41] I see
[14:04:05] By default it chooses the maximum version that is compatible with both (spin 0.4.9)
[14:04:20] With the minimal-versions option, it should choose the minimum version that is compatible with both (spin 0.4.5)
[14:04:33] Okay thanks very much. It seems like the answer you mentioned above would work, where I add the minimal-versions argument in the build command, but I just have make sure to remove the Cargo.lock files first.
[14:04:52] kevinaboos: What's your use case, by the way?
[14:05:33] It's mostly a personnel issue, where other developers are the project aren't good about using exact dependencies like "=0.4.5"
[14:05:58] Why do you need to use exact dependencies, instead of using Cargo.lock for reproducible builds?
[14:06:41] oh wait
[14:07:28] do you mean that I can provide my Cargo.lock file for everyone else to use, and they'll get the same versions that my system chose to use when I built it?
[14:07:33] Yes
[14:07:46] I thought Cargo.lock files were system-local, and shouldn't be included in a repository
[14:07:55] This is why the default for binary crates is for Cargo.lock to be included in the repository
[14:08:41] (For library crates the default is not to check in Cargo.lock, since ultimately the Cargo.lock for the binary is what will matter for consumers)
[14:08:54] I see, all of my crates are lib, but since I'm generally using virtual workspaces, I have just a few Cargo.lock files. So I think in this case it would be safe to include them.
[14:09:26] Okay, but is it safe to include my Cargo.lock files even in lib crates?
[14:09:34] Yes, it's safe
[14:09:57] oh wow. Okay, I didn't realize they worked in that fashion. That's a game changer for us, thanks!
[14:10:00] And it will mean that people who build your lib directly (not as a dependency) will do it with all the same package versions as you
[14:10:29] Hmm, okay, I think it will still work.
[14:10:31] But note that it has no effect on "downstream" projects that use your lib as a dependency; they will get the versions specified in their own Cargo.lock files
[14:10:53] Right
[14:13:13] mbrubeck: Correct me if I'm wrong, but it sounds like it will work for this specific setup: I have one virtual workspace called "kernel" that consists of about 50 crates, and they all use a single Cargo.lock file. Since I build that "kernel" project once (and then link it myself), there wouldn't be any downstream projects that depend upon it. So, building the library itself is the downstream project, so pushing that kernel/Cargo.lock file should work as you describe?
[14:13:29] Right
[14:13:44] Okay, fantastic! Thank you very much.
[14:14:11] I've spent the last year fighting these version updates (since I'm on a specific version of Rust nightly), so this is big news. :)

Add support for thread-local storage

Use the FS segment register on x86_64. First start with just storing the current task id there, and then move on towards storing the actual current TaskRef there. This allows us to instantly look up the current task without having to query the TASKLIST, which becomes a bottleneck when there are thousands of Tasks.

Then, after that's working, we can add true support for arbitrary thread-local storage. It could potentially be inside the Task struct, or elsewhere.

Discussion here: https://wiki.osdev.org/Thread_Local_Storage

• Basic FS_BASE or GS_BASE support for custom usage, e.g., TaskLocalData.
• Support for #[thread_local] attribute in statically-linked base kernel image
• Support for #[thread_local] attribute in dynamically-loaded/linked crate object files
• Theseus-specific implementation of thread_local!() macro, à la the Rust std lib

Better interrupt lock handling

Originally reported by: Kevin Boos (Bitbucket: kevinaboos, GitHub: kevinaboos)

---

Right now we just stipulate as third-party knowledge that code in an interrupt context should not hold a lock. However, this isn't always possible (e.g., context switching).

Thus, we should use an intelligent lock that disables interrupts while it is held (obviously for very short-lived operations only) such that it can be guaranteed that the lock will be released safely without causing deadlock if another interrupt occurs while it's held.

See: HeldInterrupts in Tifflin OS

---

• Bitbucket: https://bitbucket.org/kevinaboos/theseus-os/issue/3

Auto-generate grub.cfg file to load all modules

Populate the grub.cfg file automatically based on the modules in the grub isofiles directory, which come from the userspace directory right now.

Should also move the grub.cfg out of the arch-specific file.

Fix design of ApplicationTaskBuilder

Right now the design of ApplicationTaskBuilder in the spawn crate requires it to maintain a redundant copy of each of the possible options that are also available in the KernelTaskBuilder. This is a bad design that makes it easy to forget to update the options (member variables) in both Builders, opening us up to application tasks and regular tasks not having the same options when being built.

Terminal sometimes panic when scrolling down

More common when mirror to log is enabled

compiler_builtins for __floatdisf and __floatdidf are not correctly generated, causing stack overflow due to infinite recursion

This is the smallest piece of code needed to replicate a double fault error. The problem arises when converting timestamp, an i64 variable, to f64. If you set timestamp to be u64 before converting, then the double fault does not appear.

The DF also only arises when you print timestamp to the screen, or if you do operations with it and print those variables, so the problem may be with the println! macro

#![no_std]
#![feature(slice_concat_ext)]

#[macro_use] extern crate terminal_print;
extern crate http_client;

use http_client::{millis_since};


#[no_mangle]
fn main() {

    let startup_time: u64 = 9000000000000; //this can be any u64 value

    let timestamp = match millis_since(startup_time) {
                Ok(time) => time as i64,
                Err(err) => return println!("couldn't get timestamp:{}", err),
            };

    // it's during this conversion of timestamp from i64 to f64 that the double fault occurs
    println!("timestamp:{}", timestamp as f64);
    
}   

The Golden Rule of Software Development

“be clean” and “foolproof code” are just good practice that should be applied everywhere.
“Good abstraction” should be made more specific like “Nobody needs to read your code”. It is more than good abstraction.

I suggest we separate general good practice from Theseus-specific principles. I can think of two specific principles: "Handle all Errors" and "Nobody should read your code"

Delete key doesn't work in terminal

Backspace works, but not delete.

Replace VirtualAddress & PhysicalAddress type defs with new types

Currently we are using the following type defs in memory/mod.rs:

pub type VirtualAddress = usize;
pub type PhysicalAddress = usize;

These typedefs can lead to type confusion, which won't allow the compiler to check for a VirtualAddress being used where a PhysicalAddress should be used, and vice versa.

Instead, we should use type wrappers (as provided by the x86_64 crate), to strictly enforce when a VirtualAddress or PhysicalAddress is needed. They look like this:

pub struct VirtualAddress(pub usize);
pub struct PhysicalAddress(pub usize);

println! macros cause deadlock due to locking in VGA buffer

Originally reported by: Kevin Boos (Bitbucket: kevinaboos, GitHub: kevinaboos)

---

Instead of locking the vga buffer on the main thread and writing directly into it, we need to use a different thread for the vga buffer with a queue that displays data on the screen. This is basically a console driver... so we could combine it with that.

---

• Bitbucket: https://bitbucket.org/kevinaboos/theseus-os/issue/5

Use Ranges to represent PageIter and FrameIter

PageIter should be PageRange, same for frames. It's clearer, and we can use the underlying RangeInclusive type rather than our own custom start/end/contains logic.

They can still implement Iterator though, just like Ranges do.

Batch TLB shootdowns in `remap()` and `unmap()`

Right now we send separate TLB shootdown IPIs (actually NMIs) for every individual Page in the entire range of Pages covered by a MappedPages object.

This is extremely slow and stupid; instead, we should batch them together and send a single IPI for a whole range of VirtualAddresses that should be flushed from the TLB.

Killing the OS (with Ctrl-D) causes a page fault

It should smoothly shut down the OS, not hang in the page fault handler. Probably something to do with the logger cleanup routine (or maybe something in rust_main() finally going out of scope) but who knows for sure.

Debugging support sysfs-style for arbitrary crate variables

Expose all static crate variables as sysfs-style entries, to aid in debugging and analysis of program states.
Inspired by Cristiano Giuffrida's PeekFS (which I cannot find any public references to).

We can parse the debug_ sections of each object file and then use the DWARF DIE trees to get the string name of the variable symbol and it's type. And based on the sysfs directory we're in, we'll know which crate to look it.

Speaking of, the sysfs root will have a namespaces directory, which will have all of the CrateNamespaces, and each CrateNamespace has a list of LoadedCrates. In each LoadedCrate, there is a list of LoadedSections, each of which will comprise another directory.

Finally, in each LoadedCrate, there will be a directory for every static variable defined in that crate. Within each variable's directory, there will be a debug__ and display__ file that you can cat to invoke the Display or Debug traits implementations for that variable type.

Stretch goal: show the variable type's struct members as subdirectories in the variable's directory, so we can recurse into each member to further explore the struct. This should be do-able since we'll know what the type of the struct is; though we might not be able to read the struct definition directly (unless it's in the Dwarf debug sections somewhere?).