pedrolamarao / metal

Metal is a research project on "bare metal" software engineering.

project metal

summary

Welcome to project Metal!

Metal is a research project on "bare metal" software engineering.

For more information on this project’s mission, ambitions and aims, see GOAL.

prepare

Metal is built with an LLVM based toolset.

JDK 17 or newer is required in either environment JAVA_HOME or PATH.

The following tools are required in gradle property metal.path or environment PATH.

• CMake 3

• GNU grub 2

• LLVM 16 or newer

• Ninja

• QEMU 6.0 up to 7.2

For more information, see TOOLS.

build

Metal is built by Gradle Metal.

To assemble everything for host: ./gradlew build This task shall compile and run modules and unit test programs.

To assemble everything for target i686-elf: ./gradlew build -Pmetal.target=i686-elf This task shall compile and run modules and QEMU emulated system test programs.

Currently, only targets host and i686-elf are supported. We expect to include target x86_64-elf very soon.

To clean everything: ./gradlew clean

verify

Metal is verified by automated tests.

For more information, see TEST.

The check task runs all verifications, and the test task runs test programs.

metal's Issues

gradle-asm compile failure does not fail the entire build

Assemble failure is not causing the build to fail. The component is still linked successfully. Dependent projects shall fail to link for unexpected reasons.

Run test suites which assert full machine state

Enhance everything so that Gradle can run test suites with test cases capable of asserting machine status. This implies that, after termination, the test runner must obtain full machine status from the emulator, and compare that to expected values.

Create PS/2 component

Create VGA component

Make it comfortable to define hundreds of test cases

Currently, adding a test case requires adding a new Gradle project. True, subproject configuration eases this process. However, one needs to (1) add a new Gradle project, (2) add this new project to the multiproject configuration, (3) update the IDE to perceive the new project. Adding a new test case should require a single step.

Build and install tools inside the tree

Currently, tools scripts have a weird directory setup based on $HOME.
There is no need to share these tools.
Let's just install inside ${projectDir}/tools.

Demonstrate PC CMOS RTC timer

Ditto.

Try LLVM again

See https://wiki.osdev.org/LLVM_Cross-Compiler.

Run image with Gradle coordinating QEMU and GDB

Teach Gradle how to run an image by coordinating QEMU to actually run the image and GDB to debug it.

Support pure unit tests

We must support pure unit testing of components. These must target the host with a proper toolset. Then, gtest should be enough.

Update to C++ 20

Update build scripts to target C++ 20.

Demonstrate long mode interrupts with red zones

Ditto.

Generate images loadable by UEFI

Generate disk or cdrom image loadable by QEMU with OVMF.

Update to GCC 10.2

Clang is cool but for some misterious reason does not cross-compile to i386-pc-elf. Go back to GCC with version 10.2.

Create PC PIT component

Create PC CMOS RTC component

Demonstrate a PS/2 console

Ditto.

Build minimal GDB/MI java library

ditto.

Design QemuProcess driven via QEMU monitor

ditto.

Interrupt GDB

Find a way to interrupt GDB.

Explore QEMU `-kernel` without GRUB

According with documentation, QEMU is able to directly load a Multiboot kernel. This way, we could eliminate GRUB from the test equation.

Demonstrate BOOTBOOT

Explore KVM for emulation tests

Either -enable-kvm or maybe -machine type=pc-q35,accel=kvm.

Create new Gradle plugins for native components

In short, Gradle Inc. shows little interest evolving its native tools. Cross compilation is not on the roadmap. The last issue closed on Gradle native was 18 December 2019.

Fundamentally, we need library projects to provide for open sets of targets.

Suppose a fully mature x86 library. It provides for architectures x86_16, x86_32 and x86_64, for any machine, any system and any ABI. If we build x86 unit tests on x86_64-linux-gnu, Gradle must match and build the base library for x86_64-any-any-any. If we build a certain depedent application for x86-pc-none-elf, Gradle must match and build the x86 library for x86-any-any-any.

Currently, Aasgard avoids this need by requiring a x86_64-any-any-any host and lying to Gradle. We tell Gradle that libraries support targetMachines = [ machines.linux.x86, machines.linux.x86_64]. Then, all unit tests require targetMachines = [ machines.linux.x86_64] and all emulator tests require targetMachines = [ machines.linux.x86 ].

Design I/O port concept

In order to decouple interfaces such as pic and cmos from the architecture, these things are currently defined as templates on a "port" type. This allows pure unit tests. We need to specify what, exactly, is a "port".

Add `debian.sh` for preparing Debian to build tools

We have adequate scripts to build the necessary cross tools. Unfortunately, we lost all dependencies installed during development of the scripts. Also, there are some optional but useful stuff missing: gdb could benefit for expat, for example. Rerun the scripts in an empty Debian system and record the necessary dependencies in tools/debian.sh.

Run simple test case with Gradle

Enhance image running with the capacity to judge a simple test case: either the machine halts (success) or faults (failure).

aasgard.asm must contribute dependency to the unit test

Currently, aasgard.asm contributes to the main linkage but not to the unit test linkage.

Generate `grub.conf` from build configuration

Teach gradle-grub to generate grub.conf based on build configuration. The image should allow booting without discovering the program file name.

Allow test case with multiple assertions

Allow asserting multiple times in a single test case.

Sandbox for x86-16

Provide a sandbox for programming in real mode.

Run GDB to debug an image running in an external process

Teach Gradle some tasks to run GDB to debug an image running in an external process.

Create x86 LAPIC component

Demonstrate a serial terminal

Move unit testing experiment to a separate branch

The unit testing experiment is all about cross-compilation.
It requires the build system to be able to build for various targets, including the host.
Currently, we trick the build system to produce the required variants.
This is disturbing other more important experiments.
Moving to a separate branch is appropriate.

Demonstrate a VGA console

Build GRUB images with Gradle

Teach Gradle tasks for building GRUB images with grub-mkstandalone and grub-mkrescue.

Define test cases as artifacts of the main project

Currently, the test machinery requires a new Gradle project per test case. Improve this machinery such that each test case becomes an artifact of the main project -- so that "foo" test cases live inside project "foo".

Run images under QEMU with Gradle

Teach Gradle tasks capable of running GRUB images under QEMU system emulation.

Integrate ACPICA

Integrate the ACPI Component Architecture licensed as GPL.

Bind tests to check test

Run test cases with ./gradlew check.

Demonstrate initializing an ARM.

ditto. This should pressure & correct the arrangement of components and tests.

Run test suites which assert serial port I/O

Enhance image running with the capacity to judge a test case by asserting serial port I/O.

Integrate clang toolset

Ditto.

Deal with "graft syntax" in `GradleStandaloneCompile`

grub-mkstandalone embbeds files with full path in the image. To "relocate" files it is necessary to use a "graft syntaxs". Deal with this somehow in GradleStandaloneCompile. It is unusable as it currently stands.

Demonstrate PC PIT timer

Ditto.

Define test program protocol

Test programs need to communicate test results to test drivers. Define a protocol for this.

Early experiments indicate it is sufficient to define well known ELF public symbols. Writing on the test result symbol is terminating the test. Proper timeouts shall cover anomalous tests.

GDB is capable of watching for writes on well known locations. This should be sufficient for a driver to program GDB into stopping on write onto the test result locations. With the program stopped, the driver has full access to the machine state via GDB.

Demonstrate x86 LAPIC timer

ditto.

Generate gdb launcher scripts from build configuration

Teach gradle-gdb to generate launcher scripts, with gdbinit, from build configuration. This should merge from src/gdb/gdbinit.

Run cross pure tests with qemu-user

Currently, I can't run unit tests built for x86(32) architectures. My development environment is Windows Subsystem for Linux version 1 which emulates x86(64) exclusively.

Although it is a sad limitation, this brings to light the need to emulate pure tests also. We could do that with QEMU user emulation.

We need a test plugin capable of running tests in QEMU user emulation.