mikong / hanbaiki

A simple key-value store written in Rust. It uses the Redis Serialization Protocol.

Nothing to see here

puts books.last_updated_on.iso8601
# 2023-10-30
puts books.currently_reading.last(5).join("\n")
# Designing Data-Intensive Applications by M. Kleppmann
# Practical Object-Oriented Design in Ruby by S. Metz
# Building Git by J. Coglan
# Microservices Patterns by C. Richardson
# Database Internals by A. Petrov

hanbaiki's Issues

Explore Immutable Data Structures

Is your feature request related to a problem? Please describe.
Performance. The project currently uses the standard library HashMap. When the project becomes more stable and have a better set of benchmark tests, we should explore alternative data structures that might provide better performance.

Describe the solution you'd like
Use the im crate and compare its performance with the standard library HashMap.

Describe alternatives you've considered
Alternatively, we can make smaller changes first, such as using a faster hashing algorithm for the standard library HashMap. We'll have to consider the security implications of using other hashing functions though.

Additional context
The Feature Request issue template was recently added to this project, and I thought of trying it out with a feature that I've listed in my internal notes for this project.

Cross compiling for Windows

Environment: x86_64-apple-darwin
Target: x86_64-pc-windows-gnu

Note: Our target is x86_64-pc-windows-gnu instead of x86_64-pc-windows-msvc. The MSVC ABI is only needed if we want to interoperate with software produced by Visual Studio. It also requires Visual Studio or Visual C++ Build Tools.

We encounter the following error messages as described in rust-lang/rust#48272:

undefined reference to `__onexitend'
undefined reference to `__onexitbegin'

That issue in rust-lang/rust was closed after they added back the downloading of rust-mingw, but it doesn't help our case. We have our own MinGW and removing the linker from ~/.cargo/config will result in a linker error as described in rust-lang/rust#33465:

ld: unknown option: --enable-long-section-names
clang: error: linker command failed with exit code 1

To Reproduce

1. Install MinGW:

$ brew install mingw-w64
# Installs 6.0.0 as of writing

2. Install cross compiled standard library:

$ rustup target add x86_64-pc-windows-gnu

3. Update ~/.cargo/config:

[target.x86_64-pc-windows-gnu]
linker = "x86_64-w64-mingw32-gcc"

4. Cross compile with cargo:

$ cargo build --target=x86_64-pc-windows-gnu
$ cargo build --release --target=x86_64-pc-windows-gnu

Command not recognized error when using redis-rb client

When using redis-rb (redis gem 4.0.2), commands are not recognized:

irb(main):001:0> require 'redis'
=> true
irb(main):002:0> redis = Redis.new(host: "127.0.0.1", port: 6363)
=> #<Redis client v4.0.2 for redis://127.0.0.1:6363/0>
irb(main):003:0> redis.set("hello", "world")
Redis::CommandError: ERROR: Command not recognized

Debug output of command from redis-rb in hanbaiki server:

Array([BulkString("set"), BulkString("hello"), BulkString("world")])

We need to support lowercase commands to support client libraries.

One `stream.read` may receive more than a full message

It is possible that a single stream.read may receive not just the full message but also part or the whole of the next message. Message framing in the RESP reader should take this into account, making sure the current message won't include any part of the next message and processing the next message should still be able to start at the right part.

Testing with TcpStream

The RESP reader doesn't have any automated tests due to the TcpStream parameter in its functions.

I'm currently considering 3 options:

1. Use TcpStream argument in the test. This involves actually running a server and clients.

2. Use a mock stream struct that implements the Read trait. The function signatures will have to be changed to accept any type that implements Read.

3. Create a custom trait that a mock stream struct and TcpStream will implement. The function signatures will still need to be changed but it doesn't open it up to any Read type.

Limitations of a simple state machine for Redis protocol deserialization

The initial implementation of message framing a bulk string data uses a simple state machine. However, a simple state machine isn't sufficient for handling array data. An array can have elements of any type including bulk strings and arrays. When reading more data from the TCP stream while trying to build an element of an array, the state machine should have a memory of where it is in the building of the array or sub-array so it can continue where it left off after completing an element.

Before we can implement the message framing of array data, we need to:

• Update the RESP reader to use a stack for the simpler types first: simple string, error, integer and bulk string.
• Move the stream.read loop outside of the functions for handling the individual types. Each function can just return Ok(None) to indicate that it's not complete and needs more data to process the frame.