rdaum / moor Goto Github PK
View Code? Open in Web Editor NEWA system for building shared, programmable, online spaces. Compatible with LambdaMOO.
License: GNU General Public License v3.0
A system for building shared, programmable, online spaces. Compatible with LambdaMOO.
License: GNU General Public License v3.0
cache this or index it better
moor/crates/db/src/odb/rb_worldstate.rs
Line 112 in a951a75
}
fn get_players(&self) -> Result<ObjSet, WorldStateError> {
// TODO(rdaum): this is going to be not-at-all performant in the long run, and we'll need a way to
// cache this or index it better
get_all_object_keys_matching(
&self.tx,
If command matching fails, the server should call huh
on the current player object, if that verb exists.
LambdaCore and others use this to implement "feature" objects.
ever append.
moor/crates/daemon/src/rpc_session.rs
Line 34 in a951a75
client_id: Uuid,
rpc_server: Arc<RpcServer>,
player: Objid,
// TODO(rdaum): manage this buffer better -- e.g. if it grows too big, for long-running tasks, etc. it
// should be mmap'd to disk or something.
// TODO(rdaum): We could also use Boxcar or other append-only lockless container for this, since we only
// ever append.
session_buffer: Mutex<Vec<(Objid, NarrativeEvent)>>,
}
Now that I've pretty much de-cloaked on this project and there's some external people watching, it's probably best to document the decisions, vision, and architecture a bit.
Basically a couple diagrams breaking down the layers and components, thread and process and network boundaries, and a description of the why and how.
moor/crates/compiler/src/decompile.rs
Line 300 in a951a75
));
}
// Same as above, but with id.
// TODO(rdaum): we may want to consider collapsing these two VM opcodes
Op::WhileId {
id,
end_label: loop_end_label,
7
Equality/inequality
moor/crates/compiler/src/parse.rs
Line 190 in a951a75
.op(Op::infix(Rule::lor, Assoc::Left))
// 11. Logical and.
.op(Op::infix(Rule::land, Assoc::Left))
// TODO(rdaum): bitwise operators here (| 10, ^ XOR 9, & 8) if we ever get them.
// 7
// Equality/inequality
.op(Op::infix(Rule::eq, Assoc::Left) | Op::infix(Rule::neq, Assoc::Left))
we're not dependent on this.
moor/crates/daemon/src/rpc_server.rs
Line 448 in a951a75
args: Vec<String>,
attach: bool,
) -> Result<RpcResponse, RpcRequestError> {
// TODO(rdaum): change result of login to return this information, rather than just Objid, so
// we're not dependent on this.
let connect_type = if args.first() == Some(&"create".to_string()) {
ConnectType::Created
Line 99 in a951a75
fn get_verbs(&self, obj: Objid) -> Result<VerbDefs, WorldStateError>;
/// Get the binary of the given verb.
// TODO(rdaum): "binaries" returned from the db should be SliceRefs, not Vecs.
fn get_verb_binary(&self, obj: Objid, uuid: Uuid) -> Result<Vec<u8>, WorldStateError>;
/// Find & get the verb with the given name on the given object.
moor/crates/daemon/src/rpc_server.rs
Line 593 in a951a75
};
// Try to submit to do_command as a verb call first and only parse_command after that fails.
// TODO(rdaum): fold this functionality into Task.
let arguments = parse_into_words(command.as_str());
if let Ok(task_id) = self.clone().scheduler.submit_verb_task(
Performs a synchronous blocking read on the network connection. Will require some thought on how to do this right.
moor itself. And really only something with a WorldState can do that. So it's not
enough to have validated the auth token here, we will need to pepper the scheduler/task
code with checks to make sure that the player objid is valid before letting it go
moor/crates/daemon/src/rpc_server.rs
Line 997 in a951a75
}
}
// TODO(rdaum): we will need to verify that the player object id inside the token is valid inside
// moor itself. And really only something with a WorldState can do that. So it's not
// enough to have validated the auth token here, we will need to pepper the scheduler/task
// code with checks to make sure that the player objid is valid before letting it go
Core needs to be given a chance to parse the command before the builtin-parser gets to it.
moor/crates/compiler/src/unparse.rs
Line 38 in a951a75
// directly on http://en.cppreference.com/w/cpp/language/operator_precedence
// Should be kept in sync with the pratt parser in `parse.rs`
// Starting from lowest to highest precedence...
// TODO(rdaum): drive Pratt and this from one common precedence table.
let cpp_ref_prep = match self {
Expr::Scatter(_, _) | Expr::Assign { .. } => 14,
Expr::Cond { .. } => 13,
For support of e.g. MCP clients.
The tasks in the scheduler are currently not persisted. Problem because:
daemon
for upgrades, etc. will kill background tasks without recovery on restart, interrupting the user experience.scheduler.rs
and task.rs
will need some reworking to synchronize their state out to WorldState
, which will need new interfaces for this.
And for compatibility, we might need to support the textdump'd background task set from LambdaMOO textdump imports.
Right now it's task.rs
that does the logic with this.
With the command interpreter loop there's 2 separate transactions: one to match for parsing the command, and another to execute the command. And with support for $do_command
there's a third, and once I fix missing huh
handling, that's a fourth! Not a good situation. 3/4 of these transactions all occur within the scheduler, and so could be all subsumed under that one place.
I played with optionally passing an already-existing transaction into the task, but the problem with this is that to do the :huh support the transaction will already have been committed when the task completes with error.
So my proposal is to go through and clean things up potentially so that tasks always begin and end with an open transaction, and it's Scheduler that is responsible for the lifecycle of transactions -- passing ownership of them to the task, and getting them back again at completion.
/ the line number spans in the bytecode.
moor/crates/compiler/src/ast.rs
Line 191 in a951a75
/// Note that this is not necessarily the same as the line number that will be reported into
/// codegen, and may not correspond to what shows as a result of `unparse`; that line number
/// is derived from the AST, not the parser.
/// TODO(rdaum): I may or may not keep this field around.
pub parser_line_no: usize,
/// This line number is generated during a second pass over the tree, and is used to generate
/// the line number spans in the bytecode.
Hello! I have some questions about the specific way docker is being used in this project. I'd like some help understanding the purpose/usecase of having the whole project recompile itself with cargo watch
while watching for changes.
In my experience with docker, this tends to make the image more bloated and run slower.
Need a way to create a dump of the world state that can be used independent of the binary database, most especially useful for migration between versions of the DB format while things are still fairly unstable.
Does not necessarily have to be backwards compatible with LambdaMOO textdump
For moor
, these would be, most meaningfully:
memory_usage
: Just grab process RSS for the daemon PID. Simpler and more accurate than manually summing up the DB buffer pools etc.
db_disk_size
: This would be the the total byte count of all tuples in the DB. For now since I'm not yet LRU-paging out tuples, that's the exact same as the usage count of the primary buffer pool. This gets trickier to calc after paging gets supported.
Right now task.rs
just aborts on commit-conflict instead of re-running the transaction. Need to get around to the retry logic.
Many games, and maybe all of them, require the ability to have verbs that open TCP sockets as listeners or to establish outgoing network connections at the verb level.
I understand this is contrary to your vision but it will present a significant if not insurmountable barrier of entry in limiting adoption of your server.
Games like Torchship use sockets to talk to external simulation layer services like our login server which uses TCP and not REST to communicate. Similarly, our space simulation server which is written in Rust also uses TCP to communicate with the game over a custom messaging protocol I wrote for that specific implementation. Which would be unideal to move to the server layer because it's iterated on so frequently it would result in significant service downtimes or stability issues during testing, which verbs offer in safety on the VM.
Other games use those connections to talk to things like mail forwarding agents.
LambdaMOO:
=> "Thu Sep 28 13:53:54 2023 PDT"
vs moor:
=> "Thu, 28 Sep 2023 20:53:39 +0000"
To be compliant with the MOO spec for ctime, this should a) return the timezone descriptor instead of offset and b) use the local timezone, not UTC.
Symptoms are that some internal core functions that parse the ctime don't work:
@shutdown
Do you really want to shut down the server in 2 minutes? [Enter `yes' or `no']
yes
#39:english_time (line 22): Range error
... called from #51:@shutdown (this == #2) (line 31)
(End of traceback)
should be mmap'd to disk or something.
moor/crates/daemon/src/rpc_session.rs
Line 32 in a951a75
client_id: Uuid,
rpc_server: Arc<RpcServer>,
player: Objid,
// TODO(rdaum): manage this buffer better -- e.g. if it grows too big, for long-running tasks, etc. it
// should be mmap'd to disk or something.
// TODO(rdaum): We could also use Boxcar or other append-only lockless container for this, since we only
// ever append.
session_buffer: Mutex<Vec<(Objid, NarrativeEvent)>>,
}
so we can check for multiple flags at once.
moor/crates/db/src/db_worldstate.rs
Line 99 in a951a75
) -> Result<Objid, WorldStateError> {
if parent != NOTHING {
let (flags, parent_owner) = (self.flags_of(parent)?, self.owner_of(parent)?);
// TODO check_object_allows should take a BitEnum arg for `allows`
// so we can check for multiple flags at once.
self.perms(perms)?
.check_object_allows(parent_owner, flags, ObjFlag::Read)?;
self.perms(perms)?
instead of concatenation with MkSingletonList.
moor/crates/compiler/src/codegen.rs
Line 737 in a951a75
}
fn generate_arg_list(&mut self, args: &Vec<Arg>) -> Result<(), CompileError> {
// TODO(rdaum): Check recursion down to see if all literal values, and if so reduce to a Imm value with the full list,
// instead of concatenation with MkSingletonList.
if args.is_empty() {
self.emit(Op::ImmEmptyList);
moor/crates/compiler/src/program.rs
Line 101 in a951a75
writeln!(f, "V{}: {}", i, v)?;
}
// TODO(rdaum): print fork vectors
// Display main vector (program); opcodes are indexed by their offset
for (i, op) in self.main_vector.iter().enumerate() {
Not sure why it was being called this way, but 'tis caused a panic:
2023-09-25T11:50:18.029710411Z thread 'tokio-runtime-worker' panicked at 'overflow when adding duration to instant', library/std/src/time.rs:584:31
argument was: -1090650497
Not sure what I'm missing in the command matching cycle, but commands on feature objects don't seem to be getting matched.
Hi, I'm Volund of the MUSH Community. Been looking into MOO on and off and I found this thanks to reddit... well, really I found it thanks to ToastStunt community talking about it and have a few things to say and suggest....
SPECTACULAR JOB at getting this to where it is. I was thinking about doing something similar if I had infinite time - which I don't - and now I don't have to, I can learn from and possibly help out with this one instead. I can't wait to tear into this and learn from it. :D And see where it's going.
Suggest a more careful rewording of the readme section that mentions why you'd use Moor instead of ToastStunt. It can be read to imply that ToastStunt doesn't have an active developer community, and it does.
Please create a Discord or similar chat-server for the Moor community to meet up on and grow around. Evennia has one, RhostMUSH has one, ToastStunt has one, I think AresMUSH has one... projects like this need lots of cultivation from a community to grow and thrive, so let's get cracking. ๐
It would be nice to try providing an alternative to the rocksdb
db implementation that is based around a relational database.
When I first started, I used SQLite for this -- and in fact used SQL's transitive closure / recursive joins feature for inherited verb resolution even -- but it got awkward as I was hacking heavily, so it got pulled out.
But I'd like to maybe revisit this, with a Postgres (and maybe other) backend, In all likelihood it would need a caching layer in front, but in fact the Rocks stuff really needs this, too. (And the challenge at the caching layer remains transactional consistency, but that's a whole other kettle of fish).
In the end I'm interested in:
Requires at least estimating object sizes and tracking them per user / owner.
Need some hooks on the WorldState
interface.
LambdaMOO cores' connect
function uses a variant of the Unix crypt
function, ala old Unix passwd files, and stores the passwords in-db.
For compatibility with existing cores (and to make servicing existing telnet
clients feasible), it's necessary to keep this process for now. But this isn't going to cut it as a security story.
Some kind of SSO story & auth token is going to be needed. For the latter, one option is JWT
, another is PASETO
, which sounds more promising.
Then the websocket connection (and any HTTP sessions generally) will need to be authenticated with this token before client connections can proceed.
The long-term future for the telnet
modality will need to be thought about in this context.
Not a requirement for 1.0 (LambdaMOO compatible) release.
These are library code now, and shouldn't be using anyhow, really. Most places are using specific Error
trait implementations via thiserror
but not all.
the list of functions for the `function_info` built-in right now. It could be used for
validating arguments, and could be part of the registration process for the actual builtin
implementations.
moor/crates/compiler/src/builtins.rs
Line 48 in a951a75
}
// Originally generated using ./generate_bf_list.py
// TODO(rdaum): this list is inconsistently used, and falls out of date. It's only used for generating
// the list of functions for the `function_info` built-in right now. It could be used for
// validating arguments, and could be part of the registration process for the actual builtin
// implementations.
but we have no real way to signal to this loop that it should newline for
cleanliness. Need to figure out something for this.
moor/crates/console-host/src/main.rs
Line 294 in a951a75
let mut rl = DefaultEditor::new().unwrap();
loop {
// TODO(rdaum): unprovoked output from the narrative stream screws up the prompt midstream,
// but we have no real way to signal to this loop that it should newline for
// cleanliness. Need to figure out something for this.
let input_request_id = input_request_id.lock().unwrap().take();
moor/crates/compiler/src/opcode.rs
Line 116 in a951a75
use crate::{Label, Name, Offset};
/// Verify we don't go over our 16 byte budget for opcodes.
// TODO(rdaum): This is still rather bloated.
#[test]
fn size_opcode() {
use crate::opcode::Op;
So host connections can remain live while the daemon is restarted for upgrades / bug fixes.
moor/crates/compiler/src/program.rs
Line 42 in a951a75
pub fork_vectors: Vec<Vec<Op>>,
/// As each statement is pushed, the line number is recorded, along with its offset in the main
/// vector.
/// TODO(rdaum): fork vector offsets... Have to think about that one.
pub line_number_spans: Vec<(usize, usize)>,
}
Needs to dispatch properly to user_created
rather than user_connected
, and send "** Created " to the user instead of " Connected **".
LambdaMOO does this by comparing max_object()
before and after login, but we will have to find a better way because that is bad in all sorts of ways.
>@list $mcp:session_for
Verb 9 on #213 -- E_RANGE
>;verbs($mcp)
=> {"create_session", "destroy_session", "initialize_connection", "finalize_connection", "parse_version", "compare_version_range", "compare_version", "unparse_version", "session_for", "user_created", "user_disconnected", "do_out_of_band_command", "package_name", "wait_for_package", "nominate_for_core", "handles_package", "init_for_core"}
>;verb_code($mcp, "session_for")
=> E_INVARG (E_INVARG)
Either in decompilation, or in verb resolution.
moor/crates/compiler/src/decompile.rs
Line 657 in a951a75
}
// Decompile the body.
// Means decompiling until we hit EndExcept, so scan forward for that.
// TODO(rdaum): make sure that this doesn't fail with nested try/excepts?
let (body, end_except) =
self.decompile_statements_until_match(|_, o| matches!(o, Op::EndExcept(_)))?;
let Op::EndExcept(end_label) = end_except else {
This is both untested and mostly unimplemented. Forked tasks are unlikely to be currently doing any of the following properly:
@list
/ view of the verb codedisassemble
builtin outputI would like a suite of tests to check for correctness of concurrent behaviour; linearizability, consistency, serialization conflicts, etc.
Still very vague in my mind, but probably a suite of Jepsen tests and/or Stateright (https://github.com/stateright/stateright).
Essentially, instantiating kernel + db with a predefined simple set of objects and then perform concurrent evaluations of mutations to provoke serialization conflicts, retries, and verify correct behaviour under concurrent load.
This will be essential in the short run run for a) finishing the transactional layer (retries on conflict) and b) making sure the Rocks etc layer works as hopes and in the medium/long run for swapping in alternative world state storage models.
Similar to #21 , I would like to see how this would perform and work out in code. I played with this in an earlier sketch of things, but didn't go far with it. In theory everything that is done via rocks
can be done with FDB, but with distribution baked in (at the cost of performance and complexity)
They share a similar limitation though in that they're both optimized for a single-writer (per relation) but multiple reader workflow. Which may not describe the characteristics of what a transaction load is actually like in a working system.
moor/crates/daemon/src/rpc_server.rs
Line 663 in a951a75
return Err(RpcRequestError::InternalError(e.to_string()));
}
// TODO(rdaum): do we need a new response for this? Maybe just a "Thanks"?
Ok(RpcResponse::InputThanks)
}
I may not be able to do with the current structure. See if I need to
moor/crates/compiler/src/decompile.rs
Line 825 in a951a75
}
Op::EndCatch(_) | Op::Continue | Op::EndExcept(_) | Op::EndFinally => {
// Early exit; main logic is in TRY_FINALLY or CATCH etc case, above
// TODO(rdaum): MOO has "return ptr - 2;" -- doing something with the iteration, that
// I may not be able to do with the current structure. See if I need to
unreachable!("should have been handled other decompilation branches")
}
Problem in stack unwinding related to error Catch handling.
Found while running:
help $scheduler
Doesn't happen for other :help_msg
invocations it seems. Haven't tracked down what verb flow is causing this yet.
daemon log:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2023-09-24T21:59:30.157150364Z 2023-09-24T21:59:30.156994Z WARN ThreadId(15) do_process: crates/kernel/src/tasks/scheduler.rs:787: Task sys-errored task=69 error=Disconnected
2023-09-24T22:00:42.543329425Z thread 'tokio-runtime-worker' panicked at 'TODO: CatchLabel where we didn't expect it...', crates/kernel/src/vm/vm_unwind.rs:367:25
2023-09-24T22:00:42.543368941Z stack backtrace:
2023-09-24T22:00:42.543373580Z 0: rust_begin_unwind
2023-09-24T22:00:42.543376937Z at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:593:5
2023-09-24T22:00:42.543380263Z 1: core::panicking::panic_fmt
2023-09-24T22:00:42.543383389Z at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/panicking.rs:67:14
2023-09-24T22:00:42.543388429Z 2: moor_kernel::vm::vm_unwind::<impl moor_kernel::vm::VM>::unwind_stack
2023-09-24T22:00:42.543392156Z at ./crates/kernel/src/vm/vm_unwind.rs:367:25
2023-09-24T22:00:42.543410261Z 3: moor_kernel::vm::vm_execute::<impl moor_kernel::vm::VM>::exec::{{closure}}
2023-09-24T22:00:42.543413777Z at ./crates/kernel/src/vm/vm_execute.rs:502:28
2023-09-24T22:00:42.543446871Z 4: <moor_kernel::tasks::moo_vm_host::MooVmHost as moor_kernel::tasks::vm_host::VMHost<moor_kernel::vm::opcode::Program>>::exec_interpreter::{{closure}}
2023-09-24T22:00:42.543462882Z at ./crates/kernel/src/tasks/moo_vm_host.rs:182:68
If the intended owner of the new object has a property named `ownership_quota' and the value of that property is an integer, then `create()' treats that value
as a "quota". If the quota is less than or equal to zero, then the quota is considered to be exhausted and `create()' raises `E_QUOTA' instead of creating an
object. Otherwise, the quota is decremented and stored back into the `ownership_quota' property as a part of the creation of the new object.
moor/crates/db/src/db_worldstate.rs
Line 109 in a951a75
let owner = (owner != NOTHING).then_some(owner);
// TODO(rdaum): ownership_quota support
// If the intended owner of the new object has a property named `ownership_quota' and the value of that property is an integer, then `create()' treats that value
// as a "quota". If the quota is less than or equal to zero, then the quota is considered to be exhausted and `create()' raises `E_QUOTA' instead of creating an
// object. Otherwise, the quota is decremented and stored back into the `ownership_quota' property as a part of the creation of the new object.
let attrs = ObjAttrs {
owner,
name: None,
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.