gschup / bevy_ggrs

Bevy plugin for the GGRS P2P rollback networking library.

Currently

Working as a Researcher at SEBx.

My Research

I was a PhD student at the KTH Royal Institute of Technology in the Division of Robotics, Perception and Learning under supervision by Jana Tumova. I was also an affiliated PhD student in the WASP program.

In my thesis, I apply methods from Formal Verification to the domain of Multi-Robot Systems by synthesizing correct-by-design strategies over Linear Time Logic specifications in a scalable manner. Strategy negotation through assumption exchange between agents, but also between agents and humans are the overarching theme of this work.

Publications

Doctoral Thesis

• G. F. Schuppe, "Assumptions in Synthesis: An Approach to Multi-Agent Planning from Spatio-Temporal Specifications," Diss. KTH Royal Institute of Technology, 2023, [pdf]

Conference and Journal Papers

• G. F. Schuppe and J. Tumova, "Multi-Agent Strategy Synthesis for LTL Specifications through Assumption Composition," 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), [doi] [pdf]

• G. F. Schuppe and J. Tumova, "Decentralized Multi-Agent Strategy Synthesis under LTLf Specifications via Exchange of Least-Limiting Advisers," 2021 International Symposium on Multi-Robot and Multi-Agent Systems (MRS), [doi] [pdf]

• G. F. Schuppe, I. Torre, I. Leite and J. Tumova, "Follow my Advice: Assume-Guarantee Approach to Task Planning with Human in the Loop,", 2023 Robotics: Science and Systems (RSS), [pdf]

• C. Pek*, G. F. Schuppe*, F. Esposito, J. Tumova, and D. Kragic, "SpaTiaL: Monitoring and Planning of Robotic Tasks Using Spatio-Temporal Logic Specifications," 2023 Autonomous Robots (AuRo). [doi] [pdf]

• W. Wang, G. F. Schuppe and J. Tumova, "Decentralized Multi-agent Coordination under MITL Specifications and Communication Constraints," 2023 31st Mediterranean Conference on Control and Automation (MED), [doi] [pdf]

• G. F. Schuppe and D. Gurov, "Soundness and Completeness of a Model-Checking Proof System for CTL," arXiv preprint arXiv:2309.05389, [doi] [pdf]

*The authors contributed equally to this work

Academic Awards

• Nominated for Best Student Paper Finalist at IEEE MRS 2021
• Recipient of the Karl Engvers Stiftelse Research Travel Grant 2020

Posters

• "Decentralized Multi-Agent Strategy Synthesis via Exchange of Least-Limiting Advisers," presented at MRS 2021, [pdf]
• "Follow my Advice: Assume-Guarantee Approach to Task Planning with Human in the Loop," presented at RSS 2023, [pdf]

Links

• Google Scholar
• Curriculum Vitae

bevy_ggrs's Issues

Sprite Component doesn't get respawned after rollback

Describe the bug
The Sprite component doesn't seem to be respawned when
despawning an entity with .despawn_recursive().
Other components like Transform and custom Components
get properly respawned. The issue results in the entity
fully functioning but being invisible (as there is no sprite).

I fully expect this to be my fault for overlooking something,
but I just can't think of anything I would be doing wrong.
Are there any known pitfalls for using Sprite?
Any issues with rollbacks with them?

To Reproduce
The setup to reproduce this rather cumbersome as you need to
despawn an entity and rollback right after. The way I consistenly
do this is by setting a high ping (>100ms) and then almost
kill one player but save him in the last second.
This results in the receiving client mispredicting and despawning
the player and then rolling back and respawning him.

Expected behavior
The Sprite should be respawned as well.

Screenshots
I don't expect anyone to try to reproduce this, so instead here is
a video showing what I mean. The perspective is from the still (orange) player.
The sprite shows normally on the other player (as he doesn't mispredict his own input).

Desktop

• OS: Linux (Ubuntu 16 based)

Additional context
Note that I am also using matchbox in my game and it might
be caused by matchbox instead of bevy_ggrs, however I believe it is a rollback
issue and so I suspect the bug to be in bevy_ggrs
(if there is a bug at all, totally possible that I am just missing something).

Again, I am not asking for anyone to do my work and debug my own game, I am just wondering if there are any known issues with the Sprite component.

Convenient way to end a session

Is your feature request related to a problem? Please describe.
When a player joins or leaves a server, we must remake the P2P session with the appropriate amount of people (gschup/ggrs#23). In bevy_ggrs v0.13, this worked easily by just removing the Session resource before creating a new one:

commands.remove_resource::<Session<Config>>();

In v0.14, we must also reset the GgrsTime resource. Otherwise the app will panic once synchronization is completed for the 2nd time in an app's lifetime:

commands.insert_resource(Time::new_with(GgrsTime::default()));

I also noticed that the new resources LocalInputs and PlayerInputs are not removed or reset, but this doesn't appear to cause any issues for me at the moment.

Describe the solution you'd like
A convenient way to end a session. Something like:

commands.end_session::<Config>();

Describe alternatives you've considered
It's just a few lines of code for now, so this is only for convenience. Though it certainly caused me a headache to diagnose :)

Usability with non trivial reflectable types

I wanted to integrate a fixed point library to ease deterministic computations, but I face a big wall since I16F16 does not implement Reflect and the implementation is far from trivial

What would be the course of action to be able to use such structs easily with register_rollback_type?

use fixed::{traits::Fixed, types::I16F16};

#[derive(Default)]
pub struct Velocity {
    pub x: I16F16,
    pub y: I16F16,
}

Throws

.register_rollback_type::<Vector2D>();
^^^^^^^^^^^^^^^^^^^^^^ the trait `bevy::prelude::Reflect` is not implemented for `Velocity`

[Question] How to sync elapsed time since startup among peers?

I'm writing a multiplayer game with ships that are positioned on an ocean with rolling waves. The shape of the waves in turn is dependent on time.elapsed_seconds().

So in order to have a consistent game state, I need to sync the elapsed time since startup among peers. Is that possible using bevy_ggrs (and matchbox), and if so how would you go about doing that? Thanks!

When loading, correctly remove components added after saving

Components added to existing entities after saving are currently not be removed when loading.

In order for the loading and saving functionality to be correct, this should be fixed.

Trouble implementing the Config trait

Describe the bug
I'm having trouble implementing Config and wonder if something unusual is being required by this trait or some of it's dependencies. Maybe the bytemuck version fixing, though I don't know for sure.

error[E0599]: the function or associated item `new` exists for struct `GGRSPlugin<GGRSConfig>`, but its trait bounds were not satisfied
   --> src/main.rs:79:31
    |
63  | pub struct GGRSConfig;
    | ---------------------- doesn't satisfy `GGRSConfig: ggrs::Config`
...
79  |     GGRSPlugin::<GGRSConfig>::new()
    |                               ^^^ function or associated item cannot be called on `GGRSPlugin<GGRSConfig>` due to unsatisfied trait bounds
    |
    = note: the following trait bounds were not satisfied:
            `GGRSConfig: ggrs::Config`
note: the following trait must be implemented
   --> /Users/paul/.cargo/git/checkouts/ggrs-151f8bd9eb75dc5d/37b83c3/src/lib.rs:178:1
    |
178 | / pub trait Config: 'static + Send + Sync {
179 | |     /// The input type for a session. This is the only game-related data
180 | |     /// transmitted over the network.
181 | |     ///
...   |
193 | |     type Address: Clone + PartialEq + Eq + Hash + Send + Sync;
194 | | }
    | |_^

error[E0277]: the trait bound `GGRSConfig: ggrs::Config` is not satisfied
  --> src/main.rs:79:5
   |
79 |     GGRSPlugin::<GGRSConfig>::new()
   |     ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `ggrs::Config` is not implemented for `GGRSConfig`
   |
note: required by a bound in `GGRSPlugin`
  --> /Users/paul/.cargo/git/checkouts/bevy_ggrs-8ac41541d9ef4c9f/928376d/src/lib.rs:73:26
   |
73 | pub struct GGRSPlugin<T: Config + Send + Sync> {
   |                          ^^^^^^ required by this bound in `GGRSPlugin`

To Reproduce
Steps to reproduce the behavior:

1. Update bevy_ggrs_demo to use bevy 0.7, and update ggrs and bevy_ggrs to git based versions.
2. Try to run the project with cargo run

Updated Cargo.toml in the bevy_ggrs_demo:

[package]
name = "bevy_ggrs_demo"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
bevy_asset_loader = "0.8"
bevy = "0.7"
bytemuck = {version="1.7.3", features= ["derive"]}
ggrs = { git = "https://github.com/gschup/ggrs", features=["sync-send"], rev="37b83c3478114e2876c133f255bc00b29c6af796" }
bevy_ggrs = {git = "https://github.com/gschup/bevy_ggrs", rev="928376df6b37f391ef6593e0355ba5a737ec1492"}
matchbox_socket = { git = "https://github.com/johanhelsing/matchbox", features = ["ggrs-socket"], rev="50c1e69e9f1c0f1e07e0ffd5161db0ce3f9267b5" }
log = "0.4"

[target.'cfg(target_arch = "wasm32")'.dependencies]
ggrs = { git = "https://github.com/gschup/ggrs", features=["sync-send", "wasm-bindgen"], rev="37b83c3478114e2876c133f255bc00b29c6af796" }
bevy_ggrs = {git = "https://github.com/gschup/bevy_ggrs", features=["wasm-bindgen"], rev="928376df6b37f391ef6593e0355ba5a737ec1492"}
bevy-web-resizer = "0.1.0"
web-sys = "0.3"

Expected behavior
Errors only related to the bevy 0.6 -> 0.7 upgrade.

Desktop (please complete the following information):

• OS: Only tested w/ an M1 Macbook.

Avoid triggering `Added` and `Changed` queries on all rollbacks?

When a rollback is triggered, bevy_ggrs restores a saved world snapshot. It updates components by first removing them, and then adding them again. This triggers all queries with Added<Component> or Changed<Component> in it, regardless of whether the component was already there and whether it changed.

bevy_ggrs/src/world_snapshot.rs

Line 177 in b730a0b

reflect_component.remove(world, entity);

Sometimes systems have queries that react to other components being added or changed, triggering changes on every rollback. Sometimes, this just causes a small performance hit, other times, it leads to buggy behavior because setup code runs multiple times.

Describe the solution you'd like
Perhaps it's possible to only update the component if it's different?

Additional context
In regular game code, this is usually quite easy to fix or work around, but sometimes the affected code belongs to 3rd party crates, which are a lot harder to solve.

Integration Tests incorrect

I deleted the integration tests I previously merged. While I like the idea of having more testing not only on ggrs, but also the plugin, these tests did not correctly test what should happen. This should be fixed at some point.

3d example, and other questions

This is less a feature request an more a request for clarification on the documentation/examples side, that might help steer people in the right direction.

Is your feature request related to a problem? Please describe.
I am evaluating GGRS for a bevy 3d coop game.
Things that make me interested are

1. it is the only p2p multiplayer framework for bevy (perhaps someone can correct me)
2. deterministic rollback feels like the correct solution

However there are things that concern me:

1. All examples I have seen are 2d, use a single int for exchanging inputs.

• how would a 3d game even represent mouse movement (or do we treat the view vector itself as an input?)
• what about games with complex context dependant ui (minecraft crafting table, lets say) . I can maybe imagine creating an input struct with all possible actions in the game a player can take. But this breaks alot of the modularity of ecs, is there a better way?
• the input struct seems to require Pod, which I have a hard time wrapping my head around the implications of.

2. Even very simple examples have desync issue, that are tricky to fix. ex: https://johanhelsing.studio/posts/extreme-bevy-desync-detection

• Intuition warns me this might become unmanageable in a more complex game.
• Ggrs has tools for desync detection, but none for recovery. I do not understand the model well enough to know whether an out of band system for resyncing is even possible (would ggrs play nice).

3. All the example I have seen freeze the game until both characters connect. What about games that might have dynamic players, or dynamic NPCs?

• how hard would it be to get ggrs to play nice with players that spawn in as they load.
• how would a game using ggrs handle NPCs spawned via RNG

Describe the solution you'd like
A 3d "fps" example (ideally with physics). Players are just capsules and if you click on another player (determined by raycast) that players color changes or something.

Describe alternatives you've considered
Simply having a discussion here over some of these questions and perhaps adding "who is this for" section of the README.
I am still unsure if ggrs is viable for my usecase.

Upgrade for Bevy 0.9

Bump Bevy to 0.9 Cargo.toml and fix porting issues

Checksums are not portable

Describe the bug

When running bevy_ggrs on two different architectures (in my case these were 64bit x86 and 32bit WASM) false desyncs can occur because the checksum code is not portable. Several reasons contribute to this:

1. bevy::utils::FixedState is used for calculating checksums which uses the non-portable aHash crate, meaning that two machines can give two different checksums from the same number. Quoting the aHash crate repo:

Because it does not have a fixed standard, different computers or computers on different versions of the code will observe different  
hash values. As such, aHash is not recommended for use other than in-memory maps. Specifically, aHash is not intended for network  
use or in applications which persist hashed values. (In these cases HighwayHash would be a better choice)

2. Checksums are sometimes calculated from usize, for example the return of a .len() call. Even with a portable hasher, from the same source usize number you would get different hashes on 32bit and 64bit machines, as this type is 32 bit and 64 bit wide, respectively.

I resolved these issues (or at least up to what my game needed) in a fork branch. Here, I did the following:

1. Used a forked version of a portable hasher, SeaHash. The only reason for forking was to add panics when trying to hash usize/isize variables, as a prevention measure.
2. Limited the use of usize/isize types in bevy_ggrs, usually by using or casting to u64.

Expected behavior

I expect the checksums to be the same across architectures.
I'm not yet sure what would be the best solution for usize/isize variables.

Desktop (please complete the following information):

• OS: Windows 11
• Browser: Opera

Despawned entities are not restored

Describe the bug

When entities marked with .add_rollback() are despawned, e.g. with commands.despawn_recursive(entity), they are not properly restored.

To Reproduce
Steps to reproduce the behavior:

1. check out the johanhelsing:respawn-entity-test branch, e3a4aa6
2. env RUST_LOG=trace cargo run --example respawn_entity
3. Observe desync on frame 2 after rolling back from 3 to 1

Expected behavior
No desync

Additional context

dev/b…/examples respawn-entity-test↑1 …1 cargo run --example respawn_entity
    Finished dev [unoptimized + debuginfo] target(s) in 0.29s
     Running `C:\Users\Johan\dev\bevy_ggrs\target\debug\examples\respawn_entity.exe`
2023-11-08T22:14:40.981413Z  INFO respawn_entity: spawning player
[examples/manual_tests/respawn_entity.rs:19] &local_players.0 = [
    0,
]
2023-11-08T22:14:41.093510Z DEBUG ggrs{name="HandleRequests"}:schedule{name="SaveWorld"}: bevy_ggrs::schedule_systems: saving snapshot for frame 0
2023-11-08T22:14:41.095289Z TRACE bevy_ggrs::snapshot::entity_checksum: Rollback Entities have checksum F3DAE981A6AC4770
2023-11-08T22:14:41.095302Z TRACE bevy_ggrs::snapshot::component_checksum: Component Health has checksum 122CB4F66BB3ECE7
2023-11-08T22:14:41.095461Z TRACE bevy_ggrs::snapshot::checksum: Frame has checksum 0
2023-11-08T22:14:41.095571Z TRACE bevy_ggrs::snapshot::resource_snapshot: Snapshot Time<GgrsTime>
2023-11-08T22:14:41.095647Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 0 Children component(s)
2023-11-08T22:14:41.095589Z TRACE bevy_ggrs::snapshot::entity: Snapshot 1 entity(s)
2023-11-08T22:14:41.095578Z TRACE bevy_ggrs::snapshot::resource_snapshot: Snapshot RollbackOrdered
2023-11-08T22:14:41.095642Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 1 Health component(s)
2023-11-08T22:14:41.095662Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 0 Parent component(s)
2023-11-08T22:14:41.096155Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: advancing to frame: 1
2023-11-08T22:14:41.096981Z  INFO respawn_entity: Mut(Health(2))
2023-11-08T22:14:41.097092Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: frame 1 completed
[examples/manual_tests/respawn_entity.rs:19] &local_players.0 = [
    0,
]
2023-11-08T22:14:41.097574Z DEBUG ggrs{name="HandleRequests"}:schedule{name="SaveWorld"}: bevy_ggrs::schedule_systems: saving snapshot for frame 1
2023-11-08T22:14:41.097695Z TRACE bevy_ggrs::snapshot::entity_checksum: Rollback Entities have checksum F3DAE981A6AC4770
2023-11-08T22:14:41.097705Z TRACE bevy_ggrs::snapshot::component_checksum: Component Health has checksum 19D2EA55650F6000
2023-11-08T22:14:41.097853Z TRACE bevy_ggrs::snapshot::checksum: Frame has checksum EA0803D4C3A32770
2023-11-08T22:14:41.097954Z TRACE bevy_ggrs::snapshot::resource_snapshot: Snapshot Time<GgrsTime>
2023-11-08T22:14:41.097968Z TRACE bevy_ggrs::snapshot::resource_snapshot: Snapshot RollbackOrdered
2023-11-08T22:14:41.097970Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 0 Children component(s)
2023-11-08T22:14:41.097984Z TRACE bevy_ggrs::snapshot::entity: Snapshot 1 entity(s)
2023-11-08T22:14:41.097984Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 1 Health component(s)
2023-11-08T22:14:41.098006Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 0 Parent component(s)
2023-11-08T22:14:41.098392Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: advancing to frame: 2
2023-11-08T22:14:41.098513Z  INFO respawn_entity: Mut(Health(1))
2023-11-08T22:14:41.098608Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: frame 2 completed
[examples/manual_tests/respawn_entity.rs:19] &local_players.0 = [
    0,
]
2023-11-08T22:14:41.099072Z DEBUG ggrs{name="HandleRequests"}:schedule{name="SaveWorld"}: bevy_ggrs::schedule_systems: saving snapshot for frame 2
2023-11-08T22:14:41.099180Z TRACE bevy_ggrs::snapshot::entity_checksum: Rollback Entities have checksum F3DAE981A6AC4770
2023-11-08T22:14:41.099186Z TRACE bevy_ggrs::snapshot::component_checksum: Component Health has checksum 2CEA472AC0DF4860
2023-11-08T22:14:41.099338Z TRACE bevy_ggrs::snapshot::checksum: Frame has checksum DF30AEAB66730F10
2023-11-08T22:14:41.099435Z TRACE bevy_ggrs::snapshot::resource_snapshot: Snapshot Time<GgrsTime>
2023-11-08T22:14:41.099448Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 0 Children component(s)
2023-11-08T22:14:41.099448Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 1 Health component(s)
2023-11-08T22:14:41.099452Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 0 Parent component(s)
2023-11-08T22:14:41.099456Z TRACE bevy_ggrs::snapshot::entity: Snapshot 1 entity(s)
2023-11-08T22:14:41.099491Z TRACE bevy_ggrs::snapshot::resource_snapshot: Snapshot RollbackOrdered
2023-11-08T22:14:41.099876Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: advancing to frame: 3
2023-11-08T22:14:41.099993Z  INFO respawn_entity: Mut(Health(0))
2023-11-08T22:14:41.100059Z  INFO respawn_entity: despawning player
2023-11-08T22:14:41.100168Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ecs::world::entity_ref: Despawning entity 0v0
2023-11-08T22:14:41.100265Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: frame 3 completed
[examples/manual_tests/respawn_entity.rs:19] &local_players.0 = [
    0,
]
2023-11-08T22:14:41.100778Z DEBUG ggrs{name="HandleRequests"}:schedule{name="LoadWorld"}: bevy_ggrs::schedule_systems: restoring snapshot for frame 1
2023-11-08T22:14:41.102046Z TRACE bevy_ggrs::snapshot::entity: Rolled back 1 entity(s)
2023-11-08T22:14:41.102204Z TRACE bevy_ggrs::snapshot::component_snapshot: Rolled back 0 Children component(s)
2023-11-08T22:14:41.102209Z TRACE bevy_ggrs::snapshot::resource_snapshot: Rolled back Time<GgrsTime>
2023-11-08T22:14:41.102218Z TRACE bevy_ggrs::snapshot::component_snapshot: Rolled back 1 Health component(s)
2023-11-08T22:14:41.102235Z TRACE bevy_ggrs::snapshot::component_snapshot: Rolled back 0 Parent component(s)
2023-11-08T22:14:41.102249Z TRACE bevy_ggrs::snapshot::resource_snapshot: Rolled back RollbackOrdered
2023-11-08T22:14:41.102623Z TRACE ggrs{name="HandleRequests"}:schedule{name="LoadWorld"}: bevy_ggrs::snapshot::component_map: Mapped Parent
2023-11-08T22:14:41.102754Z TRACE ggrs{name="HandleRequests"}:schedule{name="LoadWorld"}: bevy_ggrs::snapshot::component_map: Mapped Children
2023-11-08T22:14:41.102901Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: advancing to frame: 2
2023-11-08T22:14:41.103056Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: frame 2 completed
2023-11-08T22:14:41.103158Z DEBUG ggrs{name="HandleRequests"}:schedule{name="SaveWorld"}: bevy_ggrs::schedule_systems: saving snapshot for frame 2
2023-11-08T22:14:41.103264Z TRACE bevy_ggrs::snapshot::entity_checksum: Rollback Entities have checksum EFC1504252A15E1
2023-11-08T22:14:41.103267Z TRACE bevy_ggrs::snapshot::component_checksum: Component Health has checksum 3A2A5EFCDFA21F3
2023-11-08T22:14:41.103430Z TRACE bevy_ggrs::snapshot::checksum: Frame has checksum D5EB0EBE8D03412
2023-11-08T22:14:41.103528Z TRACE bevy_ggrs::snapshot::resource_snapshot: Snapshot Time<GgrsTime>
2023-11-08T22:14:41.103539Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 0 Health component(s)
2023-11-08T22:14:41.103543Z TRACE bevy_ggrs::snapshot::entity: Snapshot 0 entity(s)
2023-11-08T22:14:41.103548Z TRACE bevy_ggrs::snapshot::resource_snapshot: Snapshot RollbackOrdered
2023-11-08T22:14:41.103552Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 0 Children component(s)
2023-11-08T22:14:41.103573Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 0 Parent component(s)
2023-11-08T22:14:41.103995Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: advancing to frame: 3
2023-11-08T22:14:41.104151Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: frame 3 completed
2023-11-08T22:14:41.104250Z DEBUG ggrs{name="HandleRequests"}:schedule{name="SaveWorld"}: bevy_ggrs::schedule_systems: saving snapshot for frame 3
2023-11-08T22:14:41.104366Z TRACE bevy_ggrs::snapshot::entity_checksum: Rollback Entities have checksum EFC1504252A15E1
2023-11-08T22:14:41.104370Z TRACE bevy_ggrs::snapshot::component_checksum: Component Health has checksum 3A2A5EFCDFA21F3
2023-11-08T22:14:41.104532Z TRACE bevy_ggrs::snapshot::checksum: Frame has checksum D5EB0EBE8D03412
2023-11-08T22:14:41.104636Z TRACE bevy_ggrs::snapshot::resource_snapshot: Snapshot Time<GgrsTime>
2023-11-08T22:14:41.104644Z TRACE bevy_ggrs::snapshot::resource_snapshot: Snapshot RollbackOrdered
2023-11-08T22:14:41.104654Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 0 Children component(s)
2023-11-08T22:14:41.104660Z TRACE bevy_ggrs::snapshot::entity: Snapshot 0 entity(s)
2023-11-08T22:14:41.104667Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 0 Parent component(s)
2023-11-08T22:14:41.104675Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 0 Health component(s)
2023-11-08T22:14:41.105093Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: advancing to frame: 4
2023-11-08T22:14:41.105243Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: frame 4 completed
[examples/manual_tests/respawn_entity.rs:19] &local_players.0 = [
    0,
]
2023-11-08T22:14:41.105690Z  WARN bevy_ggrs::schedule_systems: Detected checksum mismatch during rollback on frame 2.

We can see from the logs that the entity is despawned during frame 3. Then later when restoring frame 1, the logs claim to be rolling back the entity and its Health component, but when advancing to frame 2, no entities with Health are found and no info is printed from decrease_health.

Invalidated Entities

I just found that invalidating entities is still an issue, but luckily I also discovered the fix we need in the bevy_hierarchy crate. Just posting here for a heads-up in case anybody runs into it:

bevyengine/bevy#6790

For now, in my game I've just patched Bevy with the PR linked in the issue above to fix it.

Originally posted by @zicklag in #29 (comment)

---

I'm re-opening this issue because the fix I opened in Bevy wasn't accepted, because it could cause hierarchy inconsistencies. That's a valid point, but we need the behavior I added to make our entity updates work right.

I think what we have to do is we have to make a RollbackMapEntities trait that works essentially similar to the MapEntities trait, but with a slight behavior change, that we can implement for Parent and Children and that users can implement for their own component types that need entity mapping.

[Question] What Will Be the New Snapshot Strategy?

Hey there!

I saw this in the README:

Since I am not happy with using bevy-reflect to save and load snapshots of the world, I am looking forward to this refactoring!

I've been working on and exploring some Bevy networking stuff and I ended up using Reflect and a custom CompactReflectSerializer/Deserializer to serialize byte-efficient versions of all the components for snapshots.

It sounds like you were using Reflect too, but aren't going to be anymore and I was curious how you were going to do the snapshots when the new stageless implementation lands, if it isn't going to use reflection.

I might be able to make use of whatever it is.

Invalidated Entity's

To answer your side question:

GGRS rewinds by loading an old state and overwriting current values. When loading a snapshot, items that have been deleted will have to be recreated and thus get a new entity id. In the same time, your inventory will be overwritten with the entity id vector it had at that point in time. These IDs will then probably be invalid.

Originally posted by @gschup in #11 (comment)

---

I started a new thread to avoid cluttering the other one.

So, if the entities in an inventory component could end up invalidated by the rollback, does that mean Bevy's hierarchy could get corrupted too?

Because the hierarchy works by having Parent and Children components where Entitys are used for references, similar to an inventory.

Docs are broken

Describe the bug
The newest docs are broken

To Reproduce
Steps to reproduce the behavior:

1. Go to https://docs.rs/crate/bevy_ggrs/latest
2. See error

Expected behavior
I see the newest docs

Screenshots

Desktop (please complete the following information):

• OS: macOS
• Browser [e.g. chrome, safari]: Firefox Nightly
• Version [e.g. 22]: 101.0a1 (2022-04-29)

Additional context
Last working docs were 0.1.3

Document Non-Synced Bevy Gotcha's ( Events, etc. )

Is your feature request related to a problem? Please describe.
The current design of Bevy GGRS doesn't snapshot and restore Bevy events, so events that are handled in a frame may not get handled again if the frame is rolled back to.

Describe the solution you'd like
We might need a custom event resource type that is compatible with the rollback and snapshot system.

Describe alternatives you've considered
A quick attempt to snapshot the Events<T> resource from Bevy didn't turn out so well. Events<T> isn't Clone even if T: Clone, which presents challenges, and the exposed public API doesn't allow us enough access to reasonably snapshot and restore it. Even if we could, we'd have to have a way to reset the read counters on the Local<ManualEventReader<T>> storage that powers the EventReader<T> system param.

Essentially bevy's built-in event system probably just isn't suitable for snapshot and restore, and we need some alternative, or we just have to make this as a caveat and say that you have to use components for sending events or something like that.

Additional context
I'm actually a little confused why some events in my game seem to be working fine after rollback and restore. I'm still debugging. I'm still trying to get my game working in the SyncTest mode, and things get a little confusing to debug when the world is rolling back and forth the whole time. :D

There's a small chance that I'm misunderstanding something and events could work fine without changes, but I highly doubt it.

More idiomatic component/resource checksums for float types?

Is your feature request related to a problem? Please describe.

It's very useful to have desync detection for things like player positions etc. However if your player positions are Transform or glam::Vec3s or similar this currently requiries implementing Hash for those types, which is perhaps not ideal, as Hash isn't implemented for f32, and even though it's possible to do so (using e.g. to_bits), it is perhaps a bit of a footgun for other code that assumes a more correct Hash implementation

Describe the solution you'd like
I'm not sure exactly how I'd like this solved, but perhaps another trait and/or derive could solve it. I'd essentially just want to do:

#[derive(Reflect, Component, bevy_ggrs::Checksum]
#[reflect(Checksum)]
struct Foo(f32);

Describe alternatives you've considered
Maybe this is an issue with bevy_reflect, not sure what #[reflect(hash)] is typically used for outside bevy_ggrs... Maybe this is just the way we have to do it.

Allow to register Resources

Currently, only entities can be saved in a WorldSnapshot.

Allowing users to register resources is a potentially very useful feature.

Allow creating of stages inside the rollback schedule

Currently, it is not possible to create stages inside the GGRS schedule or define system orderings.

The API should allow the user to create stages or define system orderings.

RollbackIdProvider needs a smarter solution

Is your feature request related to a problem? Please describe.
Currently, when the RollbackIdProvider reaches u32::MAX, it simply panics. Since no sensible application should have that many rollbackable entitites at the same time, there should be a smarter way to provide unique (within the rollback frame window) IDs for entities.

Additional context
The relevant code is found here.

Potential Scalability Refactor

Hello,

I am happy to have stumbled upon this project, I have been working on a GGPO like project that takes the GGPO algorithm to the next level so it can be scaled up and used for games that have 1000's of units. It does this by having a never rewind policy. Basically the official correct state is never stepped forward until it has the complete input for that frame. Then to remove lag, the current frame is rendered by taking a snapshot of the most recent official state, and fast forwarding a subset of the game state (for example, objects only visible or potentially visible on the players screen in the next 300 milliseconds). The local player input is inserted into these fast forwarded frames, and the remote player inputs are guessed.

I have written an article about this technique: http://retroprose.net/2021/06/10/universal-netcode-framework/
And here is an example of it in action: http://retroprose.net/files/hurkle/

That demo uses unity and csharp, but I ported it to Rust and still need to port the networking code and stumbled on this plug in for bevy. I noticed there is still some lag in my demo, but that may just be unity input lag or something with my server (a free tier Rander server running Go).

The demo uses all integer math to get determinism. I'm still in the process of evaluating bevy to determine if I can create deterministic code with floating point math or fixed point math.

I noticed that some bevy changes may force this project to get refactored anyway, I thought this article may spark some ideas for how to apply the GGPO algorithm in a new way.

Despawned entities do not get respawned properly

Describe the bug
Despawned entities do not get respawned properly. this issue came up during the bevy_jam and is here to remind myself to look into it later.

Adding non-rollbacked children to rollbacked entities break the hierarchy

Describe the bug

I'd like it to be possible to spawn cosmetic entities as children of rolled back entities.

Used to work prior to #31

For instance to:

• compose a character of several sprites
• use 3rd-party plugins for particle effects

pub fn spawn_character_particles(
    commands: &mut Commands,
    character_entity: Entity,
    sprites: &Res<SpriteAssets>,
) {
    info!("Adding particles to character: {character_entity:?}");
    commands
        .entity(character_entity)
        .with_children(|character| {
            character
                .spawn((
                    Name::new("JetPackParticles"),
                    ParticleSystemBundle {
                        // transform: Transform::from_xyz()
                        particle_system: ParticleSystem {
                            max_particles: 10_000,
                            texture: ParticleTexture::Sprite(sprites.particle.clone()),
                            spawn_rate_per_second: 40.0.into(),
                            emitter_shape: CircleSegment {
                                radius: 0.0.into(),
                                opening_angle: 0.5,
                                direction_angle: PI * 3. / 2.,
                            }
                            .into(),

                            initial_speed: JitteredValue::jittered(1.0, -0.5..0.5),
                            // acceleration: todo!(),
                            lifetime: JitteredValue::jittered(0.3, -0.1..0.1),
                            color: ColorOverTime::Gradient(Curve::new(vec![
                                CurvePoint::new(Color::WHITE, 0.0),
                                CurvePoint::new(Color::NONE, 1.0),
                            ])),
                            scale: ValueOverTime::Constant(0.0005),
                            looping: true,
                            system_duration_seconds: 10.0,
                            z_value_override: Some(JitteredValue::new(55.0)),
                            // bursts: todo!(),
                            // space: todo!(),
                            ..default()
                        },
                        ..default()
                    },
                    bevy_particle_systems::Playing,
                    JetPackParticles,
                ))
                .log_entity_id("spawned jetpack_particles");
        });
}

Currently, this results in Children containing "dead" entities after the first time a snapshot restore happens.

To Reproduce
Steps to reproduce the behavior:

1. spawn an entity with .add_rollback(), and a child without .add_rollback()
2. trigger a rollback
3. traverse Children hierarchy, it now points to a "dead" entity

Expected behavior

I would expect the entity ids of the children without Rollback components to be left alone (like they used to).

Describe alternatives you've considered

Adding "cosmetic follower entities" in a separate hierarchy branch that contain a reference to the rolled back entity, and have a book-keeping step where entities with dangling refs are despawned, and positions are synced.

Additional context

The problem is that map_entities for unknown entities returns "dead" entities: https://github.com/bevyengine/bevy/blob/8ace2ff9e361dd7ef1bc620b84674def0cb56454/crates/bevy_hierarchy/src/components/children.rs#L26

bevyengine/bevy#6790

Desync detection appears to compare the checksum of predicted frames to confirmed frames

Describe the bug
Desyncs are reported because predicated checksums and confirmed checksums are being compared.

To Reproduce
My repro can be found here on the desync-detection-issue branch: https://github.com/gschup/bevy_ggrs/compare/main...nezuo:bevy_ggrs:desync-detection-issue?expand=1

I used clumsy so that it predicts inputs/rollbacks.

Expected behavior
Only the checksums of confirmed frames should be compared.

Screenshots
Frame 18s for player 1 (in order):

Frame 18 for player 2:

The second frame 18 for player 1 is the same as player 2's frame 18. However, it's comparing the first frame 18 with player 2's frame 18.

publish to crates.io

Will be done once bevy 0.6 is released.

panic: `Rollback requested was not created using AddRollbackCommand!`

Describe the bug

Run the example in #78

To Reproduce

cargo watch -cx "run --release --example particles -- --local-port 7001 --players 127.0.0.1:7000 localhost --input-delay 0 --desync-detection-interval 1 --rate 1" 
cargo watch -cx "run --release --example particles -- --local-port 7000 --players localhost 127.0.0.1:7001 --input-delay 0 --desync-detection-interval 1 --rate 1" 

Press space a couple of times on one of the peers.

Sometimes, it will desync, and sometimes, it will panic with the below:

2023-10-26T09:25:03.902877Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: advancing to frame: 122
2023-10-26T09:25:03.903039Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: frame 122 completed
2023-10-26T09:25:03.918700Z DEBUG ggrs{name="HandleRequests"}:schedule{name="LoadWorld"}: bevy_ggrs::schedule_systems: restoring snapshot for frame 121
2023-10-26T09:25:03.919139Z TRACE bevy_ggrs::snapshot::entity: Rolled back 0 entity(s)
2023-10-26T09:25:03.919272Z TRACE bevy_ggrs::snapshot::component_snapshot: Rolled back 0 Parent component(s)
2023-10-26T09:25:03.919355Z TRACE bevy_ggrs::snapshot::component_snapshot: Rolled back 0 Sprite component(s)
2023-10-26T09:25:03.919279Z TRACE bevy_ggrs::snapshot::component_snapshot: Rolled back 0 ComputedVisibility component(s)
2023-10-26T09:25:03.919279Z TRACE bevy_ggrs::snapshot::component_snapshot: Rolled back 0 Transform component(s)
2023-10-26T09:25:03.919289Z TRACE bevy_ggrs::snapshot::component_snapshot: Rolled back 0 Children component(s)
2023-10-26T09:25:03.919299Z TRACE bevy_ggrs::snapshot::component_snapshot: Rolled back 0 Visibility component(s)
2023-10-26T09:25:03.919311Z TRACE bevy_ggrs::snapshot::component_snapshot: Rolled back 0 Ttl component(s)
2023-10-26T09:25:03.919324Z TRACE bevy_ggrs::snapshot::component_snapshot: Rolled back 0 Velocity component(s)
2023-10-26T09:25:03.919337Z TRACE bevy_ggrs::snapshot::component_snapshot: Rolled back 0 GlobalTransform component(s)
2023-10-26T09:25:03.919345Z TRACE bevy_ggrs::snapshot::component_snapshot: Rolled back 0 Handle<Image> component(s)
2023-10-26T09:25:03.919278Z TRACE bevy_ggrs::snapshot::resource_snapshot: Rolled back ParticleRng
2023-10-26T09:25:03.920092Z TRACE ggrs{name="HandleRequests"}:schedule{name="LoadWorld"}: bevy_ggrs::snapshot::component_map: Mapped Parent
2023-10-26T09:25:03.920210Z TRACE ggrs{name="HandleRequests"}:schedule{name="LoadWorld"}: bevy_ggrs::snapshot::component_map: Mapped Children
2023-10-26T09:25:03.920346Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: advancing to frame: 122
2023-10-26T09:25:03.920512Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: frame 122 completed
2023-10-26T09:25:03.920601Z DEBUG ggrs{name="HandleRequests"}:schedule{name="SaveWorld"}: bevy_ggrs::schedule_systems: saving snapshot for frame 122
2023-10-26T09:25:03.920683Z TRACE bevy_ggrs::snapshot::component_checksum_hash: Component Velocity has checksum 987D2AEF7711FCF5
2023-10-26T09:25:03.920740Z TRACE bevy_ggrs::snapshot::checksum: Frame has checksum 987D2AEF7711FCF5
2023-10-26T09:25:03.920806Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 0 Parent component(s)
2023-10-26T09:25:03.920868Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 1 Transform component(s)
2023-10-26T09:25:03.920818Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 1 Sprite component(s)
2023-10-26T09:25:03.920824Z TRACE bevy_ggrs::snapshot::entity: Snapshot 1 entity(s)
2023-10-26T09:25:03.920825Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 1 GlobalTransform component(s)
2023-10-26T09:25:03.920836Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 1 Handle<Image> component(s)
2023-10-26T09:25:03.920841Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 1 Visibility component(s)
2023-10-26T09:25:03.920848Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 1 Ttl component(s)
2023-10-26T09:25:03.920856Z TRACE bevy_ggrs::snapshot::resource_snapshot: Snapshot ParticleRng
2023-10-26T09:25:03.920808Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 0 Children component(s)
2023-10-26T09:25:03.920879Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 1 Velocity component(s)
2023-10-26T09:25:03.920878Z TRACE bevy_ggrs::snapshot::component_snapshot: Snapshot 1 ComputedVisibility component(s)
2023-10-26T09:25:03.921549Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: advancing to frame: 123
2023-10-26T09:25:03.921676Z  INFO particles: from [0, 0, 0] to [0.9723242, 2.198806, 0]
2023-10-26T09:25:03.921753Z DEBUG ggrs{name="HandleRequests"}:schedule{name="AdvanceWorld"}: bevy_ggrs::schedule_systems: frame 123 completed
2023-10-26T09:25:03.934996Z DEBUG ggrs{name="HandleRequests"}:schedule{name="SaveWorld"}: bevy_ggrs::schedule_systems: saving snapshot for frame 123
thread 'Compute Task Pool (4)' panicked at src\rollback.rs:99:14:
Rollback requested was not created using AddRollbackCommand!

Expected behavior

No panics, since all rollback components were actually added through the .add_rollback() extension

Generating checksums for WorldSnapshots

The problem: No checksums are generated, so SyncTestSession cannot compare checksums.

In order to make SyncTestSession meaningful, the GGRS plugin either needs to automatically compute a checksum from a WorldSnapshot or the API should provide the user a possibility to provide checksums themselves.

plugin not working with non-default session generics

The plugin currently does not work with non-default ggrs session generics that were introduced to GGRS with version 0.8, e.g. ggrs::P2PSession<Vec<u8>, String>.

To fix this, a bunch of bevy_ggrs's internals will have to be generic.

Sessions `.with_desync_detection(DesyncDetection::On)` always reports false positives

Describe the bug

Check out the branch in #48 , which enables desync detection in the p2p example. As soon as it starts, it reports a desync.

Expected behavior
It should not report a desync (since it doesn't desync. only the hash is different)

Desktop (please complete the following information):

• Linux

Additional context
This happens because bevy_reflect uses ahash::AHasher::default. Since bevy uses ahasher with the default features, this means it's seeded at runtime, which means hashes will always be different across peers, see: https://docs.rs/ahash/latest/ahash/struct.AHasher.html#method.default

When rollbacks happen across entity spawning, desyncs are reported

Describe the bug

When rollbacks happen across entity spawning, desyncs are reported

To Reproduce

Run the example from #78 with --input-delay 0

Press space a couple of times, observe a desync being reported.

Expected behavior

No desyncs reported

Additional information

I'm not sure if this is a real desync or a false positive.

Related to #79 same steps to repro, perhaps the same root cause as well.