smol-rs / async-compat Goto Github PK

Hi,
I am developing a bevy game, which uses async-executor under the hood as its runtime.
I need to interact with TCP or webtransport, and both crates use the tokio runtime.

I've been able to use the tokio runtime without issue, but i'd like to avoid using it since the rest of bevy relies on smol.

I was able to replace my IO-related tasks to use Compat::new() and it seemed to work perfectly.
It works well in native, but when I tried to connect to the server in wasm I get the error:

Error(JsValue(RangeError: supplied view is not large enough.

I'm not entirely sure what this could be due to? The only thing I recall changing is adding Compat::new

Sometimes, non-poll methods expect to be able to spawn tasks. One common case is Drop implementations of types that want to do some async cleanup. Would you accept PRs for any of the following?

• enter_tokio_context + TokioEnterGuard wrapping TOKIO1.enter()
• impl Drop<T> for Compat<T> that enters the tokio context while dropping the inner value (requires some unsafe for ManuallyDrop, I think)

buf_redux is Unmaintained

Last release was over three years ago.

The maintainer(s) have been unreachable to respond to any issues that may or may not include security issues.

The repository is now archived and there is no security policy in place to contact the maintainer(s) otherwise.

The safety-undocumented unsafe in the crate may or may not be safe to use.

The crate also has a current future incompatibility warning buf_redux/23.

Possible Alternatives

The below may or may not provide alternative(s)

• Rust alloc / std vec::Vec, collections::VecDeque
• buffer-redux - fork
• bytes
• crates.io search for 'buffer'

See advisory page for additional details.

Crate twoway deprecated by the author

The commit e99b3c7 releasing version 0.2.2 explicitely deprecates twoway in favour of memchr crate.

See advisory page for additional details.

https://github.com/stjepang/smol/blob/3e899311b08fa01693e83560384e6cb8850e5573/src/context.rs seems to use num_cpu threads by default. It will probably be better if we can control the number of threads used (especially necessary when we want to run a program on a machine with more than a few dozen cores).

Similar issue in async-std: async-rs/async-std#774, async-rs/async-std#690.

I remember Tokio already has configurations like core_threads and max_threads, but currently there's no way to configure them in smol.

When using the async-compression crate, I'm able to sprinkle .compat() or Compat::new(...) generously to silence most errors.
However, encoder.shutdown needs poll_shutdown, and I see that's not available in smol::io::AsyncWriteExt.

Is there any chance of adding poll_shutdown() and shutdown(), so that the smol runtime works with async-compression?

Thanks in advance for sharing your work (in general), and for writing about it!

From my reading of the code, Compat futures will unconditionally create and enter a new single-threaded tokio runtime. This seems potentially inefficient if a tokio runtime (maybe a multi-threaded one) already exists.

Would it be possible to check if a runtime already exists before entering a new one?

tokio 0.3 has been released, it would be nice to have async-compat follow this

I tried this example from the README, using the latest async-compat, 0.1.5:

use async_compat::{Compat, CompatExt};

fn main() -> std::io::Result<()> {
    futures::executor::block_on(Compat::new(async {
        let stdin = tokio::io::stdin();
        let mut stdout = tokio::io::stdout();

        futures::io::copy(stdin.compat(), &mut stdout.compat_mut()).await?;
        Ok(())
    }))
}

When run with Tokio 0.2, it is fine. When run with Tokio 0.3, it fails with the following errors:

error[E0277]: the trait bound `tokio::io::Stdin: tokio::io::async_read::AsyncRead` is not satisfied
  --> src/main.rs:8:9
   |
8  |         futures::io::copy(stdin.compat(), &mut stdout.compat_mut()).compat().await?;
   |         ^^^^^^^^^^^^^^^^^ the trait `tokio::io::async_read::AsyncRead` is not implemented for `tokio::io::Stdin`
   | 
  ::: /home/chead/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.7/src/io/copy.rs:36:8
   |
36 |     R: AsyncRead,
   |        --------- required by this bound in `futures::io::copy`
   |
   = note: required because of the requirements on the impl of `futures::AsyncRead` for `async_compat::Compat<tokio::io::Stdin>`

error[E0277]: the trait bound `tokio::io::Stdout: tokio::io::async_write::AsyncWrite` is not satisfied
  --> src/main.rs:8:43
   |
8  |         futures::io::copy(stdin.compat(), &mut stdout.compat_mut()).compat().await?;
   |                                           ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `tokio::io::async_write::AsyncWrite` is not implemented for `tokio::io::Stdout`
   | 
  ::: /home/chead/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.7/src/io/copy.rs:37:8
   |
37 |     W: AsyncWrite + Unpin + ?Sized,
   |        ---------- required by this bound in `futures::io::copy`
   |
   = note: required because of the requirements on the impl of `tokio::io::async_write::AsyncWrite` for `&mut tokio::io::Stdout`
   = note: required because of the requirements on the impl of `futures::AsyncWrite` for `async_compat::Compat<&mut tokio::io::Stdout>`

error[E0277]: the trait bound `tokio::io::Stdin: tokio::io::async_read::AsyncRead` is not satisfied
 --> src/main.rs:8:9
  |
8 |         futures::io::copy(stdin.compat(), &mut stdout.compat_mut()).compat().await?;
  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `tokio::io::async_read::AsyncRead` is not implemented for `tokio::io::Stdin`
  |
  = note: required because of the requirements on the impl of `futures::AsyncRead` for `async_compat::Compat<tokio::io::Stdin>`
  = note: required because of the requirements on the impl of `futures::Future` for `futures::io::Copy<'_, async_compat::Compat<tokio::io::Stdin>, async_compat::Compat<&mut tokio::io::Stdout>>`

error[E0277]: the trait bound `tokio::io::Stdout: tokio::io::async_write::AsyncWrite` is not satisfied
 --> src/main.rs:8:9
  |
8 |         futures::io::copy(stdin.compat(), &mut stdout.compat_mut()).compat().await?;
  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `tokio::io::async_write::AsyncWrite` is not implemented for `tokio::io::Stdout`
  |
  = note: required because of the requirements on the impl of `tokio::io::async_write::AsyncWrite` for `&mut tokio::io::Stdout`
  = note: required because of the requirements on the impl of `futures::AsyncWrite` for `async_compat::Compat<&mut tokio::io::Stdout>`
  = note: required because of the requirements on the impl of `futures::Future` for `futures::io::Copy<'_, async_compat::Compat<tokio::io::Stdin>, async_compat::Compat<&mut tokio::io::Stdout>>`

Since CompatExt is implemented for any T, this means that IDEs give you suggestions for its functions on every object, which for most objects is just spam. Is it possible to implement CompatExt only for types that implement the Future trait? Since Compat is a future only when its inner type is a Future, it would make sense that CompatExt is only implemented for types that implement Future trait.

I am getting panics with errors like:

thread 'async-compat/tokio-1' panicked at /dev/rust/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.39.0/src/util/linked_list.rs:123:9:
assertion `left != right` failed
  left: Some(0x6000012ae400)
 right: Some(0x6000012ae400)

I'm not sure if it's due to async-compat or not

I think Compat should have a Clone impl if T has a Clone impl, for example if you've used Arc from async-dup.

It would be very cool if a Compat could be produced for consuming the Spawner in the Tokio Handler so that the non-tokio runtime can spawn Tokio tasks.

The way I'm thinking to do it is to first, have the background thread do a sleep(u64::MAX) so that it can never process any spawned tasks.

Then, we hijack the Spawner from the handler (currently impossible as spawner field along with other needed data types are pub(crate)) and process tasks from it by putting them onto the current runtime's task queue.