Event-driven network application framework for high performance protocol servers & clients, non-blocking.
Home Page: https://swiftpackageindex.com/apple/swift-nio/documentation
License: Apache License 2.0
The way to open files seems to be:
NIO.FileHandle.init(path:)
which is a blocking operation. That seems no good.
A non-blocking way to open a file. Maybe this belongs into NonBlockingFileIO.
The HEAD when I pressed "Submit".
It would be great to have a type-safe ChannelPipeline. The whole ChannelPipeline can't be all type-safe as we want to support arbitrary modifications of it at runtime (for example for HTTP/1 to HTTP/2 upgrades).
However it would be great to be able to have at least a partially type-safe pipeline which could be fully type-safe for programs who don't need the dynamism.
We normally want debugging information to keep track of where our promises are allocated in debug modes. However, if someone does Channel.newPromise instead of eventLoop.newPromise, that information is lost. The core issue is here:
swift-nio/Sources/NIO/ChannelInvoker.swift
Lines 182 to 184 in fef728c
This indirection should preserve the #file and #line specific debugging logic we use elsewhere.
It seems that clang 3.8 or later is required for use with Swift NIO.
-> /swift/swift-4.1-DEVELOPMENT-SNAPSHOT-2018-03-12-a-ubuntu16.04/usr/bin/swift --version && clang --version
Swift version 4.1-dev (LLVM 1c8b50929b, Clang 420ae40df6, Swift ec821efb01)
Target: x86_64-unknown-linux-gnu
Ubuntu clang version 3.6.2-3ubuntu2 (tags/RELEASE_362/final) (based on LLVM 3.6.2)
Target: x86_64-pc-linux-gnu
Thread model: posix
-> /swift/4.1.0-beta/usr/bin/swift --version && clang --version
Swift version 4.1-dev (LLVM 4d75b446be, Clang db92199adc, Swift 4cb7e7f233)
Target: x86_64-unknown-linux-gnu
Ubuntu clang version 3.6.2-3ubuntu2 (tags/RELEASE_362/final) (based on LLVM 3.6.2)
Target: x86_64-pc-linux-gnu
Thread model: posix-> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenialCompile CNIOAtomics src/c-atomics.c
Compile CNIOLinux shim.c
Compile CNIODarwin shim.c
Compile CNIOZlib empty.c
Compile CNIOHTTPParser c_nio_http_parser.c
Compile Swift Module 'Debugging' (4 sources)
In file included from /vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/c-atomics.c:21:
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:20:37: error: expected ';' after top level declarator
struct catmc_atomic__Bool * _Nonnull catmc_atomic__Bool_create(bool value);
^
;
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:21:70: error: expected ')'
void catmc_atomic__Bool_destroy(struct catmc_atomic__Bool * _Nonnull atomic);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:21:32: note: to match this '('
void catmc_atomic__Bool_destroy(struct catmc_atomic__Bool * _Nonnull atomic);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:22:83: error: expected ')'
bool catmc_atomic__Bool_compare_and_exchange(struct catmc_atomic__Bool * _Nonnull atomic, bool expected, bool desired);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:22:45: note: to match this '('
bool catmc_atomic__Bool_compare_and_exchange(struct catmc_atomic__Bool * _Nonnull atomic, bool expected, bool desired);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:23:66: error: expected ')'
bool catmc_atomic__Bool_add(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:23:28: note: to match this '('
bool catmc_atomic__Bool_add(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:24:66: error: expected ')'
bool catmc_atomic__Bool_sub(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:24:28: note: to match this '('
bool catmc_atomic__Bool_sub(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:25:71: error: expected ')'
bool catmc_atomic__Bool_exchange(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:25:33: note: to match this '('
bool catmc_atomic__Bool_exchange(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:26:67: error: expected ')'
bool catmc_atomic__Bool_load(struct catmc_atomic__Bool * _Nonnull atomic);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:26:29: note: to match this '('
bool catmc_atomic__Bool_load(struct catmc_atomic__Bool * _Nonnull atomic);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:27:68: error: expected ')'
void catmc_atomic__Bool_store(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:27:30: note: to match this '('
void catmc_atomic__Bool_store(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:29:28: error: redefinition of '_Nonnull' with a different type: 'struct catmc_atomic_char *' vs 'struct catmc_atomic__Bool *'
struct catmc_atomic_char * _Nonnull catmc_atomic_char_create(char value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:20:29: note: previous definition is here
struct catmc_atomic__Bool * _Nonnull catmc_atomic__Bool_create(bool value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:29:36: error: expected ';' after top level declarator
struct catmc_atomic_char * _Nonnull catmc_atomic_char_create(char value);
^
;
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:30:68: error: expected ')'
void catmc_atomic_char_destroy(struct catmc_atomic_char * _Nonnull atomic);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:30:31: note: to match this '('
void catmc_atomic_char_destroy(struct catmc_atomic_char * _Nonnull atomic);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:31:81: error: expected ')'
bool catmc_atomic_char_compare_and_exchange(struct catmc_atomic_char * _Nonnull atomic, char expected, char desired);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:31:44: note: to match this '('
bool catmc_atomic_char_compare_and_exchange(struct catmc_atomic_char * _Nonnull atomic, char expected, char desired);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:32:64: error: expected ')'
char catmc_atomic_char_add(struct catmc_atomic_char * _Nonnull atomic, char value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:32:27: note: to match this '('
char catmc_atomic_char_add(struct catmc_atomic_char * _Nonnull atomic, char value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:33:64: error: expected ')'
char catmc_atomic_char_sub(struct catmc_atomic_char * _Nonnull atomic, char value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:33:27: note: to match this '('
char catmc_atomic_char_sub(struct catmc_atomic_char * _Nonnull atomic, char value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:34:69: error: expected ')'
char catmc_atomic_char_exchange(struct catmc_atomic_char * _Nonnull atomic, char value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:34:32: note: to match this '('
char catmc_atomic_char_exchange(struct catmc_atomic_char * _Nonnull atomic, char value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:35:65: error: expected ')'
char catmc_atomic_char_load(struct catmc_atomic_char * _Nonnull atomic);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:35:28: note: to match this '('
char catmc_atomic_char_load(struct catmc_atomic_char * _Nonnull atomic);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:36:66: error: expected ')'
void catmc_atomic_char_store(struct catmc_atomic_char * _Nonnull atomic, char value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:36:29: note: to match this '('
void catmc_atomic_char_store(struct catmc_atomic_char * _Nonnull atomic, char value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:38:29: error: redefinition of '_Nonnull' with a different type: 'struct catmc_atomic_short *' vs 'struct catmc_atomic__Bool *'
struct catmc_atomic_short * _Nonnull catmc_atomic_short_create(short value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:20:29: note: previous definition is here
struct catmc_atomic__Bool * _Nonnull catmc_atomic__Bool_create(bool value);
^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:38:37: error: expected ';' after top level declarator
struct catmc_atomic_short * _Nonnull catmc_atomic_short_create(short value);
^
;
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
error: terminated(1): /swift/4.1.0-beta/usr/bin/swift-build-tool -f /vapor/hi-nodes-production/code/.build/release.yaml main output:
Fetching https://github.com/vapor/console.git
Fetching https://github.com/vapor/database-kit.git
Fetching https://github.com/vapor/routing.git
Fetching https://github.com/apple/swift-nio.git
Fetching https://github.com/vapor/service.git
Fetching https://github.com/vapor/validation.git
Fetching https://github.com/vapor/crypto.git
Fetching https://github.com/apple/swift-nio-zlib-support.git
Fetching https://github.com/vapor/template-kit.git
Fetching https://github.com/vapor/vapor.git
Fetching https://github.com/vapor/core.git
Fetching https://github.com/vapor/engine.git
Fetching https://github.com/vapor/copenssl.git
Cloning https://github.com/apple/swift-nio-zlib-support.git
Resolving https://github.com/apple/swift-nio-zlib-support.git at 1.0.0
Cloning https://github.com/vapor/vapor.git
Resolving https://github.com/vapor/vapor.git at nio
Cloning https://github.com/vapor/core.git
Resolving https://github.com/vapor/core.git at nio
Cloning https://github.com/vapor/copenssl.git
Resolving https://github.com/vapor/copenssl.git at 1.0.0-rc.1
Cloning https://github.com/vapor/console.git
Resolving https://github.com/vapor/console.git at nio
Cloning https://github.com/vapor/database-kit.git
Resolving https://github.com/vapor/database-kit.git at nio
Cloning https://github.com/vapor/routing.git
Resolving https://github.com/vapor/routing.git at nio
Cloning https://github.com/apple/swift-nio.git
Resolving https://github.com/apple/swift-nio.git at 1.1.0
Cloning https://github.com/vapor/service.git
Resolving https://github.com/vapor/service.git at nio
Cloning https://github.com/vapor/validation.git
Resolving https://github.com/vapor/validation.git at nio
Cloning https://github.com/vapor/crypto.git
Resolving https://github.com/vapor/crypto.git at nio
Cloning https://github.com/vapor/template-kit.git
Resolving https://github.com/vapor/template-kit.git at nio
Cloning https://github.com/vapor/engine.git
Resolving https://github.com/vapor/engine.git at nio
Error: deploy failed.While SwiftNIO is more or fairly close to Netty 4.1 there are a few things which differ (in many cases things we want to fix in Netty 5.x as well). We should document these differences.
This line is getting hit when MultiThreadedEventLoopGroup is configured with numThreads > 1 right here.
You can recreate this with the following steps:
1: clone swift-nio master
2: configure group to System.coreCount threads or just hard-code to 8
3: run NIOHTTP1Server executable
4: run curl localhost:8888
This doesn't get hit in release mode (obviously) and wrk seems to work fine, but I'm assuming that assert there is important for performance / stability reasons.
It can be very useful for users of SwiftNIO to attach metadata to an EventLoop, we should consider having an API for that.
suggested by @ddunbar
SwiftNIO might be appealing to developers of network servers that need good but maybe not the best network performance. In these cases, the allocate ByteBuffer, accumulate, ctx.write, ..., ctx.flush might be a bit awkward. A simpler API might also be a good starting point. For ChannelHandlers with OutboundOut = ByteBuffer, @ddunbar proposed the following API:
ctx.writeWithStream { stream in
stream <<< “this” <<< Format.asJSON(…) <<< “bla bla bla”
}
which would basically do:
{
var buf = ctx.channel.allocate(capacity: 256 /* random guess */)
buf.write(staticString: "this")
buf.write(bytes: Format.asJSON(...)) // Format.asJSON is actually more sophisticated than that. It returns a "formatter", which is a struct which knows how to render the output as JSON directly onto the stream. This avoids creating unnecessary temporary allocations of the JSON data.
buf.write(staticString: "bla bla bla")
return ctx.write(self.wrapOutboundOut(.byteBuffer(buf)))
}()
and I guess we could also offer a ctx.writeAndFlushWithStream API. Or maybe ctx.writeWithStream(flush: Bool = false, (Stream) -> EventLoopPromise<()>) -> EventLoopPromise<()> 🤔.
The custom <<< operator would be defined for everything we accept: ByteBuffer, (Contiguous)Collection of UInt8, String, StaticString, the various integers, ...
We should definitely evaluate having such a more high-level API.
This should probably live in some yet to be created NIOExtras module. And we need to think about how to make it perform well and in an unsurprising way.
I'm interested in beginning translating the project into Spanish and I was wondering if I could start pull requests of the files. Many thanks in advance!
I saw that in the Docker configuration multiple versions of Swift are defined (4.0.2 & 4.0). There is a particular reason for it @tomerd?
Upgrading it to the latest version (4.0.3) makes all the Test Suite pass on Ubuntu.
Should I do directly a PR with these changes?
All tests in the Test Suite to pass.
1 Failing Test Case:
HTTPUpgradeTestCase.testDelayedUpgradeBehaviour
Logs:
test_1 | Test Case 'HTTPUpgradeTestCase.testDelayedUpgradeBehaviour' started at 2018-03-03 03:28:58.078
test_1 | /code/Tests/NIOHTTP1Tests/HTTPUpgradeTests.swift:694: error: HTTPUpgradeTestCase.testDelayedUpgradeBehaviour : XCTAssertFalse failed -
test_1 | Test Case 'HTTPUpgradeTestCase.testDelayedUpgradeBehaviour' failed (0.003 seconds)
Just follow the README instructions on how to run the test suite for Ubuntu using Docker.
1.0.0 / 5ff6d9b
swift --version && uname -a)Specified in the Docker configuration.
Given that channelRead is the most prominent ChannelInboundHandler method, and that write is the most prominent ChannelOutboundHandler method, it's easy for users to assume that inbound means reads, and outbound means writes. This will cause user confusion when they discover that channelWritabilityChanged is an Inbound method, and that read is an Outbound one!
We should write some narrative docs for these two protocols to make it clear that this is not actually how they work: instead, that inbound/outbound is about causality. Specifically, Inbound events are things the socket tells the pipeline, while Outbound events are things the pipeline tells the socket. From that construction it becomes clear why read is Outbound (the pipeline tells the socket to do a read), and why channelWritabilityChanged is Inbound (the socket tells the pipeline whether it's writable or not).
docker-compose -f docker/docker-compose.yaml up test runs the test suite.
Building the docker image fails:
Step 26/41 : ARG version=4.0.3
---> Using cache
---> 98a838f71a66
Step 27/41 : RUN mkdir $HOME/.swift
---> Using cache
---> c1011b5f61d1
Step 28/41 : RUN wget -q https://swift.org/builds/swift-${version}-release/ubuntu1404/swift-${version}-RELEASE/swift-${version}-RELEASE-ubuntu14.04.tar.gz -O $HOME/swift.tar.gz
---> Running in 3c69231a1f55
ERROR: Service 'swift-nio' failed to build: The command '/bin/sh -c wget -q https://swift.org/builds/swift-${version}-release/ubuntu1404/swift-${version}-RELEASE/swift-${version}-RELEASE-ubuntu14.04.tar.gz -O $HOME/swift.tar.gz' returned a non-zero code: 4
Run docker-compose -f docker/docker-compose.yaml up test
swift --version && uname -a)Apple Swift version 4.0.3 (swiftlang-900.0.74.1 clang-900.0.39.2)
Target: x86_64-apple-macosx10.9
Darwin ZeaPro.local 16.7.0 Darwin Kernel Version 16.7.0: Thu Jan 11 22:59:40 PST 2018; root:xnu-3789.73.8~1/RELEASE_X86_64 x86_64
Docker version 17.12.0-ce, build c97c6d6
P.S.: Can we just use the official image please?
see title, for @weissi to do as he promised that a while ago and then forgot :|
You'd expect all memory allocated when creating a ThreadSpecificVariable to be deallocated when it goes out of scope.
The pthread_key might actually never be deallocated as we never call pthread_key_delete.
all versions up to and including 1.1.0
swift --version && uname -a)doesn't matter
If we call pthread_key_delete in ThreadSpecificVariable's deinit (after turning it into a class), then the destructor passed to pthread_key_create will not be called anymore. That means we'd leak values that have been set to this ThreadSpecificVariable which is unacceptable.
That means we can only call pthread_key_delete if both the following conditions are met:
ThreadSpecificVariable has been deallocatedThreadSpecificVariable have exited (and therefore run the destructor passed to pthread_key_create)The only way I could see that we achieve this is to store a Box<(ThreadSpecficVariable, T)>? in the pthread_setspecific because that way the ThreadSpecificVariable will be alive as long as all threads that have a value set.
No channelRead is triggered on the inbound handler if autoRead is off, and the outbound handler does no read() request.
After connecting to a server socket, one read seems to happen regardless of the autoRead option:
.childChannelOption(ChannelOptions.autoRead, value: false)Subsequent input is not reported (just the first stuff entered in nc).
I have no minimal reproduce setup yet. Lazy.
1.1
swift --version && uname -a)4.0.3 , macOS
Blocking the merge of #191:
15:41:06 Test Case 'ChannelNotificationTest.testActiveBeforeChannelRead' started at 2018-03-20 14:41:06.816
15:41:06 Fatal error: leaking promise created at (file: "/code/Tests/NIOTests/ChannelNotificationTest.swift", line: 377): file /code/Tests/NIOTests/ChannelNotificationTest.swift, line 377
15:41:06 Current stack trace:
15:41:06 0 libswiftCore.so 0x00007f5a5c3d4f90 _swift_stdlib_reportFatalErrorInFile + 221
15:41:06 1 libswiftCore.so 0x00007f5a5c0de330 closure #1 in closure #1 in closure #1 in _assertionFailure(_:_:file:line:flags:) + 284
15:41:06 2 libswiftCore.so 0x00007f5a5c37e9a1 <unavailable> + 4155809
15:41:06 3 libswiftCore.so 0x00007f5a5c395059 <unavailable> + 4247641
15:41:06 4 libswiftCore.so 0x00007f5a5c0ddc0b <unavailable> + 1399819
15:41:06 5 libswiftCore.so 0x00007f5a5c2f76b9 <unavailable> + 3602105
15:41:06 6 libswiftCore.so 0x00007f5a5c33579f <unavailable> + 3856287
15:41:06 7 libswiftCore.so 0x00007f5a5c0ddc0b <unavailable> + 1399819
15:41:06 8 libswiftCore.so 0x00007f5a5c2777a0 specialized _assertionFailure(_:_:file:line:flags:) + 144
15:41:06 9 swift-nioPackageTests.xctest 0x00000000004b5dac <unavailable> + 744876
15:41:06 10 swift-nioPackageTests.xctest 0x00000000004b5f0c <unavailable> + 745228
15:41:06 11 swift-nioPackageTests.xctest 0x0000000000557ebc <unavailable> + 1408700
15:41:06 12 swift-nioPackageTests.xctest 0x0000000000557dbd <unavailable> + 1408445
15:41:06 13 swift-nioPackageTests.xctest 0x00000000004b5951 <unavailable> + 743761
15:41:06 14 swift-nioPackageTests.xctest 0x00000000004b5f3f <unavailable> + 745279
15:41:06 15 libswiftCore.so 0x00007f5a5c39c1cb <unavailable> + 4276683
15:41:06 16 swift-nioPackageTests.xctest 0x000000000075a841 <unavailable> + 3516481
15:41:06 17 swift-nioPackageTests.xctest 0x000000000075e0a2 <unavailable> + 3530914
15:41:06 18 swift-nioPackageTests.xctest 0x00000000004414c6 <unavailable> + 267462
15:41:06 19 swift-nioPackageTests.xctest 0x000000000075e15b <unavailable> + 3531099
15:41:06 20 libXCTest.so 0x00007f5a5c859c7f <unavailable> + 203903
15:41:06 21 libXCTest.so 0x00007f5a5c857721 <unavailable> + 194337
15:41:06 22 libXCTest.so 0x00007f5a5c85996f <unavailable> + 203119
15:41:06 23 libXCTest.so 0x00007f5a5c8598ab <unavailable> + 202923
15:41:06 24 libXCTest.so 0x00007f5a5c85b119 <unavailable> + 209177
15:41:06 25 libXCTest.so 0x00007f5a5c85a5c0 <unavailable> + 206272
15:41:06 26 libXCTest.so 0x00007f5a5c83d800 XCTestCase.invokeTest() + 102
15:41:06 27 libXCTest.so 0x00007f5a5c84dea6 <unavailable> + 155302
15:41:06 28 libXCTest.so 0x00007f5a5c83d740 XCTestCase.perform(_:) + 14
15:41:06 29 libXCTest.so 0x00007f5a5c837f20 XCTest.run() + 662
15:41:06 30 libXCTest.so 0x00007f5a5c84e77b <unavailable> + 157563
15:41:06 31 libXCTest.so 0x00007f5a5c84e6b0 <unavailable> + 157360
15:41:06 32 libXCTest.so 0x00007f5a5c84ea84 <unavailable> + 158340
15:41:06 33 libXCTest.so 0x00007f5a5c83a3b0 XCTMain(_:) + 3770
15:41:06 34 swift-nioPackageTests.xctest 0x00000000009c61ed <unavailable> + 6054381
15:41:06 35 libc.so.6 0x00007f5a5a269e50 __libc_start_main + 245
15:41:06 36 swift-nioPackageTests.xctest 0x000000000041b089 <unavailable> + 110729
Is there a reason this package was not added to the products array in the package manifest?
If that's just an oversight, I'm happy to submit a PR. :)
One of the HTTP delayed upgrade tests occasionally fails with the following backtrace:
* thread #7, name = 'NIO-ELT-#0', stop reason = Fatal error: Can't remove last element from an empty collection
frame #0: 0x00000001094ff100 libswiftCore.dylib`_swift_runtime_on_report
frame #1: 0x0000000109542bb1 libswiftCore.dylib`_swift_stdlib_reportFatalError + 113
frame #2: 0x0000000109273d66 libswiftCore.dylib`function signature specialization <Arg[1] = [Closure Propagated : reabstraction thunk helper from @callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> () to @callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> (@out ()), Argument Types : [@callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> ()]> of generic specialization <()> of Swift.StaticString.withUTF8Buffer<A>((Swift.UnsafeBufferPointer<Swift.UInt8>) -> A) -> A + 54
frame #3: 0x00000001094e3df3 libswiftCore.dylib`partial apply forwarder for closure #2 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in Swift._fatalErrorMessage(Swift.StaticString, Swift.StaticString, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never + 99
frame #4: 0x0000000109273d66 libswiftCore.dylib`function signature specialization <Arg[1] = [Closure Propagated : reabstraction thunk helper from @callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> () to @callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> (@out ()), Argument Types : [@callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> ()]> of generic specialization <()> of Swift.StaticString.withUTF8Buffer<A>((Swift.UnsafeBufferPointer<Swift.UInt8>) -> A) -> A + 54
frame #5: 0x00000001093fbfb0 libswiftCore.dylib`function signature specialization <Arg[2] = Dead, Arg[3] = Dead> of Swift._fatalErrorMessage(Swift.StaticString, Swift.StaticString, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never + 96
frame #6: 0x0000000109368d8e libswiftCore.dylib`merged (extension in Swift):Swift.RangeReplaceableCollection< where A: Swift.BidirectionalCollection>.removeLast() -> A.Element + 526
frame #7: 0x0000000109270dd0 libswiftCore.dylib`(extension in Swift):Swift.RangeReplaceableCollection< where A: Swift.BidirectionalCollection>.removeLast() -> A.Element + 16
frame #8: 0x000000010438016d NIOHTTP1Tests`implicit closure #4 in assertResponseIs(lines=0x000070000f838f80) at HTTPUpgradeTests.swift:91
frame #9: 0x00000001043a04e7 NIOHTTP1Tests`partial apply for implicit closure #4 in assertResponseIs(response:expectedResponseLine:expectedResponseHeaders:) at HTTPUpgradeTests.swift:0
frame #10: 0x000000010431a67a NIOHTTP1Tests`thunk for @callee_guaranteed () -> (@owned String, @error @owned Error) at HTTPHeadersTest.swift:0
frame #11: 0x00000001043a054b NIOHTTP1Tests`thunk for @callee_guaranteed () -> (@owned String, @error @owned Error)partial apply at HTTPUpgradeTests.swift:0
frame #12: 0x0000000109c484f2 libswiftXCTest.dylib`merged closure #1 () throws -> () in XCTest.XCTAssertEqual<A where A: Swift.Equatable>(@autoclosure () throws -> A, @autoclosure () throws -> A, @autoclosure () -> Swift.String, file: Swift.StaticString, line: Swift.UInt) -> () + 162
frame #13: 0x0000000109c550da libswiftXCTest.dylib`merged partial apply forwarder for closure #1 () throws -> () in XCTest.XCTAssertGreaterThan<A where A: Swift.Comparable>(@autoclosure () throws -> A, @autoclosure () throws -> A, @autoclosure () -> Swift.String, file: Swift.StaticString, line: Swift.UInt) -> () + 42
frame #14: 0x0000000109c55247 libswiftXCTest.dylib`partial apply forwarder for closure #1 () throws -> () in XCTest.XCTAssertEqual<A where A: Swift.Equatable>(@autoclosure () throws -> A, @autoclosure () throws -> A, @autoclosure () -> Swift.String, file: Swift.StaticString, line: Swift.UInt) -> () + 23
frame #15: 0x0000000109c54f1b libswiftXCTest.dylib`partial apply forwarder for closure #1 () -> () in XCTest._XCTRunThrowableBlock(() throws -> ()) -> XCTest._XCTThrowableBlockResult + 27
frame #16: 0x0000000109c4794e libswiftXCTest.dylib`reabstraction thunk helper from @callee_guaranteed () -> () to @callee_unowned @convention(block) () -> () + 14
frame #17: 0x0000000109c5577e libswiftXCTest.dylib`_XCTRunThrowableBlockBridge + 14
frame #18: 0x0000000109c48deb libswiftXCTest.dylib`function signature specialization <Arg[0] = Owned To Guaranteed> of XCTest._XCTRunThrowableBlock(() throws -> ()) -> XCTest._XCTThrowableBlockResult + 203
frame #19: 0x0000000109c4c575 libswiftXCTest.dylib`function signature specialization <Arg[0] = Owned To Guaranteed, Arg[1] = Owned To Guaranteed, Arg[2] = Owned To Guaranteed, Arg[3] = Exploded> of XCTest.XCTAssertEqual<A where A: Swift.Equatable>(@autoclosure () throws -> A, @autoclosure () throws -> A, @autoclosure () -> Swift.String, file: Swift.StaticString, line: Swift.UInt) -> () + 325
frame #20: 0x0000000109c485e9 libswiftXCTest.dylib`merged XCTest.XCTAssertEqual<A where A: Swift.Equatable>(@autoclosure () throws -> A, @autoclosure () throws -> A, @autoclosure () -> Swift.String, file: Swift.StaticString, line: Swift.UInt) -> () + 41
frame #21: 0x0000000109c47c51 libswiftXCTest.dylib`XCTest.XCTAssertEqual<A where A: Swift.Equatable>(@autoclosure () throws -> A, @autoclosure () throws -> A, @autoclosure () -> Swift.String, file: Swift.StaticString, line: Swift.UInt) -> () + 81
frame #22: 0x000000010437f5cf NIOHTTP1Tests`assertResponseIs(response="", expectedResponseLine="HTTP/1.1 101 Switching Protocols", expectedResponseHeaders=3 values) at HTTPUpgradeTests.swift:91
* frame #23: 0x0000000104399ab6 NIOHTTP1Tests`closure #2 in HTTPUpgradeTestCase.testDelayedUpgradeBehaviour(buffers=0 values, completePromise=NIO.EventLoopPromise<Swift.Void> @ 0x000070000f8390e0) at HTTPUpgradeTests.swift:658
frame #24: 0x0000000104399b4d NIOHTTP1Tests`partial apply for closure #2 in HTTPUpgradeTestCase.testDelayedUpgradeBehaviour() at HTTPUpgradeTests.swift:0
frame #25: 0x000000010434d270 NIOHTTP1Tests`closure #1 in ArrayAccumulationHandler.init(completion=0x0000000104399b40 NIOHTTP1Tests`partial apply forwarder for closure #2 (Swift.Array<NIO.ByteBuffer>) -> () in NIOHTTP1Tests.HTTPUpgradeTestCase.testDelayedUpgradeBehaviour() throws -> () at HTTPUpgradeTests.swift, self=0x000000010234efa0) at HTTPServerClientTest.swift:58
frame #26: 0x000000010434d314 NIOHTTP1Tests`partial apply for closure #1 in ArrayAccumulationHandler.init(completion:) at HTTPServerClientTest.swift:0
frame #27: 0x000000010434d34d NIOHTTP1Tests`thunk for @escaping @callee_guaranteed () -> () at HTTPServerClientTest.swift:0
frame #28: 0x00007fff5a4fc0c9 libdispatch.dylib`_dispatch_client_callout + 8
frame #29: 0x00007fff5a50ca2f libdispatch.dylib`_dispatch_block_invoke_direct + 294
frame #30: 0x000000010434d91f NIOHTTP1Tests`ArrayAccumulationHandler.channelUnregistered(ctx=0x00000001027d2170, self=0x000000010234efa0) at HTTPServerClientTest.swift:67
frame #31: 0x000000010434dc51 NIOHTTP1Tests`protocol witness for _ChannelInboundHandler.channelUnregistered(ctx:) in conformance ArrayAccumulationHandler<A> at HTTPServerClientTest.swift:0
frame #32: 0x0000000104599b20 NIO`ChannelHandlerContext.invokeChannelUnregistered(self=0x00000001027d2170) at ChannelPipeline.swift:1028
frame #33: 0x00000001045932de NIO`ChannelPipeline.fireChannelUnregistered0(self=0x0000000102608b30) at ChannelPipeline.swift:615
frame #34: 0x000000010462ea30 NIO`BaseSocketChannel.close0(error=alreadyClosed, mode=all, promise=nil, self=0x00000001026084a0) at SocketChannel.swift:495
frame #35: 0x00000001046376a4 NIO`SocketChannel.close0(error=alreadyClosed, mode=all, promise=nil, self=0x00000001026084a0) at SocketChannel.swift:929
frame #36: 0x0000000104632047 NIO`protocol witness for ChannelCore.close0(error:mode:promise:) in conformance BaseSocketChannel<A> at SocketChannel.swift:0
frame #37: 0x000000010459cbc9 NIO`HeadChannelHandler.close(ctx=0x0000000102608b80, mode=all, promise=nil, self=0x000000010279ec00) at ChannelPipeline.swift:707
frame #38: 0x000000010459cf2f NIO`protocol witness for _ChannelOutboundHandler.close(ctx:mode:promise:) in conformance HeadChannelHandler at ChannelPipeline.swift:0
frame #39: 0x0000000104596611 NIO`ChannelHandlerContext.invokeClose(mode=all, promise=nil, self=0x0000000102608b80) at ChannelPipeline.swift:1189
frame #40: 0x00000001045966cb NIO`ChannelHandlerContext.invokeClose(mode=all, promise=nil, self=0x00000001027d2170) at ChannelPipeline.swift:1191
frame #41: 0x00000001045943f2 NIO`ChannelPipeline.close0(mode=all, promise=nil, self=0x0000000102608b30) at ChannelPipeline.swift:541
frame #42: 0x00000001045944fe NIO`closure #1 in ChannelPipeline.close(self=0x0000000102608b30, mode=all, promise=nil) at ChannelPipeline.swift:444
frame #43: 0x000000010459b245 NIO`partial apply for closure #1 in ChannelPipeline.close(mode:promise:) at ChannelPipeline.swift:0
frame #44: 0x0000000104561cfc NIO`thunk for @escaping @callee_guaranteed () -> () at Bootstrap.swift:0
frame #45: 0x00000001045bba81 NIO`partial apply for thunk for @escaping @callee_guaranteed () -> () at EventLoop.swift:0
frame #46: 0x00000001045bb827 NIO`thunk for @escaping @callee_guaranteed (@in ()) -> (@out ()) at EventLoop.swift:0
frame #47: 0x00000001045bba21 NIO`thunk for @escaping @callee_guaranteed (@in ()) -> (@out ())partial apply at EventLoop.swift:0
frame #48: 0x00000001045bb847 NIO`closure #2 in SelectableEventLoop.run(task=0x00000001045bba10 NIO`reabstraction thunk helper from @escaping @callee_guaranteed (@in ()) -> (@out ()) to @escaping @callee_guaranteed () -> ()partial apply forwarder with unmangled suffix ".57" at EventLoop.swift) at EventLoop.swift:527
frame #49: 0x00000001045bb8a1 NIO`partial apply for closure #2 in SelectableEventLoop.run() at EventLoop.swift:0
frame #50: 0x0000000104558da6 NIO`thunk for @callee_guaranteed () -> (@error @owned Error) at BlockingIOThreadPool.swift:0
frame #51: 0x00000001045bb90b NIO`thunk for @callee_guaranteed () -> (@error @owned Error)partial apply at EventLoop.swift:0
frame #52: 0x00000001045b5eb2 NIO`closure #1 in withAutoReleasePool<A>(execute=0x00000001045bb8f0 NIO`reabstraction thunk helper from @callee_guaranteed () -> (@error @owned Swift.Error) to @callee_guaranteed () -> (@out (), @error @owned Swift.Error)partial apply forwarder with unmangled suffix ".50" at EventLoop.swift) at EventLoop.swift:261
frame #53: 0x00000001045c33f6 NIO`partial apply for closure #1 in withAutoReleasePool<A>(_:) at EventLoop.swift:0
frame #54: 0x0000000109bb687f libswiftObjectiveC.dylib`ObjectiveC.autoreleasepool<A>(invoking: () throws -> A) throws -> A + 47
frame #55: 0x00000001045b5e28 NIO`withAutoReleasePool<A>(execute=0x00000001045bb8f0 NIO`reabstraction thunk helper from @callee_guaranteed () -> (@error @owned Swift.Error) to @callee_guaranteed () -> (@out (), @error @owned Swift.Error)partial apply forwarder with unmangled suffix ".50" at EventLoop.swift) at EventLoop.swift:260
frame #56: 0x00000001045bada2 NIO`SelectableEventLoop.run(self=0x00000001027d10c0) at EventLoop.swift:526
frame #57: 0x00000001045bdedf NIO`closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(t=(pthread = 0x000070000f83b000), initializer=0x00000001045c2e30 NIO`partial apply forwarder for reabstraction thunk helper from @escaping @callee_guaranteed (@in NIO.Thread) -> (@out ()) to @escaping @callee_guaranteed (@owned NIO.Thread) -> () at EventLoop.swift, lock=(mutex = 0x0000000109d38ff0), _loop=0x00000001027d10c0, loopUpAndRunningGroup=0x0000000109d38e50) at EventLoop.swift:666
frame #58: 0x00000001045c2f0d NIO`partial apply for closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:initializer:) at EventLoop.swift:0
frame #59: 0x00000001045be48f NIO`thunk for @escaping @callee_guaranteed (@owned Thread) -> () at EventLoop.swift:0
frame #60: 0x000000010464cca1 NIO`partial apply for thunk for @escaping @callee_guaranteed (@owned Thread) -> () at Thread.swift:0
frame #61: 0x000000010464bcf8 NIO`closure #1 in static Thread.spawnAndRun(p=(_rawValue = 0x0000000109d37040 -> 0x00000001095b86a0 libswiftCore.dylib`InitialAllocationPool + 13968)) at Thread.swift:105
frame #62: 0x000000010464bd79 NIO`@objc closure #1 in static Thread.spawnAndRun(name:body:) at Thread.swift:0
frame #63: 0x00007fff5a851642 libsystem_pthread.dylib`_pthread_body + 126
frame #64: 0x00007fff5a8515c4 libsystem_pthread.dylib`_pthread_start + 61
frame #65: 0x00007fff5a851229 libsystem_pthread.dylib`thread_start + 13
We should investigate why this is happening, but it seems to just be a problem in the test code itself: probably a race.
Seriously. An Apple project w/o a Logo? Maybe some swallow crashing into a S/390 or something? Or a bug running around in an Xserve? Neo from Matrix? Something!
When the ChatClient sends a message, other clients see
(##remoteAddress##) - ##message##
The ChatServer force-unwraps the remoteAddress field, encountering nil, and crashes.
Modifying the above line referenced in part 4 to the following produces what appears to be expected behavior. However, I am unclear as to the difference between ctx.channel.remoteAddress and ctx.remoteAddress.
if let remoteAddress = ctx.remoteAddress {
buffer.write(string: "(\(remoteAddress)) - ")
} else {
buffer.write(string: "(no address) - ")
}Provided ChatServer and ChatClient exhibit this behavior
2c1d993 (latest as this issue)
swift --version && uname -a)Apple Swift version 4.0.3 (swiftlang-900.0.74.1 clang-900.0.39.2)
Target: x86_64-apple-macosx10.9
Darwin Ezekiels-MacBook-Pro-133.local 17.5.0 Darwin Kernel Version 17.5.0: Sun Feb 11 23:54:24 PST 2018; root:xnu-4570.50.279~25/RELEASE_X86_64 x86_64
No warnings, clean compile.
A set of warnings, no clean compile.
Compile Swift Module 'NIO' (47 sources)
/Users/helge/dev/Swift/SwiftXcode/Images/SwiftNIO_XcodeImage/.build/checkouts/swift-nio.git-3108475404973543938/Sources/NIO/EventLoopFuture.swift:316:25: warning: will never be executed
if let me = eventLoop as? SelectableEventLoop {
^
/Users/helge/dev/Swift/SwiftXcode/Images/SwiftNIO_XcodeImage/.build/checkouts/swift-nio.git-3108475404973543938/Sources/NIO/EventLoopFuture.swift:315:12: note: condition always evaluates to false
if _isDebugAssertConfiguration() {
^
/Users/helge/dev/Swift/SwiftXcode/Images/SwiftNIO_XcodeImage/.build/checkouts/swift-nio.git-3108475404973543938/Sources/NIO/EventLoopFuture.swift:338:64: warning: will never be executed
if _isDebugAssertConfiguration(), let eventLoop = self.eventLoop as? SelectableEventLoop {
^
/Users/helge/dev/Swift/SwiftXcode/Images/SwiftNIO_XcodeImage/.build/checkouts/swift-nio.git-3108475404973543938/Sources/NIO/EventLoopFuture.swift:338:12: note: condition always evaluates to false
if _isDebugAssertConfiguration(), let eventLoop = self.eventLoop as? SelectableEventLoop {
^
Compile Swift Module 'NIOTLS' (3 sources)
Compile Swift Module 'NIOHTTP1' (7 sources)
Compile Swift Module 'NIOOpenSSL' (11 sources)
Compile Swift Module 'MicroExpress' (7 sources)
/Users/helge/dev/Swift/SwiftXcode/Images/SwiftNIO_XcodeImage/.build/checkouts/MicroExpress.git-6340581023861930988/Sources/MicroExpress/Express.swift:23:26: warning: 'addHTTPServerHandlers(first:)' is deprecated: Please use configureHTTPServerPipeline
channel.pipeline.addHTTPServerHandlers().then {
^
swift build -c release
1.2.1
swift --version && uname -a)Apple Swift version 4.0.3 (swiftlang-900.0.74.1 clang-900.0.39.2)
Target: x86_64-apple-macosx10.9
Darwin ZeaPro.local 16.7.0 Darwin Kernel Version 16.7.0: Thu Jan 11 22:59:40 PST 2018; root:xnu-3789.73.8~1/RELEASE_X86_64 x86_64
In certain cases nicer APIs can be created if there's a public way to get the current EventLoop from within a SwiftNIO app. We should consider doing that. The most straight-forward implementation is obviously to just store in in a ThreadSpecificVariable.
The current implementation of hasRequestBody uses unlikely for a lot of cases where a body is actually very likely or a plain .yes (not sure what the different is, a PUT can also have an empty body).
Examples MKCOL, PROPFIND, REPORT, MKCALENDAR, etc.
Users generally expect Swift NIO not to fail with internal assertions and fatalErrors they couldn't catch.
Fatal error: Invalid FSM transition attempt: state complete, input connectTimeoutElapsed: file swift-nio/Sources/NIO/HappyEyeballs.swift, line 417
Illegal instruction: 4
Compile and run the attached code against the master.
b8d883da2c2090d02ec4890b5ab5c7c9bfa30935
swift --version && uname -a)Apple Swift version 4.1 (swiftlang-902.0.38 clang-902.0.30)
Target: x86_64-apple-darwin17.4.0
Darwin Levs-MacBook-Pro.local 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64
[what you expected to happen]
[what actually happened]
[anything to help us reproducing the issue]
[the SwiftNIO tag/commit hash]
swift --version && uname -a)We should consider making ThreadSpecificVariable public.
Related to #60 (consider having an API to allow attaching metadata to an EventLoop)
The Channel callbacks lifecycle should start with channelActive and end with channelInactive.
Sometimes under high concurrency the channelRead is invoked prior to channelActive.
Run the attached code.
The code fails in one of three ways:
q.sync)Precondition failed: channelRead before channelActive!: file StuckTest/RemoteSystem.swift, line 224
Illegal instruction: 4
The first two failures might not be relevant, but it warrants to take a look. This particular issue is mainly about failure number 3, Precondition failed.
681ddd8bab4fac997cbc70ca15a1e13c74da50b9 (nmaurer/order_fix)
swift --version && uname -a)Apple Swift version 4.1 (swiftlang-902.0.38 clang-902.0.30)
Target: x86_64-apple-darwin17.4.0
Darwin Levs-MacBook-Pro.local 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64
Setting up the local image from the Dockerfile produces quite a few errors, looks as if mostly related to not having an interactive terminal. For example:
---> 53ee6818e23b
Step 4/39 : RUN dpkg-reconfigure locales
---> Running in a4728f63b96c
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
Generating locales...
en_US.ISO-8859-1... up-to-date
Removing any system startup links for /etc/init.d/rsync ...
update-rc.d: warning: default stop runlevel arguments (0 1 6) do not match rsync Default-Stop values (none)
Adding system startup for /etc/init.d/rsync ...
/etc/rc0.d/K20rsync -> ../init.d/rsync
...
invoke-rc.d: policy-rc.d denied execution of restart.
There are quite a lot of those errors, maybe around 20-30 (showing up red in the macOS terminal).
docker-compose -f docker/docker-compose.yaml up testdocker-compose -f docker/docker-compose.yaml up test
This is on the helje5:bugs/fix-issue-110 fork branch, hash b2920b6, but that shouldn't matter.
swift --version && uname -a)Apple Swift version 4.0.3 (swiftlang-900.0.74.1 clang-900.0.39.2)
Target: x86_64-apple-macosx10.9
Darwin ZeeMBP 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64
Docker: Version 17.12.0-ce-mac55 (23011), Channel: stable, d62ef8d1b0
A lot of the EventLoopFuture code is quite prone to subtle correctness bugs. This happens for a bunch of reasons, most of them relating to the way the code has grown over time. While we've patched over them a bit, we should really take a step to clean this code up more profoundly, ideally one method at a time.
Here are some things we should aim to do:
Make map faster.
Right now map performs worse than many users would expect. This is because map is basically then with a call to eventLoop.newSucceededFuture. This means that a call to map forces allocation of a new EventLoopPromise and a new EventLoopFuture. While this is appealing from the perspective of correctness ("well, map is really just the combination of monadic uplift and bind, blah blah blah haskell") it discourages our internal code from using it for performance reasons.
This allocation is fundamentally unnecessary, and a better internal API could draw a distinction between these two behaviours to improve the performance of map.
Define a better internal API.
Quite a few methods try to be fast by skipping the external API, not least because map is slow. This is problematic, because the internal API exposes all the warty insides of these types ("what's a CallbackList? Why do I need to return one? How is then implemented, and how do I approximate it?"). We should clean these interfaces up to make it easier to be confident that the code that uses them is correct. In particular, the way they interact with the thread-hopping behaviour of EventLoopFuture needs to be made extremely clear, so that we can validate that code that uses them is actually safe.
More utility functions.
While we have and<T, U> -> EventLoopFuture<(T, U)> and andAll<Void> -> EventLoopFuture<Void>, there are a few other primitives we should probably provide. In particular, it's easy enough to extend andAll<Void> to andAll<T> -> EventLoopFuture<[T]>, as well as provide a variant of andAll that fails slow instead of failing fast (e.g. collate<T> -> EventLoopFuture<[Result<T>]>).
Any other suggestions?
Currently the HTTP performance is not as good as we have hoped for.... This may be caused by multiple things:
We should look into what we can do to make it faster.
When asking getaddrinfo about an IP address, we should not get two results that contain the same IP address.
we do get two, example here:
(lldb) p info.pointee
(addrinfo) $R13 = {
ai_flags = 0
ai_family = 2
ai_socktype = 2
ai_protocol = 17
ai_addrlen = 16
ai_canonname = nil
ai_addr = 0x0000000104904ab0
ai_next = 0x0000000104904f60
}
(lldb) p info.pointee.ai_next.pointee
(addrinfo) $R14 = {
ai_flags = 0
ai_family = 2
ai_socktype = 1
ai_protocol = 6
ai_addrlen = 16
ai_canonname = nil
ai_addr = 0x0000000104904f90
ai_next = nil
}
whereby ai_protocol = 17 is IPPROTO_UDP and ai_protocol = 6 is IPPROTO_TCP. But since we know which one we want we should query correctly.
127.0.0.1private func parseResults(_ info: UnsafeMutablePointer<addrinfo>, host: String)bootstrap.connect(host: "127.0.0.1", port: 1234)
1.2.0
swift --version && uname -a)$ uname -a
Darwin jwmbp.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Jan 22 18:03:58 PST 2018; root:xnu-4570.50.252~1/DEVELOPMENT_X86_64 x86_64
$ swift --version
Apple Swift version 4.0.3 (swiftlang-900.0.71 clang-900.0.38)
Target: x86_64-apple-macosx10.9
currently we have
private let localAddressCached: AtomicBox<Box<SocketAddress?>> = AtomicBox(value: Box(nil))
which is quite bad as it's a box in a box. Given that SocketAddress already has its private box, we might want to make that private box internal and use it here :).
Btw, AtomicBox also has one level of indirection more than it should have.
It is likely I'm missing something, but here we go:
I should be able to get more than 6 concurrent requests from NIOHTTP1Server
I'm tracking concurrent requests like so:
https://gist.github.com/GeorgeLyon/df6f34c7260997f5c373ac9a1e81c9a7
If I spam the server with requests to /write-delay, my max stays at 6. I would like that to be much higher.
It seems to me that NIO is allowing 1 active channel per core only. Do I need to do something special to "suspend" the Channel for the duration of the long running task so as to let the next request through? I didn't see anything in ChannelOptions, ServerBootstrap or ChannelHandlerContext to adjust this.
swift --version && uname -a)Apple Swift version 4.1 (swiftlang-902.0.43 clang-902.0.37.1)
Target: x86_64-apple-darwin17.4.0
Darwin Georges-MacBook-Pro-2.local 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64
The traditional UNIX signal model is not really ideal for programs like NIO, where signal handlers are basically incapable of doing any real work that affects the event loops. Signal handlers are not able to obtain locks or do basically anything that may require that other code on the thread runs, so in a NIO program a signal handler can basically do nothing of interest: it can't even call malloc.
This is a bad scene, particularly in Swift, where there is very little Swift code that you can confidently run from a signal handler (no malloc, remember!). Given that interacting with signals is a common thing to want to do for servers (e.g. SIGHUP for reloading config), we should build an abstraction that allows NIO-using libraries to handle signals more gracefully. Naturally this will need to be opt-in, but that's ok.
The easiest way to do this is to implement the anonymous self-pipe trick, the standard solution to the problem (TL;DR at program start you create an anonymous pipe that you select on in your event loop, and in a signal handler you write a byte to the pipe and return, allowing you to handle the signal in the loop instead). An alternative option would be to spawn a thread dedicated to signal handling, though that may be a bit excessive (and it requires masking away the signals on all the other threads which is a PITA).
Regardless, we should at some point sit down and provide this abstraction.
No API breakage if an HTTP method is added to the enum.
API breakage if an HTTP method is added to the enum.
1.1
As long as Swift doesn't provide an open enum construct (which I think was in discussion, don't know the state of it), it is not wise to use a (closed) enum for constants which are unstable (in an API which is supposed to be stable). In this case, HTTP methods are added once in a while by new RFCs (I myself added MKCALENDAR to http_parser).
This was also discussed a lot in the HTTP WG effort, where we finally ended up with the common iOS solution: https://github.com/swift-server/http/blob/develop/Sources/HTTP/HTTPMethod.swift#L10 (I'm not sure this is the best solution, probably not, but it at least provides better API stability).
In short: Adding a new case to HTTPMethod currently requires a major SemVer bump. That is non-ideal as the addition of a HTTP methods is otherwise not a big event / breaking change ;-)
When a ByteToMessageDecoder calls ctx.close() during its processing, and that close completes synchronously, receiving channelInactive will not cause ByteToMessageDecoder to re-parse previously parsed data.
When this happens, ByteToMessageDecoder.channelInactive calls decodeLast with a fresh copy self.cumulationBuffer. By default, decodeLast is the same as decode, so this essentially calls decode in almost all cases. This means that the call stack ends up looking like this:
decode()ctx.close()decodeLast()decode()Worse, this second call to decode does it with an out of date copy of the cumulationBuffer, which it necessarily must do because we always operate on a copy of the cumulationBuffer and save it off after our calls to decode.
final class BufferLengthReporter: ByteToMessageDecoder {
typealias InboundIn = ByteBuffer
typealias InboundOut = Int
var cumulationBuffer: ByteBuffer?
func decode(ctx: ChannelHandlerContext, buffer: inout ByteBuffer) throws -> DecodingState {
ctx.fireChannelRead(self.wrapInboundOut(buffer.readableBytes))
buffer.moveReaderIndex(forwardBy: 4)
ctx.close(promise: nil)
return .needMoreData
}
func decodeLast(ctx: ChannelHandlerContext, buffer: inout ByteBuffer) throws -> DecodingState {
ctx.fireChannelRead(self.wrapInboundOut(buffer.readableBytes))
return .needMoreData
}
}
class Test: XCTestCase {
func testDontRepeatedlyParse() throws {
let channel = EmbeddedChannel()
defer {
_ = try? channel.finish()
}
XCTAssertNoThrow(try channel.pipeline.add(handler: BufferLengthReporter()).wait())
var buffer = channel.allocator.buffer(capacity: 5)
buffer.write(staticString: "hello")
XCTAssertTrue(try channel.writeInbound(buffer))
XCTAssertFalse(channel.isActive)
XCTAssertEqual(5, channel.readInbound())
XCTAssertEqual(1, channel.readInbound())
XCTAssertEqual(nil as Int?, channel.readInbound())
}
}The second XCTAssertEqual in the final block fails, reporting 5 instead of 1.
1.1.0
decodeLast.decodeLast is a strange function. It exists for protocols for whom EOF is a meaningful signal, but those protocols are not that common. More importantly, in the case of those protocols they could simply implement channelInactive themselves to handle the case. Right now, for the default protocol case, decodeLast is simply a waste of CPU: the protocol couldn't decode the buffer before, no reason to assume it can now.
The advantage of removing decodeLast is that it becomes clear that this is a thing users need to handle themselves. The disadvantage is that it doesn't solve this problem for handlers that treat EOF as meaningful: they still need a way to work out whether they're synchronously receiving EOF from the remote peer.
decodeLast.A smaller-scale version of (1), this would just change decodeLast to be a no-op. This avoids the current problem, but still provides the hook. The downside of this is, again, we end up letting users fall back into this trap.
ByteToMessageDecoder.This is probably my preferred option, but it makes B2MD substantially more complex. Ultimately the core issue is that B2MD will eagerly dispatch channelInactive, which means that it's partway through parsing when that message is processed. That's just a bad idea. So we could add state to B2MD to record that it has seen channelInactive, but to not dispatch it while we're inside decode. Instead, we'd let decode return and then spot the change, and transition to calling decodeLast.
This is likely to behave better in the long-term.
channelInactive firing synchronously.This is not a great fix, but most users do not realise that the moment they call out to third-party code they are at risk of being re-entrantly called by all kinds of events. We could, in principle, prevent close re-entrantly firing channelInactive, but frankly I don't think that's a very good idea. I just wanted to note it here.
All-in-all I think I prefer (3), but I'd like to hear what others think.
There doesn't seem to be anything specific to blocking I/O in this and I don't want to create yet another thread pool for other synchronous operations, but just reuse this.
Discussions made it clear that the CoW API of ByteBuffer means that slice cannot be used to provide a writable window into a larger ByteBuffer. This is because any attempt to write into the ByteBuffer produced by slice will cause a CoW, leading to the write happening in a new buffer.
We should add a new method (or family of methods), tentatively called withWritableBufferView, that will provide a temporary window into the ByteBuffer's underlying storage that escapes the CoW implementation. This can probably only be done by wrapping an UnsafeMutablePointer to the inner buffer, which is why it seems sensible to make this a with... method: it ensures that the lifetime of the writable buffer view is strictly controlled.
All tests in the integration Test Suite to pass.
1 Failing integration test:
test_16_tcp_client_ip.sh
Logs:
Running test 'test_16_tcp_client_ip.sh'... FAILURE (0)
--- OUTPUT BEGIN ---
+ set -o pipefail
+ test=/Users/martinho/Development/swift-nio/IntegrationTests/tests_01_http/test_16_tcp_client_ip.sh
+ tmp=/tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t
+ root=/Users/martinho/Development/swift-nio/IntegrationTests/..
+++ dirname /Users/martinho/Development/swift-nio/IntegrationTests/run-single-test.sh
++ cd /Users/martinho/Development/swift-nio/IntegrationTests
++ pwd
+ here=/Users/martinho/Development/swift-nio/IntegrationTests
+ source /Users/martinho/Development/swift-nio/IntegrationTests/test_functions.sh
+ source /Users/martinho/Development/swift-nio/IntegrationTests/tests_01_http/test_16_tcp_client_ip.sh
++ source defines.sh
+++ create_token
+++ mktemp /tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/server_token_XXXXXX
++ token=/tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/server_token_hmM3vi
++ start_server /tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/server_token_hmM3vi tcp
++ local token=/tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/server_token_hmM3vi
++ local type=--uds
++ local port=/tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/port.sock
++ local tmp_server_pid
++ local tok_type=--unix-socket
++ local curl_port=80
++ maybe_host=
++ maybe_nio_host=
++ [[ tcp == \t\c\p ]]
++ type=
++ port=0
++ tok_type=
++ maybe_host=localhost
+++ host -4 -t a localhost
+++ tr ' ' '\n'
+++ tail -1
++ maybe_nio_host='3(NXDOMAIN)'
--- OUTPUT END ---
1.0.0 / 5ff6d9b
swift --version && uname -a)Apple Swift version 4.0.3 (swiftlang-900.0.74.1 clang-900.0.39.2)
Target: x86_64-apple-macosx10.9
Darwin Tiagos-MacBook-Pro-2.local 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64
Swift NIO reuses the official Swift Docker image instead of brewing its own soup.
The Swift NIO Docker builds its own environment, with all the dependencies etc.
docker build
First of all, people w/ Docker on the Mac, quite likely already have the image downloaded and can get started quicker.
Second, you use a consistent environment for Setty which other people already use. You'll catch issues you may not be seeing.
I don't know if you need anything extra to what the official image provides. If not, it would also result in a reproducible build/test environment. (If yes, you may want to push an own repo based on swift:4.0.3).
You can find it over here, it even says "OFFICIAL REPOSITORY" 😁: https://hub.docker.com/_/swift/
No 0 bytes in strings
0 bytes in strings
In Xcode:
print("Server running on:", serverChannel.localAddress!)It says:
Server running on: [IPv6]::1<NUL><NUL><NUL>....lots...
Some editors seem to eat the \0's.
I think it is
private func descriptionForAddress(family: CInt, bytes: UnsafeRawPointer, length byteCount: Int) -> String {
...
return String(decoding: UnsafeBufferPointer<UInt8>(start: addressBytesPtr, count: byteCount), as: Unicode.ASCII.self)That leaves the 0's from the address buffer in the String. I think you probably want some
return String(cString: addressBytesPtr)
variant here, so that the string terminates at the \0.
Other potential causes: incorrect indentation, 80-column source line length overflows all over.
1.1
swift --version && uname -a)Apple Swift version 4.0.3 (swiftlang-900.0.74.1 clang-900.0.39.2)
Target: x86_64-apple-macosx10.9
Darwin ZeaPro.local 16.7.0 Darwin Kernel Version 16.7.0: Thu Jan 11 22:59:40 PST 2018; root:xnu-3789.73.8~1/RELEASE_X86_64 x86_64
Sorry, if this question is dumb, because the whole purpose of the object may be to allow exactly this, but the documentation does not specifically clarify this.
Is it safe to write from one thread and read from another?
I expect there not to be so many futex calls.
916 20:12:30 epoll_create(128) = 3 <0.000055>
916 20:12:30 futex(0x667b10, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000023>
916 20:12:30 futex(0x667b20, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000033>
916 20:12:30 eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK) = 4 <0.000064>
916 20:12:30 futex(0x667af0, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000032>
916 20:12:30 futex(0x667b00, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000022>
916 20:12:30 timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC|TFD_NONBLOCK) = 5 <0.000058>
916 20:12:30 futex(0x667b60, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000022>
916 20:12:30 futex(0x667b80, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000033>
916 20:12:30 futex(0x667b90, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000032>
916 20:12:30 futex(0x667b30, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000023>
916 20:12:30 epoll_ctl(3, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLERR|EPOLLRDHUP, {u32=4, u64=4}}) = 0 <0.000068>
916 20:12:30 futex(0x667ba0, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000020>
916 20:12:30 epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLIN|EPOLLERR|EPOLLRDHUP|EPOLLET, {u32=5, u64=5}}) = 0 <0.000053>
916 20:12:30 futex(0x6686c8, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000068>
916 20:12:30 futex(0x6680b0, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000056>
916 20:12:30 futex(0x7f81323e9530, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000064>
916 20:12:30 futex(0x7f81323e9928, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000068>
docker run --security-opt seccomp:unconfined -v$CWD:/code -it --rm swiftniocd /code && swift buildstrace -s 1024 -fF -tT -o trace.out ./.build/x86_64-unknown-linux/debug/NIOHTTP1Server 0.0.0.0 8888 .There are no futex calls here:
root@1ba3a0b6d2c0:/code# cat epoll.c
#include <sys/epoll.h>
int main() {
int fd = epoll_create(1);
epoll_wait(fd, 0, 0, 0);
return 0;
}
root@1ba3a0b6d2c0:/code# clang epoll.c
root@1ba3a0b6d2c0:/code# strace -s 1024 -f -tT ./a.out
21:46:15 execve("./a.out", ["./a.out"], [/* 11 vars */]) = 0 <0.001591>
21:46:15 brk(NULL) = 0x1c96000 <0.000028>
21:46:15 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) <0.000200>
21:46:15 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) <0.000012>
21:46:15 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 <0.000253>
21:46:15 fstat(3, {st_mode=S_IFREG|0644, st_size=18870, ...}) = 0 <0.000017>
21:46:15 mmap(NULL, 18870, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f74ce3e6000 <0.000023>
21:46:15 close(3) = 0 <0.000015>
21:46:15 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) <0.000036>
21:46:15 open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 <0.000022>
21:46:15 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\t\2\0\0\0\0\0@\0\0\0\0\0\0\0\270r\34\0\0\0\0\0\0\0\0\0@\0008\0\n\0@\0H\0G\0\6\0\0\0\5\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0000\2\0\0\0\0\0\0000\2\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0`f\31\0\0\0\0\0`f\31\0\0\0\0\0`f\31\0\0\0\0\0\34\0\0\0\0\0\0\0\34\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\373\33\0\0\0\0\0\20\373\33\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0\300\7\34\0\0\0\0\0\300\7<\0\0\0\0\0\300\7<\0\0\0\0\0`O\0\0\0\0\0\0\340\221\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\240;\34\0\0\0\0\0\240;<\0\0\0\0\0\240;<\0\0\0\0\0\340\1\0\0\0\0\0\0\340\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0D\0\0\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\7\0\0\0\4\0\0\0\300\7\34\0\0\0\0\0\300\7<\0\0\0\0\0\300\7<\0\0\0\0\0\20\0\0\0\0\0\0\0x\0\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0|f\31\0\0\0\0\0|f\31\0\0\0\0\0|f\31\0\0\0\0\0\274T\0\0\0\0\0\0\274T\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0\300\7\34\0\0\0\0\0\300\7<\0\0\0\0\0\300\7<\0\0\0\0\0@8\0\0\0\0\0\0@8\0\0\0\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\2658\32Ey\6\322y\0078\"\245\316\262LK\376\371M\333\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\2\0\0\0\6\0\0\0 \0\0\0\0\0\0\0\363\3\0\0\n\0\0\0\0\1\0\0\16\0\0\0\0000\20D\240 \2\1\210\3\346\220\305E\214\0\304\0\10\0\5\204\0`\300\200\0\r\212\f\0\4\20\0\210@2\10*@\210T<, \0162H&\204\300\214\4\10\0\2\2\16\241\254\32\4f\300\0\3002\0\300\0P\1 \201\10\204\v ($\0\4 Z\0\20X\200\312DB(\0\6\200\20\30B\0 @\200\0IP\0Q\212@\22\0\0\0\0\10\0\0\21\20", 832) = 832 <0.000024>
21:46:15 fstat(3, {st_mode=S_IFREG|0755, st_size=1868984, ...}) = 0 <0.000013>
21:46:15 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f74ce3e5000 <0.000111>
21:46:15 mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f74cddfc000 <0.000057>
21:46:15 mprotect(0x7f74cdfbc000, 2097152, PROT_NONE) = 0 <0.000041>
21:46:15 mmap(0x7f74ce1bc000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c0000) = 0x7f74ce1bc000 <0.000044>
21:46:15 mmap(0x7f74ce1c2000, 14752, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f74ce1c2000 <0.000019>
21:46:15 close(3) = 0 <0.000013>
21:46:15 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f74ce3e4000 <0.000020>
21:46:15 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f74ce3e3000 <0.000011>
21:46:15 arch_prctl(ARCH_SET_FS, 0x7f74ce3e4700) = 0 <0.000159>
21:46:15 mprotect(0x7f74ce1bc000, 16384, PROT_READ) = 0 <0.000030>
21:46:15 mprotect(0x600000, 4096, PROT_READ) = 0 <0.000068>
21:46:15 mprotect(0x7f74ce3eb000, 4096, PROT_READ) = 0 <0.000024>
21:46:15 munmap(0x7f74ce3e6000, 18870) = 0 <0.000019>
21:46:15 epoll_create(1) = 3 <0.000044>
21:46:15 epoll_wait(3, NULL, 0, 0) = -1 EINVAL (Invalid argument) <0.000015>
21:46:15 exit_group(0) = ?
21:46:15 +++ exited with 0 +++
swift --version && uname -a)Swift version 4.0 (swift-4.0-RELEASE)
Target: x86_64-unknown-linux-gnu
Linux 1ba3a0b6d2c0 4.9.60-linuxkit-aufs #1 SMP Mon Nov 6 16:00:12 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
It would be better if there is usage examples for
NIOChatClient
NIOEchoClient
NIOHTTP1Server
NIOEchoServer
I am interested in using SwiftNIO's HTTP2 implementation (once available) to add an NIO-based server implementation to https://github.com/grpc/grpc-swift. Given that that project currently depends on several other fairly large libraries, it would be great to also move the client code to SwiftNIO to eliminate those dependencies.
For that to happen, though, SwiftNIO would need to support those client platforms — namely iOS (preferably 10+) and macOS 10.11. Neither currently shows up in the "Supported Platforms" list — does that mean it's technically impossible to support those platforms (e.g. because some low-level networking APIs are missing), or has the library simply not been tested on those platforms, but might already work as-is? Other than that, what level of effort would be required to add support for these two platforms, or is it already planned, and with what timeline?
The included BackPressureHandler only applies to a non-existing use case and gives a false sense of back-pressure-security. It is DoS-dangerous and should rather be dropped completely, or at least renamed to something like PassThroughBackPressureHandler. It should not be advertised to be used, only with very specific instructions when it actually applies (like echo ;-) ).
The idea of a back pressure is that you don't read more data than what the stack can process. I.e. that you stop reading data you can't process anyways at the socket level. This is particularly important with async services because they don't block the reading side during processing (BP is automatic/implicit w/ synchronous services).
The included BackPressureHandler ties readability to the writability of the socket, which is non-sensical except for the super rare use case of a pure "transforming" aka "echo" service. Aka "don't read more unless I can write".
What makes it so bad is that the current setup suggests that you have proper BP handling, when in fact you don't have any at all.
In async services BP is a must have all over. For the common case the socket will stay writable all the time, which makes the BP included useless. Only if the processing completes, a service will issue a 200 OK. Maybe sometimes it will issue a 100 Continue to keep the socket alive.
Lets imagine you enable iCloud on your device for the first time. This will result in an upload of your vCards and iCalendars in bulk.
Overly simplified it will be a POST-2000-vCards, and a POST-5000-iCalendar-Events.
Ignoring watermarks and such to facilitate batch processing (which NIO doesn't have), imagine this as:
Nothing is every written to the socket-out (except maybe the keep-alive 100-continues). Which means the included BP handler will never kick in.
In the current setup a service would accumulate all 2000 vCards in memory (schedule them to be inserted), w/o considering database insert performance (which will lag a lot).
Pretty much everything which involves outgoing I/O. Say fetching weather info or stock prices. In other words, which makes async stuff async ;-)
The BP must not need to be tied to the client-socket writability, but to the database-socket writability. I.e. can the DB keep up w/ inserting the records.
I hope this makes sense and isn't too confusing (and god I hope I'm not talking non-sense and missed something which actually makes it work as desired ;-) ).
FWIW I'm not advocating that NIO does BP, it should probably be done at the layer above. My suggestion is that "The BackPressure handler should be dropped" as in the title. It is pretty much useless and very much misleading.
The current mode for using channel writability thresholds looks like this:
try firstChannel.setOption(option: ChannelOptions.writeBufferWaterMark, value: 0..<1024)Under the covers this uses .upperBound and .lowerBound to manage the writability. Specifically, the channel is marked "not writable" when the buffered byte count is > .upperBound, and marked writable again when it drops < .lowerBound.
I found this behaviour to be moderately surprising for two reasons. Firstly, there's this:
1> (0..<1024).upperBound
$R0: Int = 1024
2> (0...1024).upperBound
$R1: Int = 1024
This means that for either a closed or semi-open range Swift reports the same upper bound. Frankly this behaviour is so surprising I kinda want to call it a bug in Swift, but as a user I'd expect that if I typed 0..<1024 that we'd mark the channel not writable when we write the 1024'th byte, not the 1025th.
The bigger problem is that in both of these cases, once the channel is marked not writable it will never be marked writable again. This is definitely bad, and if we think that behaviour makes sense for this kind of range then we should forbid ranges that are zero-bounded at the bottom. However, I'd argue that we should do a <= check instead of a < check for the lower bound.
Alternatively, we may just want to chuck this slightly confusing interface altogether and instead split the option into two: a lower threshold and an upper one. We can then define the semantic for each threshold directly, rather than have users see a range and expect they know what it does.
Regardless, at the barest minimum we should either do some sanity checking of the range the user passes us or change the type definition to Range<UInt> because right now we also allow this:
try firstChannel.setOption(option: ChannelOptions.writeBufferWaterMark, value: -100..<(-50))And that's obviously totally 🍌.
Thank you for this library, this is some great work.
I was wondering if there is any plan to work on a gRPC library in Swift, built on top of swift-nio. I'd be more than interested to contribute and wanted to check if gRPC was part of some roadmap, directly (or undirectly) related to swift-nio and this project.
Thank you.
At the moment, it's too hard to make HTTP/1.1 pipelining work correctly. SwiftNIO just forwards you the HTTP request parts as they come in and the user needs to make sure everything works correctly with pipelining which is hard (even our example NIOHTTP1Server doesn't do that (see #9)).
We should offer a HTTP1PipeliningHandler which forwards the n-th request only if the (n-1)th request has been completed. That way the user can basically forget about pipelining which is a good thing :)
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.