Missing way to read HttpRequest body/data for POST/PUT... about uwebsockets

[christoth]

I'll test on linux.

It was working earlier with read() on linux when the payload was small and I just res->end(data) as above (without multiple calls to read()).

But it consistently crashed when I loadtested POST for 20 seconds with 10 concurrent connections.

Console output:
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted

P.S. The same loadtest with GET worked flawlessly

This is one way of doing it (however this can be improved for small posts)

    }).post("/file", [](auto *res, auto *req) {
        /* Allocate automatic, stack, variable as usual */
        std::string buffer;
        /* Move it to storage of lambda */
        res->onData([res, buffer = std::move(buffer)](std::string_view data, bool last) mutable {
            /* Mutate the captured data */
            buffer.append(data.data(), data.length());

            if (last) {
                /* Use the data */
                std::cout << "We got all data, length: " << buffer.length() << std::endl;

                us_listen_socket_close(listen_socket);
                res->end("Thanks for the data!");
                /* When this socket dies (times out) it will RAII release everything */
            }
        });
        /* Unwind stack, delete buffer, will just skip (heap) destruction since it was moved */
    }).

[brandery]

@mfbalin I also met this issue, but solved it by providing a onAborted handler at the same time.

res->read([](std::string_view data) {

});

No response closing

[christoth]

Thanks for pointing me in the right direction.

Is there a way to tell when the request data is finished? e.g. zero length data

I'm currently reading the content-length header, appending data to by own buffer and comparing the length of the buffer to content-length.

Are you open to me adding an echo HTTP example (using POST and returning the request body) either on it's own or to the echo server example?

Thanks again

I added a bool "last" and renamed it to onData. I've tested this on Windows and it seems to work. Probably should add more helper functions later on.

Thanks for reporting

.post("/file", [](auto *res, auto *req) {
        res->onData([res](std::string_view data, bool last) {
            std::cout << "Got data: " << data.length() << ", it's the last: " << last << std::endl;

            if (last) {
                us_listen_socket_close(listen_socket);
                res->end("Thanks for the data!");
            }
        });

[christoth]

Using onData works with single requests.

When I test with concurrent requests it times-out, but it doesn't crash now, which is great and it will respond to single requests after the short loadtest - previously with read() it just crashed.

It looks like concurrent requests trigger onData(), but last is never true?

Here's my code:

    }).post("/echo", [](auto *res, auto *req) {
      const uint64_t contentLength = std::stoi(std::string(req->getHeader("content-length")));
      std::cout << ">: " << contentLength << " ";
      if(contentLength == 0) {
        res->end("'/echo' request needs a payload");
        std::cout << "contentLength == 0" << std::endl;
        return;
      }

      std::string requestData;

      res->onData([&](std::string_view data, bool last) {
        if (last) {
          if (requestData.size() == 0) {
            res->end(data);
            std::cout << "<: " << data.length() << std::endl; 
          } else {
            requestData.append(data.data(), data.size());
            res->end(requestData);
            std::cout << "[" << data.size() << "] <: " << requestData.length() << std::endl;
          }
          return;
        } else {
          if (requestData.length() == 0) {
            requestData.reserve(contentLength);
          }
          requestData.append(data.data(), data.size());
          std::cout << "[" << data.size() << "] ";
        }
      });
    })

It shouldn't crash now or before, do you have code that crashes it and the lib version/commit?

[ddevienne]

A bit OT, but I notice the example uses C++17-only std::string_view.

I thought wWebSocket was C++11, but the doc of latest 0.15 code explicitly states C++17 now.
Since I was planning on using uWebSocket, waiting for 0.15, and I'm on C++14, that's a bit of a bummer to me...

Are there other C++17-only features beside std::string_view used?
Any chance the code could be downgraded to C++11 or C++14 via a compile-time define?
I'm happy to do my in a private fork, provided you think it's not too difficult, and possibly point out the kind of changes necessary. Thanks, --DD

I don't know exactly but there are other features used. v0.14 was C++11, v0.15 is C++17.

The auto [lala] = thing, the if constexpr, the std::string_view, auto in lambdas (maybe that's 14?), etc. Probably possible to downgrade but even easier to just upgrade your compiler. I know some users complain it's not C++98

[ddevienne]

Thanks for the feedback. Destructuring assignement, "static if" definitely C++17.
auto in lambdas in C++14 indeed. It's your library, choose your C++ standard :).
I just can't use it now, unfortunately. In Corp env, I don't own the "platform" I code on...

[christoth]

I'm cloning both uWebSockets and uSockets repos into a Docker container (using the gcc:latest image)

Library versions:

root@a1572b50f18e:/code/uWebSockets/examples# ldconfig -p|grep ssl
libssl.so.1.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libssl.so.1.1
libssl.so.1.0.2 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
libssl.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libssl.so
libevent_openssl-2.0.so.5 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libevent_openssl-2.0.so.5

root@a1572b50f18e:/code/uWebSockets/examples# ldconfig -p|grep z
libz.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libz.so.1
libz.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libz.so
liblz4.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/liblz4.so.1
liblzo2.so.2 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblzo2.so.2
liblzma.so.5 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblzma.so.5
liblzma.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/liblzma.so
libharfbuzz.so.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0
libbz2.so.1.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/libbz2.so.1.0
libbz2.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libbz2.so

root@a1572b50f18e:/code/uWebSockets/examples# ldd ./SimpleHttp
linux-vdso.so.1 (0x00007ffdc63da000)
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f9992f0a000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f9992a71000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f9992857000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f999263a000)
libstdc++.so.6 => /usr/local/lib64/libstdc++.so.6 (0x00007f99922b8000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9991fb4000)
libgcc_s.so.1 => /usr/local/lib64/libgcc_s.so.1 (0x00007f9991d9c000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f99919fd000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f99917f9000)
/lib64/ld-linux-x86-64.so.2 (0x00007f9993176000)

I have created a Gist with a simplified program that causes a seg fault when it receives a POST, GET works fine. The Gist also includes a Dockerfile to recreate my environment and runs my program.

Instructions (in case you need them :))

1. Download both files into a new directory
2. Run docker build -t uws . to build the Docker image
3. Run docker container run -it -p3020:3000 --name uws_test uws to create a container and run the test program
4. Run ```
   curl --request POST
   --url http://localhost:3020/hello
   --header 'content-type: application/json'
   --data '{ "arg1": 4.0, "arg2": 0.0, "arg3": "sf", "arg4": 0.0, "arg5": 1.0, "arg6": "http", "arg7": 1.0 }'

to invoke with my payload

i don't see any files

you didn't post any code?

[christoth]

Sorry, my bad - Gist

You are appending to a stack allocated buffer that's potentially gone later on

[christoth]

Good point, I was assuming that the post lambda would last longer than the onData lambda.

I'll create a pointer and clean it up after res->end() and report back.

Thanks!

That's not exactly how it works, the buffer is never captured by any lambda it just happens to be created on the stack while executing a lambda. I think the "move capture" in C++14 is what you want here, it's efficient and correct

[christoth]

Ok. As you can tell I'm new to lambda coding...thanks for the guidance.

[christoth]

If I understand correctly, the post() lambda might go away before the onData() lambda is finished using any reference captured variables.

I need to concatenate the data from ALL the onRead() lambda calls - so I have a complete request body, before I process the request data and return the response.

Reading up on the "move capture" (e.g. [x = std::move(x)]), my buffer could be modified in the outer lambda, but not in the inner lambda. I need to be able to append to it in the inner lambda. Am I misinterpreting your suggestion?

If the post() lambda can have a shorter life than the "final" onData() lambda, then wouldn't I need some sort of persistence mechanism, such as a user data context on HttpResponseData to persist the buffer across all the onData() calls?

[mfbalin]

When I try the onData function to read the data, I get the following error:
Returning from a request handler without responding or attaching an abort handler is forbidden!