yhirose / cpp-httplib Goto Github PK

I'm finding I have crashes on shutdown due to https://github.com/yhirose/cpp-httplib/blob/master/httplib.h#L1596.

In this case, the worker thread is copying this by value. If, while the thread is servicing the request the server is shutdown, listen() returns, and Server is destroyed, then e.g. Server::process_request can try to access freed memory running on the thread.

A simple way to fix this would be to have listen() store the std::threads instead of detach()ing them and then .join() before exiting listen(). However, that would be very inefficient for a long running server as that list would grow during each request and not be cleared until stop().

Implementing the "// TODO: Use thread pool..." near there might be a good way to fix this (assuming the pool is joined before shutdown).

A more direct slightly hacky way might be to increment an atomic when a thread is started, decrement it on thread exit, and wait until it's zero before exiting listen().

If the HTTP server the client is requesting from sends a large page, the client will not read all of the response, because there is only one call made to the socket recv() method.

Something like this will fix it:

@@ -508,10 +508,16 @@ bool read_content(Stream& strm, T& x)
 {
     auto len = get_header_value_int(x.headers, "Content-Length", 0);
     if (len) {
       x.body.assign(len, 0);
-        if (!strm.read(&x.body[0], x.body.size())) {
-            return false;
-        }
+      auto r = 0;
+      while(r < len){
+       auto r_incr = strm.read(&x.body[r], x.body.size());
+       r += r_incr;
+       if(r_incr <= 0) break;
+      }
+      if (!r) {
+       return false;
+      }
     }
     return true;
 }

POST requests are extremely slow. Each request is taking about 1 second to complete. I am using the library on windows.

Hi again!
I try cpp-httplib with MinGW (x86_64-7.3.0-release-win32-seh-rt_v5-rev0.7z)

I have had to add to httplib.h the lines (and #define WIN32_ and #define MINGW)

#ifdef MINGW
#include "mingw.mutex.h"
#include "mingw.thread.h"
#else
#include <mutex>
#include <thread>
#endif

NOTE: I got mingw.mutex.h and mingw.thread.h from https://github.com/meganz/mingw-std-threads

After that, i get:

..\..\inc\cpp-httplib/httplib.h: In function 'std::__cxx11::string httplib::detail::get_remote_addr(socket_t)':
..\..\inc\cpp-httplib/httplib.h:629:9: error: 'inet_ntop' was not declared in this scope
         inet_ntop(AF_INET, &s->sin_addr, ipstr, sizeof(ipstr));
         ^~~~~~~~~
..\..\inc\cpp-httplib/httplib.h:629:9: note: suggested alternative: 'inet_ntoa'
         inet_ntop(AF_INET, &s->sin_addr, ipstr, sizeof(ipstr));
         ^~~~~~~~~
         inet_ntoa
..\..\inc\cpp-httplib/httplib.h:633:9: error: 'inet_ntop' was not declared in this scope
         inet_ntop(AF_INET6, &s->sin6_addr, ipstr, sizeof(ipstr));
         ^~~~~~~~~
..\..\inc\cpp-httplib/httplib.h:633:9: note: suggested alternative: 'inet_ntoa'
         inet_ntop(AF_INET6, &s->sin6_addr, ipstr, sizeof(ipstr));
         ^~~~~~~~~
         inet_ntoa

Apparently, I have solved this problem by adding the following code to httplib.h.

#ifdef MINGW
const char* inet_ntop(int af, const void* src, char* dst, int cnt){

    struct sockaddr_in srcaddr;

    memset(&srcaddr, 0, sizeof(struct sockaddr_in));
    memcpy(&(srcaddr.sin_addr), src, sizeof(srcaddr.sin_addr));

    srcaddr.sin_family = af;
    if (WSAAddressToString((struct sockaddr*) &srcaddr, sizeof(struct sockaddr_in), 0, dst, (LPDWORD) &cnt) != 0) {
        DWORD rv = WSAGetLastError();
        printf("WSAAddressToString() : %d\n",rv);
        return NULL;
    }
    return dst;
}
#endif

I am going to test it but...Is there any better solution?

DJuego

It would be nice to have optional support for gzip.

How does the library Server object support stopping the listener ?

LNK2019 unresolved external symbol __imp_WSAStartup referenced in function "public: __cdecl httplib::detail::WSInit::WSInit(void)" (??0WSInit@detail@httplib@@qeaa@XZ)

got this error when i add <httplib.h> file in my project....
pls help...

I think this only really affects the test code, but I thought perhaps you would want to know.

In investigating a possible patch for #46, I split listen() into helper functions locally. After I did that I was experiencing an occasional crash in the server tests. I made a repro by adding a short sleep() in the body of listen() on master like this: https://github.com/yhirose/cpp-httplib/compare/master...sgraham:start-stop-crash?expand=1

When I run this (on a Mac), ASAN reports a crash in the thread:
https://gist.github.com/sgraham/461f286144ef8c7e531dad75ec2960d1

I think this is because the svr_.is_running() in the test SetUp() passes while listen() is in the sleep, then the server is immediately stopped, then the thread continues on to detail::select_read(), but svr_sock_ is now -1, so select_read() crashes in FD_SET(-1, &fds).

(Sorry for the long explanation without a fix, I wasn't immediately sure where the correct place to synchronize would be.)

Every now and then my server crashes with following error.

main: ext/httplib.h:651: std::string httplib::detail::decode_url(const string&): Assertion `s[i]' failed.

How can we define custom timeouts for requests?

My current code:

    httplib::Client cli(serverAddress.c_str(), port);
    auto res = cli.post(path, data, "text/plain");
    if (res == nullptr) {
      LOGE() << "failed to send data to " << path << ": got no response";
      return;
    }
    if (res->status != 200) {
      LOGE() << "failed to send data to " << path << ", response code: " << res->status;
      return;
    }

This just blocks when the server is not reachable. I'd like to report an error after 5 seconds.

Hi!
It is said that cpp-httplib is C++11 library. But when I try to build example on gcc, I get an error:

error: use of ‘auto’ in lambda parameter declaration only available with -std=c++14 or -std=gnu++14

While this relates only to examples code it... should be at least said somewhere...

The client, when doing a request to a server, does not include the host in the header. This causes some failures on some sites. Something like this fixes the problem (presumably also needed for HEAD and POST methods):

@@ -1078,6 +1084,7 @@ inline std::shared_ptr<Response> Client::get(const char* path)
     Request req;
     req.method = "GET";
     req.path = path;
+    req.set_header("Host", host_.c_str());
 
     auto res = std::make_shared<Response>();

Hey there,

I really like the idea behind this library, but I am running into a few issues when trying to include this with my project. I am using g++ with Mingw on Windows, rather than Visual Studio.

My current command to run everything looks like this:

g++ test.cpp httplib.h -DWEBVIEW_WINAPI=1 -lole32 -lcomctl32 -loleaut32 -luuid -mwindows -DGTEST_USE_OWN_TR1_TUPLE -I.. -I. -Wall -Wextraw

I am also using a Webview library to create a browser. The errors that I receive are related to " undefined reference to `_imp__freeaddrinfo@4'" for example, so linker errors.

Any ideas?

First of all. Contratulation on your excellent work. Iam coming from nodejs world so i found it easy to understand your vision.

1. Does this library support some kind of middleware functions (auth or ip filter middleware etc.).

2. Secondly, is there any benchmark that compares performance with other frameworks ? (cpprestapi
   casablanca etc.)
   Thank you.

See #71 for the detail.

Add static files support. Need a method like 'set_htdocs_path'.

when writing the response body to a file, i found that it contained extra unwanted data. I was trying to make it get a file from a api, the data is in base64 and it appends value before the expected data

the expected result =

Any help would be appreciated

used code -

httplib::SSLClient client("example.com", 443);
auto logincheck = "/load?username=" + user + "&product=" + product;
const char* chr = _strdup(logincheck.c_str());
auto res = client.get(chr);
if (res && res->status == 200)
encoded << res->body;

sorry for the bad code format

edit : found that it might be caused by it displaying chunks as the server is sending it with "Transfer-Encoding: chunked"

This could cause bad security problem.

Hello,

I had been using the default setting for the server which was HTTP/1.0 until recently. In test situations, the client code sends one request, and after the server has sent its response, it calls Server::stop(), ending the test. This still works correctly, however, I believe because of the change to 1.1, causes the server to wait for 5s (CPPHTTPLIB_KEEPALIVE_TIMEOUT_SECOND) before termination, which makes the tests take longer than necessary.

I can work around this by locally patching CPPHTTPLIB_KEEPALIVE_MAX_COUNT to 1, but I was just wondering if there's something different I should be sending from the client (I'm only using cpp-httplib for the server, not the client side). Alternatively, maybe it would be possible to have Server::stop() signal to the select() somehow that it should shut down immediately?

I was trying to use cpp-httplib for running tests, so I'd like to start it on a system-selected port, and then have clients connect to that port. I don't want to pre-select a port number to avoid collisions with other things on the system, or when tests are run in parallel.

When I do the common thing of passing 0 as the server port everything seems to work, and I was able to hunt down the actual port by running netstat, and when I then connect to that port things work just fine.

Would it be possible to have a method to retrieve port that the server's running on? (Or is that already possible and I missed how to do so?)

Some browsers seem to send metadata about MultipartFiles as lower-case 'content-type' and 'content-disposition'. This seems to be consistent with the specification (RFC7578). Currently, these are missed with this http server and are therefore not handled correctly.

The simple diff below changes the regex(es) that look for this metadata to case-insensitive and fixes this problem.

$ git diff httplib.h
diff --git a/httplib.h b/httplib.h
index 1716d0e..461c1e4 100644
--- a/httplib.h
+++ b/httplib.h
@@ -1026,10 +1026,9 @@ inline bool parse_multipart_formdata(
     static std::string crlf = "\r\n";

     static std::regex re_content_type(
-        "Content-Type: (.*?)");
+        "Content-Type: (.*?)", std::regex_constants::icase);
     static std::regex re_content_disposition(
-        "Content-Disposition: form-data; name=\"(.*?)\"(?:; filename=\"(.*?)\")?");
+        "Content-Disposition: form-data; name=\"(.*?)\"(?:; filename=\"(.*?)\")?", std::regex_constants::icase);

     auto dash_boundary = dash + boundary;

Multi-thread support to accept simultaneous connections as bellow.

ab -n 10 -c 5 http://host:port/slow_request_which_takes_10_secondes

Allow users to disable this feature with CPPHTTPLIB_NO_MULTI_THREAD_SUPPORT.

Under POST with parameters in the README it is showed how to use httplib::Map to post data. This does not exist anymore and should be updated to httplib::Params (?).

Hi,

I just starting trying your library and noticed a warning:

cpp-httplib/httplib.h:256:54: warning: unused parameter ‘bufsiz’ [-Wunused-parameter]
inline bool socket_gets(Stream& strm, char* buf, int bufsiz)

Here

I could ignore it, but since it's a buffer size, wouldn't that be a bug and the bufsize should be used to prevent buffer overflow, no ?

Thanks

Baptiste

There are no 'Put' in the current example of 'Put'. :-) (README)
DJuego

The first file is certificate, the second is private key, not the other way around.

Allow server to accept case insensitive HTTP header such as 'content-type: text/plain'.

Great library -- one issue is that the sample code in the README.md doesn't compile because
the parameters const Response& res are actually supposed to be non-const: Response& res

svr.get("/hi", [](const Request& req, const Response& res) {
        res.set_content("Hello World!", "text/plain");
    });

and

    svr.get(R"(/numbers/(\d+))", [&](const Request& req, const Response& res) {
        auto numbers = req.matches[1];
        res.set_content(numbers, "text/plain");
    });

Need solution file for VC11.

how to post file to server, content_type multpart/form-data

It would be nice if Client:send added default_headers (HOST, User-Agent, Accept) when they are not set :)

First of all, great little library!

Is it possible to get the full URL as sent by the client from the Request struct? I know I can get the path and params, but sometimes it's useful to get the full URL.
So, if the browser requests: http://127.0.0.1:1234/app?var1=value1&var2=value2
With Request::path I can get e . g. /app. With Request::params I can access the variables, but seems there is no way to access the full URL (besides reconstructing it yourself)?

I think something like Request::url which would return e. g. http://127.0.0.1:1234/app?var1=value1&var2=value2 or at least app?var1=value1&var2=value2 would be quite useful.

Having a pretty simple server, which is set up very close to the examples.

After some time of fine running, it just starts failing with SIGPIPE here:

inline int SocketStream::write(const char* ptr, size_t size) // ptr = "Connection: close\r\n"
    {
        return send(sock_, ptr, size, 0);
    }

Linux 4.4.0-53-generic #74-Ubuntu SMP Fri Dec 2 15:59:10 UTC 2016 x86_64 x86_64 
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)

Update:
Sometimes it runs really long without it. So one can say, that sometimes it is not happening.

It would be really nice if we could process forms encoded multipart/form-data, including file uploads. Unfortunately this feature lacks in most lightweight C++ HTTP libraries, though it doesn't takes much slocs to implement.

First of all, thanks for providing such a great library!

I had a problem to get connections to certain HTTPS hosts working with using httplib::SSLClient. After investigating with openssl s_client, I found that the problem is that the host sends the wrong certificate to the client. This is because, the server doesn't know for which site the client expects the certificate if multiple sites/certificates are on the same host (i.e. on a virtual server).
The solution to this problem in TLS is the extension RFC 6066, Server Name Indication. Using this TLS extension the client will send during the handshake the servername for which he wants to receive the certificate, so that the correct certificate can be supplied to the client. The extension is already implemented in openssl and can be used with the "-servername" parameter, i.e. openssl s_client -connect httpbin.org:443 -servername httpbin.org

SNI (Server Name Indication) is widely used on the internet and I think it would be a valuable addition to cpp-httplib.

A minimal example which shows the problem using httplib::SSLClient (openssl-1.0.2l, Visual Studio 2017):

#include <iostream>

#define CPPHTTPLIB_OPENSSL_SUPPORT
#include <httplib.h>
#pragma comment(lib, "Ws2_32.lib")
#pragma comment(lib, "libeay32.lib")
#pragma comment(lib, "ssleay32.lib")

int main(int argc, char* argv[]) {
  httplib::SSLClient cli("httpbin.org", 443);
  auto res = cli.get("/get");
  if (res && res->status == 200) {
    std::cout << res->body << std::endl;
  }
  return 0;
}

One possible workaround in read_and_close_socket_ssl is:

SSL_set_bio(ssl, bio, bio);
SSL_set_tlsext_host_name(ssl, "httpbin.org"); // workaround
SSL_connect_or_accept(ssl);

It would be more elegant, if the hostname would come from the Client::host_, but it is not accessible from there with the current interface.

Need an easy way to access it, perhaps in request structure

It is allowed in HTTP 1.0 to have no "Content-Length" header field in the response-header: In this case closing of the connection indicates the end of the response-body. But in the current implementation reading the body of the response is skipped if there is no "Content-Length" field, resulting into an empty body.
The template function read_content(Stream& strm, T& x) is used for the server (handle request) and for the client (handle response), but the handling for no "Content-Length" should only be applied to the response.

An example which demonstrates the problem (openssl-1.0.2l, Windows, Visual Studio 2017):

#include <iostream>
#define CPPHTTPLIB_OPENSSL_SUPPORT
#include <httplib.h>
#pragma comment(lib, "Ws2_32.lib")
#pragma comment(lib, "libeay32.lib")
#pragma comment(lib, "ssleay32.lib")

int main(int argc, char* argv[]) {
  httplib::SSLClient cli("bittrex.com", 443);
  auto res = cli.get("/api/v1.1/public/getmarkets");
  if (res && res->status == 200) {
    std::cout << res->body << std::endl;
  }
  return 0;
}

I have created a template specialization for Response which will work (but is maybe not optimal, because it adds bytes to the body one-by-one, similar to socket_gets).

bool read_content(Stream& strm, Response& x)
{
  auto len = get_header_value_int(x.headers, "Content-Length", 0);
  if (len) {
    x.body.assign(len, 0);
    auto r = 0;
    while (r < len) {
      auto r_incr = strm.read(&x.body[r], len - r);
      if (r_incr <= 0) {
        return false;
      }
      r += r_incr;
    }
  } else { /* Start Added code */
    for (;;) {
      char byte;
      auto n = strm.read(&byte, 1);
      if (n < 1) {
        if (x.body.size() == 0) {
          return true; // no body
        }
        else {
          break;
        }
      }
      x.body += byte;
    }
  } /* End Added code */
  return true;
}

I need to build a static executable using cpp-httplib.

Then I add -static to CFLAGS and -ldl -Wl,--whole-archive -lpthread -Wl,--no-whole-archive to the LDFLAGS.

After my changes the test/Makefile contains:

CFLAGS = -std=c++11 -DGTEST_USE_OWN_TR1_TUPLE -I.. -I. -Wall -Wextra -static 
OPENSSL_SUPPORT = -DCPPHTTPLIB_OPENSSL_SUPPORT -I/usr/local/opt/openssl/include -L/usr/local/opt/openssl/lib -lssl -lcrypto
ZLIB_SUPPORT = -DCPPHTTPLIB_ZLIB_SUPPORT -lz
EXTRA_LDFLAGS = -Wl,--whole-archive -lpthread -Wl,--no-whole-archive -ldl

all : test
	./test

test : test.cc ../httplib.h Makefile
	$(CC) -o test $(CFLAGS) test.cc gtest/gtest-all.cc gtest/gtest_main.cc $(OPENSSL_SUPPORT) $(ZLIB_SUPPORT) $(EXTRA_LDFLAGS)

pem:
	openssl genrsa 2048 > key.pem
	openssl req -new -key key.pem | openssl x509 -days 3650 -req -signkey key.pem > cert.pem

clean:
	rm test *.pem

When I run the test suite its ok until

[----------] 3 tests from ConnectionErrorTest
[ RUN      ] ConnectionErrorTest.InvalidHost
Makefile:9: recipe for target 'all' failed
make: *** [all] Falha de segmentação

In RFC 7230(HTTP1.1),

if the connection's incoming TCP port number
differs from the default port for the effective request URI's
scheme, then a colon (":") and the incoming port number (in
decimal form) are appended to the authority component.

While I know your library supports HTTP/1.0,
I'm glad to append port number to HOST header like following:

#-    req.set_header("Host", host_.c_str());
+    req.set_header("Host", (host_ + ":" + std::to_string(port_)).c_str());

OpenSSL support.

Thank you for this project! Thank you for your effort!

Header-only, no dependencies. cross-platform, light and elegant... It's the closest thing to the solution I was looking for.

Only I miss a more complete interface (with PUT and DELETE and perhaps, others). Any reason not to implement those features?

In any case I am not an expert and I think that what your project offers is enough for my purposes. :-)

Thanks again!

DJuego

added functionality to get downloading portions of big files using range_low & range_high
can be easily achieved with following code:
( default parameter values must be defined in the class definition, they are just in the implementation for ovesight )
`
inline void Client::add_range_header (Request &req, int64_t range_low, int64_t range_high)
{
std::string range_str = "bytes=";

            if ( range_low >= 0 )
            {
                    range_str += std::to_string (range_low);
            }

            range_str += "-";

            if ( range_high >= 0 )
                    range_str += std::to_string (range_high);
            
            req.set_header ("Range", range_str.c_str ());
    }

    inline std::shared_ptr<Response> Client::get (const char* path, Progress callback, int64_t range_low=-1, int64_t range_high=-1)
    {
            Request req;
            req.method = "GET";
            req.path = path;
            req.progress = callback;
            add_default_headers (req);


            if ( range_low >= 0 )
                    add_range_header (req, range_low, range_high);

            auto res = std::make_shared<Response> ();

            return send (req, *res) ? res : nullptr;
    }

`

Hello
Thanks for the lib!

I got an issue with "httplib::Params".

According to readme we can fill in post parameters like this :

httplib::Params params;
params["name"] = "john";

But it doesn't compile : error C2676: binary '[': 'httplib::Params' does not define this operator or a conversion to a type acceptable to the predefined operator

Params is a std::multimap and it seems it doesn't support this way to insert element.

When using the code sample :

` ``
std::thread signaling_health_thread([] () {
printf("hey from new thread!\n");
httplib::Client cli("localhost", 1234);

auto res = cli.get("/hi");
printf("I have got a res!\n");
if (res && res->status == 200) {
  std::cout << res->body << std::endl;
}
});
signaling_health_thread.join(); 

I get a SIGPIPE signal if there is no port open on localhost:1234.

Is that expected ? Thanks.

It would be really helpfull to split the library into multiple .hpp files for development and then merge them into single .hpp / .h for distribution :)
Examples:
https://github.com/ThePhD/sol2/blob/develop/single.py
https://github.com/oftc/jsoncpp/blob/master/amalgamate.py
https://github.com/connormanning/arbiter/blob/master/amalgamate.py

This would not only allow for easier development but also for possible PRs/contributions or simple modifications/extensions that others might need to make.
It would also improve code reuse and abstraction mechanisms.
For example sprintf_socket could be made as a method writefmt of Stream and not implementation detail.

Currently you are assuming that windows compiler is always MSVC (by checking_MSC_VER) which might not be true (as is in my case).
You should change it to check for (_WIN32) in all places first.
Also you might consider adding autodetection for headers when c++17 compiler is found...
Example:

#if ( __cplusplus > 201703L)
#if __has_include(<openssl/ssl.h>)
#ifndef CPPHTTPLIB_OPENSSL_SUPPORT
#define CPPHTTPLIB_OPENSSL_SUPPORT
#endif
#endif
#endif

https://en.wikipedia.org/wiki/HTTP_persistent_connection