mynameistrez / webserv

webserv

Used links

• Siege (HTTP load tester)
• RFC key word summary
• HTTP response status codes
• webserv eval sheet
• CGI programming is simple!
• CGI C examples
• NGINX configuration guide
• See comments on this answer about how non-blocking is impossible
• Non-blocking I/O with regular files
• File operations always block
• Non-blocking disk I/O
• NGINX configuration format explanation
• NGINX "location" directive explanation
• Full example NGINX configuration
• How NGINX processes a request
• How NGINX redirection and try_files works
• HTTP header status flowchart
• PATH_INFO and QUERY_STRING

Setting up multiple servers with different hostnames

This will print example.com's response: curl http://example.com/

This will print localhost's response: (by search-and-replacing example.com) curl --resolve example.com:8080:127.0.0.1 http://example.com:8080/

• Go to default (first) server in NGINX with curl --resolve example.com:8080:127.0.0.1 http://example.com:8080/
• Go to server_name "f1r4s8" in NGINX with curl http://f1r4s8:8080
• Go to server_name "f1r4s8.codam.nl" in NGINX with curl http://f1r4s8.codam.nl:8080

Running nginx

• Build and run docker with docker build -t nginx nginx/ && docker run --rm -it -v $(pwd):/code -p 8080:80 -p 8081:81 nginx
• Start nginx with nginx
• View the help menu with nginx -h
• Reload configuration file without closing connections with nginx -s reload
• View the website with http://localhost:8080/
• View the nginx log with cat /var/log/nginx/access.log

nginx tests

# Assuming root /code;
# index public/a.html => a
# index public/nonexistent => 403
# index public/ => 403
# index public => 301 with "Location: http://localhost:8080/public/"
# direct nonexistent => 404

# Assuming root /code/public;
# index foo/../foo/a.html => a
# index /foo/../foo/a.html => a
# index /foo/../foo/ => 500
# index /foo/../foo => 301 with "Location: http://localhost:8080/foo/../foo/"

# Assuming root /code/public/../public;
# index foo/../foo/a.html => a
# index foo/../foo => 301 with "Location: http://localhost:8080/foo/../foo/"

Running siege

• Benchmark a URL indefinitely with siege -b http://localhost:8080
• Siege a URL with c clients repeated r times with siege -c2 -r3 http://localhost:8080
• Let 5 clients send a POST request sending a with siege -c5 -r1 '127.0.0.1:8080/cgis/python/uppercase.py POST </home/sbos/Programming/webserv/public/a.html'
• Let 5 clients send a POST request sending 1k_lines.txt with siege -c5 -r1 '127.0.0.1:8080/cgis/python/uppercase.py POST </home/sbos/Programming/webserv/tests/sent/1k_lines.txt'
• Let 10 clients send 10 GET requests to the CGI script debug.py with siege -c10 -r10 http://localhost:8080/cgis/python/debug.py

Memory usage

• valgrind --tool=massif ./webserv
• ms_print massif.out.* > mem.log

Notes

• Sometimes siege http://localhost:8080 prints [error] socket: unable to connect sock.c:282: Operation timed out, which is expected according to the author of siege, as siege may overload the web server

Running Codam's tester

• First run export REQUEST_METHOD="GET" SERVER_PROTOCOL="HTTP/1.1" PATH_INFO="/foo/bar"
• Then run ./cgi_tester, and press Enter once

Using curl

• GET: curl localhost:8080
• POST: curl --data-binary @tests/1_line.txt localhost:8080
• POST chunked: curl -H "Transfer-Encoding: chunked" --data-binary @tests/1_line.txt localhost:8080
• POST with newline trimming: curl -d @tests/1_line.txt localhost:8080
• DELETE: curl -X DELETE localhost:8080
• Nonexistent header type: curl -X FOO localhost:8080
• Start CGI script debug.py: curl http://localhost:8080/cgis/python/debug.py
• Check whether the CGI is still running: ps -aux | grep print.py
• Check who is causing "Address already in use": netstat -tulpn | grep 8080
• Create foo.txt containing two "foo"s: yes foo | dd of=foo.txt count=2 bs=4
• Create stack_overflow.json containing repeated {"":: yes '{"":' | dd of=stack_overflow.json count=420420 bs=5
• POST a file containing 10k lines: curl --data-binary @tests/10k_lines.txt localhost:8080

Using nc

• echo "GET /foo/../foo/a.html" | nc localhost 8080
• nc -l 8081 start nc server
• echo "GET /foo/../foo/a.html" | nc localhost 8081 connect with nc server

Fuzzing the config parser

1. Run docker build -t aflplusplus-webserv fuzzing && docker run --rm -it -v .:/src aflplusplus-webserv to build and run docker
2. Run setup.sh to compile for afl-cmin + afl-tmin, generate tests, and compile for AFL
3. Run FUZZ_CLIENT=1 setup.sh to do the same but fuzzing the client instead of the config.
4. Run coverage.sh to fuzz while generating coverage
5. Run FUZZ_CLIENT=1 coverage.sh to do the same but fuzzing the client instead of the config.
6. Run AFL_DEBUG=1 FUZZ_CLIENT=1 fuzz.sh to get debug information in case it crashes.
7. Run minimize_crashes.sh to minimize the crashes, which are then put in /src/fuzzing/afl/minimized-crashes/

Debugging main_fuzzing_client.cpp after running setup.sh

clear && AFL_DEBUG=1 afl-fuzz -D -i /src/fuzzing/tests_fuzzing_client -o /src/fuzzing/afl/afl-output -M master -- /src/fuzzing/fuzzing_ctmin /src/fuzzing/fuzzing_client_master_webserv.json

Compiling main_fuzzing_client.cpp manually in the fuzzing Docker container

clear && g++ -Wall -Wextra -Werror -Wfatal-errors -Wshadow -Wswitch -Wimplicit-fallthrough -Wno-c99-designator -Werror=type-limits -std=c++2b -DSUPPRESS_LOGGING src/config/Config.cpp src/config/JSON.cpp src/config/Node.cpp src/config/Tokenizer.cpp src/Client.cpp src/Logger.cpp src/Server.cpp src/Throwing.cpp src/Utils.cpp fuzzing/src/main_fuzzing_client.cpp; echo foo | ./a.out /src/fuzzing/fuzzing_client_master_webserv.json

clear && dict_path=/src/fuzzing/conf_fuzzing_client.dict; AFL_DEBUG=1 afl-fuzz -D -i /src/fuzzing/afl/trimmed-tests -x $dict_path -o /src/fuzzing/afl/afl-output -M master -- /src/fuzzing/fuzzing_afl /src/fuzzing/fuzzing_client_master_webserv.json 2> /src/fuzzing/master.log

Fuzzing with multiple cores

FUZZ_CLIENT=1 fuzz.sh ps aux > foo.log clear && afl-whatsup /src/fuzzing/afl/afl-output/

Manual multipart request submission

clear && cat manual_multi_form_request.txt | REQUEST_METHOD=POST HTTP_CONTENT_TYPE='multipart/form-data; boundary=----WebKitFormBoundaryBRGOgydgtRaDx2Ju' python3 cgis/python/upload.py

Plan of attack

Assuming root /code/public;, with this file tree:

public/
	foo/
		a.png
		b.html
		c.mp4
	d.html

server_index = get_server_index_from_client_server_name(client)
server = servers[server_index]

location = resolve_location(target, server)

if not location.resolved:
	raise NOT_FOUND

if location.has_redirect:
	client.redirect = location.path
	raise MOVED_TEMPORARILY

if not is_allowed_method(location, method):
	raise METHOD_NOT_ALLOWED

if location.path[-1] == '/:
	if method == "GET":
		if location.has_index:
			client.respond_with_file(location.index_path)
			enable_writing_to_client(client)
		elif location.autoindex:
			client.respond_with_directory_listing(location.path)
			enable_writing_to_client(client)
		else
			raise FORBIDDEN
	else:
		raise FORBIDDEN
else:
	if is_directory(location.path):
		if method == "DELETE":
			raise METHOD_NOT_ALLOWED
		else:
			raise MOVED_PERMANENTLY

	if location.is_cgi_directory:
		start_cgi(client, location.cgi_settings, location.path, location.path_info, location.query_string)
	elif method == "GET":
		respond_with_file(location.path)
		enable_writing_to_client(client)
	elif method == "POST":
		create_file(location.path)
		enable_writing_to_client(client)
	else:
		delete_file(location.path)
		enable_writing_to_client(client)

// Construct this by looping over all servers and looping over their ports,
// letting the key be the server's `server_name` + ":" + `listen` port.
// If the key was already present, throw a config exception.
//
// This map is then used once we've fully read a client's header
// to map a Host header like `f1r4s8:8080` to a virtual server, saving it in the client
// If the Host header isn't in this map, use _port_to_default_server_index
std::unordered_map<std::string, size_t> _http_host_header_to_server_index;

// Construct this by looping over all servers and looping over their ports,
// letting the key be the server's `listen` port.
// If the key was already present, don't overwrite it, and just continue.
//
// This map is used as a fallback for _http_host_header_to_server_index
// The `port` is gotten by substr()-ing `8080` from `f1r4s8:8080` from the Host header
std::unordered_map<std::string, size_t> _port_to_default_server_index;

// Construct an unordered_set of all port numbers
uint16_t port;
if (!parseNumber(key, &port)) throw InvalidLineException();
std::unordered_set<uint16_t> _port_numbers;
_port_numbers.emplace(port);

// Use the unordered_set to call bind for every port number
port_fd = socket(AF_INET, SOCK_STREAM, 0);
servaddr.sin_addr.s_addr = htonl(INADDR_ANY);
servaddr.sin_port = htons(port);
bind(port_fd, (sockaddr *)&servaddr, sizeof(servaddr));

// Done when a port_fd has POLLIN
int client_fd = accept(port_fd, NULL, NULL);

// The client's request_target is finally used to find the location directive
// in their virtual server

webserv's Issues

?!

What in the name of all that is holy is this?

webserv/src/Client.cpp

Lines 36 to 51 in d441743

	Client::Client(Client const &src)
	: server_fd(src.server_fd), status(src.status), client_to_server_state(src.client_to_server_state),
	server_to_cgi_state(src.server_to_cgi_state), cgi_to_server_state(src.cgi_to_server_state),
	server_to_client_state(src.server_to_client_state), request_method(src.request_method),
	request_target(src.request_target), protocol(src.protocol), headers(src.headers), body(src.body),
	body_index(src.body_index), client_max_body_size(src.client_max_body_size), response(src.response),
	response_index(src.response_index), client_fd(src.client_fd), server_to_cgi_fd(src.server_to_cgi_fd),
	cgi_to_server_fd(src.cgi_to_server_fd), cgi_pid(src.cgi_pid), cgi_exit_status(src.cgi_exit_status),
	cgi_killed(src.cgi_killed), being_removed(src.being_removed), redirect(src.redirect),
	server_name(src.server_name), unuppercased_server_name(src.unuppercased_server_name),
	server_port(src.server_port), content_type(src.content_type), content_length(src.content_length),
	time_of_last_read(src.time_of_last_read), _custom_reason_phrase(src._custom_reason_phrase),
	_response_content_type(src._response_content_type), _header(src._header), _is_chunked(src._is_chunked),
	_chunked_remaining_content_length(src._chunked_remaining_content_length),
	_chunked_body_buffer(src._chunked_body_buffer), _chunked_read_state(src._chunked_read_state)
	{

)