neusta-code-challenge

Installation

Prerequisites according to package.json

node ^8.9.3
npm ^5.5.1

npm install

Usage

npm start

The server will be available at http://localhost:3000/.

Design decisions

Node.js

The task is I/O-heavy. Node.js is built for that. Splitting a task into subtasks and running them in parallel is a breeze.

Express

Very little code necessary to get a secure high-performance REST API going.

Node.js' stream API and Highland streaming library

While looking unfamiliar to developers not used to node development, it has huge advantages compared to other approaches:

Super fast, parallel computing
Shared back-pressure
- Data is processed as fast as the slowest consumer in the stream's pipeline.
- During a file upload, incoming HTTP data is throttled if a downstream task (e. g. inserting into database) is slow to prevent buffers and memory size from growing.
As an example, data processing of HTTP requests is started as soon as the first chunks of data arrive at the server (while the rest of the request is still being received).
The same applies for error handling. If there is a duplicate room on the first two lines of a 10 GB file, the server responds with an error message immediately, not only after several minutes of uploading and analyzing the file.
After having been processed, chunks of data do not stay in memory. At no time is a large request body fully held in memory.
With its very low memory usage, it can handle input files of any size, even if they exceed the system's memory.

Data model

Instead of a database, a local variable is used as storage in order to comply with the challenge's rules. This should not be done in production and renders some of the stream API's advantages useless (like e. g. unlimited file size or streaming from and to the database).
The data-modelling library Mongoose is used to define the Person schema. Thus, the application takes advantage of its structured way of defining a schema, built-in validation support, good testability, and the possibility to switch to a Mongo DB as the persistence layer with just a few lines.
The internal data model is based on the `Person` schema only. There is nothing like a `Room` schema, since rooms do not have any attributes yet.
To keep a consistent naming scheme, some things have been renamed (redirects are in place where applicable):
- `people` are called `person` or `persons` respectively throughout the application.
- A person's attributes are converted from space-separated to camelCase internally to stay consistent with JavaScript conventions and best practices.

Performance

This application is able to parse and save ~64,000 persons per second over HTTP on a standard office notebook.

Project structure

The project structure is heavily inspired by popular REST/MVC frameworks like Sails.js.

coverage

Coverage reports for combined unit and end to end tests in HTML format

e2e

Test specifications for end-to-end tests

e2e/data

Original and additional test data for end-to-end tests

src

classes: The Person schema and model (Mongoose)
controllers: REST API controller functions (the ones that actually respond to the request). These are the functions you're looking for...
middlewares: Global error handling middleware
responses: Response helper functions (e. g. static error messages)
services: Helper modules for logging and parsing
config.ts: Global configuration file
index.ts: Main server entry point used by `npm start`. Spawns 1 web worker in this version. Will spawn 1 web worker per CPU core as soon as a real database may be used.
router.ts: Hierarchical routing tree representing the URL paths

Static code analysis

npm run lint

Testing

Run all tests

npm test

Coverage

classes are covered by unit tests in *.unit.spec.ts files. controllers are covered by end to end tests in *.e2e.spec.ts files.

Currently, the line coverage status is as follows:

Folder	Line coverage
classes	100.0 %
services	100.0 %
controllers	100.0 %
responses	100.0 %

Unit

npm run test:unit

End to end

During end to end testing, real HTTP requests are fired against the real REST API server.

Don't forget to disable proxy servers for localhost!

npm run test:e2e

Coverage reports

Located in the coverage folder.

erkstruwe / neusta-code-challenge Goto Github PK