howprogrammingworks / nodejsstarterkit

Starter Kit for Node.js 16 or later, minimum dependencies 🚀

Node.js Starter Kit

Concept

You can begin development from this Starter Kit, but it is not for production usage. The purpose of this Starter Kit is to show simplicity, basic concepts, give structure and architecture examples. All parts of this implementation are optimized for readability and understanding, but not for performance and scalability.

So it is good for development and education. However, for production deployment, you may need the Metarhia Example App an open-source application server on the top of Node.js.

Feature list

• Pure node.js and framework-agnostic approach
• Minimum code size and dependencies
• Layered architecture: core, domain, API, client
• Protocol-agnostic API with auto-routing, HTTP(S), WS(S)
• Graceful shutdown
• Code sandboxing for security, dependency injection and context isolation
• Serve multiple ports
• Serve static files with memory cache
• Application configuration
• Simple logger
• Database access layer (Postgresql)
• Persistent sessions
• Unit-tests and API tests example
• Request queue with timeout and size
• Execution timeout and error handling

Requirements

• Node.js v16 or later
• Linux (tested on Fedora, Ubuntu, and CentOS)
• Postgresql 9.5 or later (v12 preferred)
• OpenSSL v1.1.1 or later
• certbot (recommended but optional)

Usage

1. Fork and clone this repository (optionally subscribe to repo changes)
2. Remove unneeded dependencies if your project doesn't require them
3. Run npm ci --production to install dependencies and generate certificate
4. Add your license to LICENSE file but don't remove starter kit license
5. Start your project by modifying this starter kit
6. Run project with node server.js and stop with Ctrl+C

Help

Ask questions at https://t.me/nodeua and post issues on github.

License

Copyright (c) 2020-2023 How.Programming.Works contributors. This starter kit is MIT licensed.

nodejsstarterkit's Issues

Add terminal emulator as a client-side

From this repo: https://github.com/HowProgrammingWorks/ServiceWorker

Use destructuring syntax for API arguments

({ name1, name2 }) => { /* function implementation */ };

Freeze default sandbox

Session sandboxes will not be frozen but default should

Generate TLS with Let's Encrypt

TLS certificate autogeneration with https://letsencrypt.org/

Request execution queue

• Configuration parameters: { size: Number, timeout: Number }
• Wait in queue implementation (N requests)
• Error message on queue overflow
• Integrate with execution timeout #15
• Add to feature list

Note: execution timeout and queue timeout are not the same timeouts

Don't forget to test queue in end-to-end tests #11

Request execution timeout

• Configuration parameters: { timeout: Number }
• Timeout implementation
• Add to feature list

Don't forget to test timeout in end-to-end tests #11

Extract sandboxing to separate module

Pack static files to gz in cahce

If file is not already compressed like gif, jpeg, zip, etc... we will load files and hold them after gz in shared memory.

Serve multiple ports in threads

Server initialization order

0. Load core layer modules
1. Load internal node.js dependencies
2. Load external dependencies (installed from npm)
3. Create Application instance
4. Create Config instance
5. Create Logger instance and wait for open event
6. Load lib layer modules
7. Connect to database
8. Load domain layer, api methods and init scripts
9. Open server ports for listening

Prepare contract specifications

All parts of this kit will be described as a contract to be implemented and replaced easily even by third-party developers. We will prepare tests for contracts and integration tests to be sure parts will work together.

Fix tests: exit non-zero on fail

See example: https://travis-ci.org/github/HowProgrammingWorks/NodejsStarterKit/builds/677618387

Don't serve clients from main thread

To prevent critical exceptions in main thread we need to minimize code execution in it.

Potential semaphore queue overflow of expired tasks

If getting 'Semaphore timeout' exception in those 2 methods(Client.apiws and Client.api) we don't reach semaphore.leave. Therefore request which was added to the queue, it wouldn't be removed from the queue. And soon the queue will be busy with tasks with an expired timer

  async api() {
    const { semaphore } = this.application.server;
    await semaphore.enter();
    // ....
  /* Lets assume, we getting 'Simaphore timeout' and this error won't be handled by this method
  So we won't be able to reach semaphore.leave()
  */    
    try {
    // ...
      const proc = this.application.runScript(method, sandbox);
      const result = await proc(context)(args);
   // ...
    } catch (err) {
      this.application.logger.error(err.stack);
      this.error(this.res, err.message === 'Not found' ? 404 : 500);
    }
    semaphore.leave();
  }
}

Implement dependency injection in core

We need to pass application to all core modules not with aggregation (as we implemented this in v1) but with dependency injection, so we can use either global or sandbox context.

Decompose Server class

Here are following options:

• Class Server and two subclasses: HttpServer and WsServer
• Rewrite without classes just functions
• Extract almost all functions but use outer classes to preserve interfaces
• Use strategy pattern
• Implement Client class

Graceful shutdown

For following:

• Changing system code layer (domain code will autoreloaded)
• Update dependencies
• Restart hardware server

Path separator in Windows

In Windows, the fs.reader method gets paths like '\index.html' while the static() method waits '/index.html'

./lib/application.js

...
  async cacheDirectory(directoryPath) {
    const files = await fsp.readdir(directoryPath, { withFileTypes: true });
    for (const file of files) {
      const filePath = path.join(directoryPath, file.name);
      if (file.isDirectory()) await this.cacheDirectory(filePath);
      else await this.cacheFile(filePath);
    }
    fs.watch(directoryPath, (event, fileName) => {
      const filePath = path.join(directoryPath, fileName);
      this.cacheFile(filePath);
    });
  }
...

./lib/server.js

...
class Client {
  constructor(req, res, application) {
    this.req = req;
    this.res = res;
    this.application = application;
  }

  static() {
    const { url } = this.req;
    const filePath = url === '/' ? '/index.html' : url;
    const fileExt = path.extname(filePath).substring(1);
    const mimeType = MIME_TYPES[fileExt] || MIME_TYPES.html;
    this.res.writeHead(200, { 'Content-Type': mimeType });
    const data = this.application.cache.get(filePath);
    if (data) this.res.end(data);
    else this.error(404);
  }

...

Serve static files and API from different threads

It's better to have different kinds of threads:

• serve static files with http(s)
• serve API with ws(s)
• other types...

Hash passwords

Use crypto.scrypt and implement hashing in /api/registerUser.js and /api/signIn.js

Load SSL certificates

Use shared memory for cache static files

Load static files into cache using SharedArrayBuffer

Add node.js network API to sandboxed contexts

Possible isolation violation

Object.freeze() does not freeze objects recursively, so after Object.freeze(lib) it is still possible to change object properties of lib.

NodejsStarterKit/lib/application.js

Line 37 in a84aec7

for (const lib of libs) Object.freeze(lib);

For example it is possible to break http.STATUS_CODES object

Simple logger

Support logging from sandboxes with console interface wrapped and injected.
No logs rotation support.

Run API handlers in sandboxes

Setup PostgreSQL in TravisCI

Parallel execution concurrency limit

• Concurrency limit implementation
• Configuration parameters: { concurrency: Number }
• Integrate with Queue: #16
• Add to feature list

Sandbox pool

We can prepare N sandboxes in pool to speed up session starting

Simple testing

• Unittests
• API tests
• End-to-end tests

Session-level sandboxes

After session started all requests should be sandboxed in a session sandbox.
Now we have shared sandbox for all API methods in application, see: #3
This shared sandbox will be used just for requests without sessions.

Implement persistent sessions

Add websocket transport support

Live reload

• Static files
• API handlers

Layered architecture

Layer	Description	Directory
core	loader, multi-threading, sandboxing, general purpose tools	./lib
domain	layer for domain model and processes	./domain
api	external network interface	./api
client	browser api to wrap websocket and hide rpc calls	./static

Fix server static files

Remove utf8 in readFile for static files and backport this to v1.x

Fix SQL injection in Database.insert

Thanks to @belochub who reminded me this case from our previous implementation

Extract dependency loader to separate module

Dependency loader: /lib/dependencies.js

Restart worker on fatal exception

Cache static files

Improve ws API handler

We need everything we have in lib/server.js/Client.api in lib/server.js/apiws: sessions, context passing.

Add API introspection

• Over http(s)
• Over ws(s)

Add abstract transport

Make API transport-agnostic

Refactor domain and init

Load automatically /domain/*.js

Add access marker to API methods

We need to mark methods available without auth and requires auth

Write separate logs for each thread

Write logs to separate folder where each thread will create separate file with thread id in name.

Decompose server.js

Serve static files and API separately, decompose /lib/server.js/handler

Config structure

Use ({ field: 'value' }); instead of module.exports = { field: 'value' };

Show file names in api stack-traces

To see file names in api stack traces we need to pass file names to vm.Script.

Simple query builder

• async select(table, fields, where)
• async insert(table, record)
• async update(table, delta, where)
• async delete(table, where)