nodezoo-npm - a nodezoo.com Micro-service
The nodezoo-npm micro-service provides an interface to the NPM registry.
One of the micro-services that implement nodezoo.com, a search engine for Node.js modules. The NodeZoo search engine is an example of a real-world service built using Node.js micro-services. Each micro-service is published in its own github repository. The codebase is intended to be used as a larger-scale example, and as a starting point for your own projects.
The search-engine is under development in an open manner, and a blog series (see below) covers the full details.
This micro-service uses the Seneca micro-service toolkit. For an introduction to Seneca itself, see the senecajs.org site.
Feel free to contact me on twitter if you have any questions! :) @rjrodger
Tested on: Seneca 0.5.20, Node 0.10.31
Create a project folder for all the nodezoo micro-service repositories. The clone this repository into your project folder:
git clone https://github.com/rjrodger/nodezoo-npm.git
Alternatively, fork the repository so that you can make your own changes.
This micro-service recognizes the following messages:
- role:npm,cmd:get - get module details by name
- role:npm,cmd:query - query module details from NPM
- role:npm,cmd:extract - extract relevant data from NPM result
And issues the following messages:
- role:search,cmd:insert - insert module details into search engine index, OPTIONAL
It overrides these messages:
- role:entity,cmd:save,name:npm - insert module details into search engine in parallel to save
The main business logic is in the npm.js file. However, to run the service, you make use of different small scripts depending on your needs.
When running in a local configuration, this service is exposed over HTTP on port 9001.
Micro-services should be testable standalone. That means you can run them and they should "work" (for some value of work), without any other micro-services running. This is important for automated testing.
In the nodezoo-npm folder:
$ node srvs/npm-test.js --seneca.log=plugin:npm
To test, try:
$ curl -d '{"role":"npm","cmd":"get","name":"underscore"}' http://localhost:9001/act
This will print the details for the underscore module. The module details are saved as JSON file in the data folder. In production, you would use a more robust storage mechanism, but JSON files make local debugging easier.
You'll also observe that the log output of the process prints an error. That's ok. There's no implementation of the search engine index available. The micro-service does what it can and proceeds. In a production enviroment, this error is collected and the monitoring system might decide to issue an alert.
If you run the curl command a second time, you'll see no error. As the module data already exists, no action is taken.
In this scenario, you run the system on your local machine, usually with a single instance of each service.
You'll need:
- a redis server,
- an elasticsearch server
- the nodezoo-index service
In the nodezoo-npm folder, run the development service:
$ node srvs/npm-dev.js --seneca.log=plugin:npm
Use curl as before:
$ curl -d '{"role":"npm","cmd":"get","name":"underscore","update":true}' http://localhost:9001/act
This time, you have to use the update parameter to force the service to update the data. The module should now be indexed, which you can confirm by querying elastic search:
$ curl http://localhost:9200/zoo/mod/underscore
Production deployment on AWS and Joyent using Docker and nscale will be covered later in the project.
The development of the system is covered in detail in a series of blog posts on the skillsmatter.com blog:
Hosting and development is sponsored by nearForm.