ipfs-search / nsfw-server

A simple Node.js server to run nsfw.js for images from IPFS and return its results.

ipfs-search.com

Search engine for the Interplanetary Filesystem. Sniffs the DHT gossip and indexes file and directory hashes.

Metadata and contents are extracted using ipfs-tika, searching is done using OpenSearch, queueing is done using RabbitMQ. The crawler is implemented in Go, the API and frontend are built using Node.js.

The ipfs-search command consists of two components: the crawler and the sniffer. The sniffer extracts hashes from the gossip between nodes. The crawler extracts data from the hashes and indexes them.

Docs

Documentation is hosted on Read the Docs, based on files contained in the docs folder. In addition, there's extensive Go docs for the internal API as well as SwaggerHub OpenAPI documentation for the REST API.

Contact

Please find us on our Freenode/Riot/Matrix channel #ipfs-search:matrix.org.

Snapshots

ipfs-search provides the daily snapshot for all of the indexed data using snapshots. To learn more about downloading and restoring snapshots please refer to the relevant section in our documentation.

Related repo's

• frontend
• search API
• deployment
• nsfw-server
• ipfs-tika

Contributors wanted

Building a search engine like this takes a considerable amount of resources (money and TLC). If you are able to help out with either of them, do reach out (see the contact section in this file).

Please read the Contributing.md file before contributing.

Roadmap

For discussing and suggesting features, look at the issues.

External dependencies

• Go 1.19
• OpenSearch 2.3.x
• RabbitMQ / AMQP server
• NodeJS 9.x
• IPFS 0.7
• Redis

Internal dependencies

• nsfw-server
• ipfs-tika

Building

$ go get ./...
$ make

Running

Docker

The most convenient way to run the crawler is through Docker. Simply run:

docker-compose up

This will start the crawler, the sniffer and all its dependencies. Hashes can also be queued for crawling manually by running ipfs-search a <hash> from within the running container. For example:

docker-compose exec ipfs-crawler ipfs-search add QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv

Ansible deployment

Automated deployment can be done on any (virtual) Ubuntu 16.04 machine. The full production stack is automated and can be found in it's own repository.

Contributors

This project exists thanks to all the people who contribute.

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors

ipfs-search is supported by NLNet through the EU's Next Generation Internet (NGI0) programme.

RedPencil is supporting the hosting of ipfs-search.com.

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

nsfw-server's Issues

Run unittests in CI

Maybe best to do in GitLab?

Configurable listener host

Could be a single parameter together with port, see other servers within the ipfs-search camp.

Instrumentation

OpenTelemetry instrumentation (tracing and preferably metrics)

OpenAPI 2 spec, client implementations

502 CORS errors from nsfw server on unsupprted file formats

When searching images, the browser makes a lot of calls to the nsfw server.
A bunch of them return 502 errors with a reason about CORS headers not being set. It seems related to unsupported fileformats, such as TIFF and WEBP.

E.g.:
https://ipfs-search.com/#/search?q=references.name%3A%28%22%2a.webp%22%29&page=1&type=images
https://ipfs-search.com/#/search?q=references.name%3A%28%22%2a.tiff%22%29&page=2&type=images&last_seen=%2a

Expected behavior for unsupported fileformats is to get a 415 error, not a 5xx error.

Consider API similar to nyats thumbnail server

Specifically:

1. Only allow lookups based on CID, and specific protocols.
2. Use URL's for responses, and consider them (in principle) immutable (allow for long-term caching).
3. Allow IPFS-based caching of API responses.

Ref: https://github.com/ipfs-search/nyats/blob/main/server/server.js

502 Gateway errors

On roughly half of the calls to the nsfw api, a 502 error (bad gateway) is being returned.

Allow concurrent requests

From logs here below it seems that requests somehow are handled sequentially.
Need to figure out if this is because of an operating system limitation, browser throttling or a server configuration, and fix it.

1640794675385 info 127.0.0.1 GET /classify {"url":"https://gateway.ipfs.io/ipfs/bafkreiazgl737iphtrr4777ju7o4chrniet7knzmtnydizhzbechwhnofi"} 415 (18366 ms)
1640794675517 info 127.0.0.1 GET /classify {"url":"https://gateway.ipfs.io/ipfs/QmbNA4nN3eLR12MBd6kVLkLiR82Pf69qZQGG4DdoM6siVq"} 200 (129 ms)
1640794675630 info 127.0.0.1 GET /classify {"url":"https://gateway.ipfs.io/ipfs/QmXnEU8ftNL4xxZeN1ksLNqVW9f7DHmeKZzEhYVKbmw76G"} 200 (111 ms)
1640794675792 info 127.0.0.1 GET /classify {"url":"https://gateway.ipfs.io/ipfs/QmSZzv7ux1LGwpehVcCMQ9ec945X6qE4qyjKDhCVwY25iw"} 200 (161 ms)
1640794675919 info 127.0.0.1 GET /classify {"url":"https://gateway.ipfs.io/ipfs/QmcPw2yPbtTiZQuabDnPgEHquURhwyDojomhSAc56y4DNM"} 200 (125 ms)
1640794678124 info 127.0.0.1 GET /classify {"url":"https://gateway.ipfs.io/ipfs/QmWx6ThfE8r6yYjAHSsc2xzJLS5nGdPyua6A75aRrLkP84"} 200 (18374 ms)

Explicitly load weights and add weights metadata

So we know what (version of) weights our data is made with.

Should be such that we can load them from IPFS.

Excessive memory usage

I'm seeing 20g+ memory usage for NSFW-server in production.

Would be great to have a 'neat' way to limit this.

Add Dockerfile

No CORS headers present on requested resources both from NSFW server and metadata API.

We need to make cross-origin requests on the api but the server (or gateway) refuses this both for NSFW and metadata.

Enable code climate

• Configuration, reference: https://github.com/ipfs-search/dweb-search-frontend/blob/master/.codeclimate.yml
• README badge
• Implement fixups

Publish to NPM

Coverage reporting for Gitlab to CodeClimate

Reference: https://github.com/ipfs-search/ipfs-search/blob/master/.gitlab-ci.yml#L15

Bump CodeClimate maintainability up to at least B

IPFS_API_ADDRESS setting undocumented

Related to #24

Another question is whether we can't simply hash files without actually starting a node. It does seem the start: false simply doesn't come through. If it did, there would be no need to use or configure a locally running node.

Still starting JS ipfs node by default

After clean GIT clone, with go-ipfs running:

Separate metadata and data

I would suggest to separate metadata from the (NSFW) data, like so:

{
    "nsfwjs_version":"2.4.1",
    "nsfw": {"porn":0.9564213156700134,"sexy":0.018271932378411293,"hentai":0.013146454468369484,"neutral":0.01208603847771883,"drawing":0.00007423922215821221,"nsfwjsVersion":"2.4.1"}
}

Reference: https://github.com/ipfs-search/tika-extractor#api