Coder Social home page Coder Social logo

pinion's Introduction

Pinion

Pinion is a lightweight pinning service that supports both IPFS content and orbit-db stores. It relies solely on ipfs-pubsub-peer-monitor, orbit-db, and js-ipfs-http-client to communicate with an IPFS node.

Pinion can:

  • Pin IPFS content
  • Pin orbit-db store content
  • Keep listening to updates from any given orbit-db store

Installation

To install pinion, run:

yarn add global @colony/pinion

or

npm i -g @colony/pinion

Usage (for the impatient)

And then run pinion passing an IPFS node endpoint and a pinning room:

PINION_ROOM=YOUR_PINNING_ROOM pinion

In this configuration we're assuming some sensible defaults. See below.

Custom configuration

Pinion can be configured by either passing in the configuration programatically to its constructor (with the only required value being the room, see defaults in the example):

import Pinion from 'Pinion';

const pinner = new Pinion('YOUR_PINNING_ROOM', {
  ipfsRepo: './ipfs',
  ipfsPrivateKey: 'CAA...',
  maxOpenStores: 100,
  orbitDBDir: './orbitdb',
});

Or using environment variables when running it from the command line:

PINION_IPFS_CONFIG_FILE=./ipfsConfig.production.json PINION_ROOM=YOUR_PINNING_ROOM PINION_IPFS_REPO=./ipfs PINION_IPFS_PRIVATE_KEY="CAA..." PINION_MAX_OPEN_STORES=100 PINION_ORBIT_DB_DIR=./orbitdb pinion

PINION_ROOM

(required)

The IPFS pubsub room pinion is going to join and listen to new messages to.

PINION_MAX_OPEN_STORES

(optional)

You can also specify the limit of how many stores you wanna keep open simultaneously by passing in an environment variable MAX_OPEN_STORES. The stores will be automatically allocated using a LRU algorithm. The limit is by default set to 100 stores.

PINION_IPFS_CONFIG_FILE

(optional)

Define a config file to use (see ipfsConfig.production.example.json for an example).

PINION_IPFS_PRIVATE_KEY

(optional)

The private key that is used to initialize the IPFS repo. Will generate a random key when omitted.

PINION_IPFS_REPO

(optional)

You can specify the an IPFS repo path of your preference. The default is ./ipfs.

PINION_ORBIT_DB_DIR

(optional)

You can specify the orbit-db path option so stores data are kept in the place of your preference. The default is ./orbitdb

Debug

Pinion is still on its infancy and you might need debug info or a more detailed output to figure out if it misbehaves. To run it on verbose/debug mode, please also set an environment var like so DEBUG='pinner:*'.

API

Requests

REPLICATE

Opens a store, loads it and keep listening to it until it's being cleaned up by the LRU cache.

Parameters
  1. address - An orbit-db-store address.
Payload example
 {
   type: 'REPLICATE',
   payload: { address: '/orbitdb/Qma=/my-store' },
 };

PIN_HASH

Request the IPFS node to pin the content hash.

Parameters
  1. ipfsHash - An IPFS multihash. Emits a pinnedHash event passing the ipfs hash back.
Payload example
 {
   type: 'PIN_HASH',
   payload: { ipfsHash: 'Qma=...' },
 };

Responses

HAVE_HEADS

Published when the pinner has opened a store and it's ready. It will contain the count of heads that the pinner has for this store.

Payload example
 {
   type: 'HAVE_HEADS',
   to: '/orbitdb/Qma=/my-store',
   payload: {
     address: '/orbitdb/Qma=/my-store/<signature>',
     count: 100,
     timestamp: 10010203993
  },
 }
ANNOUNCE_PINNER

Published when the pinner has started, or in response to an ANNOUNCE_CLIENT message.

Payload example
 {
   type: 'ANNOUNCE_PINNER',
   payload: {
     ipfsId: 'Qm...',
  },
 }

Contributing

We welcome all contributions to Pinion. You can help by testing, suggesting new features, improving performance or documentation.

Please read our Contributing Guidelines for how to get started.

To run the tests

Start an ipfs node on localhost on port 4001. You can use the commands provided in the package.json using either yarn ipfsd-go or yarn ipfsd-js (Docker has to be running on your system).

Then, in another terminal window do:

yarn test

License

Pinion is MIT licensed

pinion's People

Contributors

chmanie avatar dependabot[bot] avatar jameslefrere avatar renovate-bot avatar thiagodelgado111 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pinion's Issues

Open source pinion

It doesn't look like there's much that prevents us from open sourcing this (actually I can't come up with anything, really). @thiagodelgado111 what do you think?

Edit: I think we have a logo now. So left to do:

  • Add CODE_OF_CONDUCT
  • Add logo to README
  • Use official orbit-db release
  • Migrate to TS
  • Make it stable

Add 'unpin' action

Add an UNPIN_HASH action or similar that calls ipfs.pin.rm(hash)

Use case:

  • a user uploads an avatar
  • pinion pins the avatar
  • the user later removes their avatar
  • pinion should unpin the avatar

"Module not found error" with `npm i -g @colony/pinion`

Node 12.10.0

Error: Cannot find module 'multicodec/src/name-table'
Require stack:
- /home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/node_modules/ipfs-bitswap/src/types/message/index.js
- /home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/node_modules/ipfs-bitswap/src/want-manager/index.js
- /home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/node_modules/ipfs-bitswap/src/index.js
- /home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/node_modules/ipfs/src/core/components/start.js
- /home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/node_modules/ipfs/src/core/components/index.js
- /home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/node_modules/ipfs/src/core/index.js
- /home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/lib/IPFSNode.js
- /home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/lib/Pinion.js
- /home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/lib/index.js
- /home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/bin/index.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:772:15)
    at Function.Module._load (internal/modules/cjs/loader.js:677:27)
    at Module.require (internal/modules/cjs/loader.js:830:19)
    at require (internal/modules/cjs/helpers.js:68:18)
    at Object.<anonymous> (/home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/node_modules/ipfs-bitswap/src/types/message/index.js:10:19)
    at Module._compile (internal/modules/cjs/loader.js:936:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:947:10)
    at Module.load (internal/modules/cjs/loader.js:790:32)
    at Function.Module._load (internal/modules/cjs/loader.js:703:12)
    at Module.require (internal/modules/cjs/loader.js:830:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/node_modules/ipfs-bitswap/src/types/message/index.js',
    '/home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/node_modules/ipfs-bitswap/src/want-manager/index.js',
    '/home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/node_modules/ipfs-bitswap/src/index.js',
    '/home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/node_modules/ipfs/src/core/components/start.js',
    '/home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/node_modules/ipfs/src/core/components/index.js',
    '/home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/node_modules/ipfs/src/core/index.js',
    '/home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/lib/IPFSNode.js',
    '/home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/lib/Pinion.js',
    '/home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/lib/index.js',
    '/home/mark/.nvm/versions/node/v12.10.0/lib/node_modules/@colony/pinion/bin/index.js'
  ]
}

Check store sync issue

Currently, we have seeing this warning on the console: "WARNING! Head hash didn't match the contents. We need to investigate and fix that

Close open stores on timeout

After opening a store, we wanna make sure we close them after a period of time so we make sure we don't have an open store we are not using.

We should debounce the store close call using two timeout levels, one for the store being opened and one more for the last message received. So if a store is opened, we will close it after some X time except if a message is received for that store then we extend that timeout a little bit longer

Deploy pinion

For Colony Contribute to use it, we need to deploy pinion and make it accessible to our users :)

cc'ing @area @chmanie

Use orbit-db official release

We've been using a fork but we need to install official releases as soon as we can use them with our changes and the new ipfs-log, opening this just to track that.

Closed #7 in favour of this one

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.