Coder Social home page Coder Social logo

peerlibrary / peerlibrary Goto Github PK

View Code? Open in Web Editor NEW
263.0 41.0 49.0 11.8 MB

Facilitating the global conversation on academic literature

Home Page: https://peerlibrary.org/

License: Other

Shell 0.49% CoffeeScript 45.27% CSS 6.17% JavaScript 35.63% HTML 8.16% XSLT 4.29%

peerlibrary's Introduction

PeerLibrary

Facilitating the global conversation on academic literature.

https://peerlibrary.org/ | http://blog.peerlibrary.org/ | @PeerLibrary | Facebook

PeerLibrary outreach is done in another repository.

Weekly meeting

Regular general weekly meeting is every Wednesday 5 PM PST (temporary on hiatus). Meeting is open to really everyone. Remote participation is possible using Google Hangout through permanently opened video session. Additionally, join us on our IRC channel at that time for more information. We are also using our IRC channel for a backchannel to the meeting. Notes are taken and sent to the mailing lists, and their address follow the following format http://pad.peerlibrary.org/p/meeting-YYYY-MM-DD.

Development installation

PeerLibrary is built upon the Meteor platform. You can install it with:

curl http://meteor.peerlibrary.org/ | sh

Meteor provides node.js (if you have it installed already you can use the one provided on your system). To add it into your environment, add ~/.meteor/tools/latest/bin to your environment PATH variable. For example, by running:

export PATH=~/.meteor/tools/latest/bin:$PATH

To add tools to your shell permanently, run:

echo 'export PATH=~/.meteor/tools/latest/bin:$PATH' >> ~/.bash_profile

Maybe on your system you have to add the line to ~/.profile file instead.

PeerLibrary requires additional Meteor packages which are provided through Meteorite, a Meteor package manager. Install it as well:

npm install -g meteorite

Other requirements to run PeerLibrary

On first run, PeerLibrary compiles and locally installs additional Meteor packages, some of them have non-Meteor dependencies. The following libraries have to be available on your system for PeerLibrary to successfully run:

On Mac OS X you can get Cairo by installing X11 (Pango and FreeType are already available on the system) and run the following before you run mrt to configure the environment:

export PKG_CONFIG_PATH=/opt/X11/lib/pkgconfig

To be able to compile Meteor packages, you need Xcode with command line tools installed (from Preferences > Downloads > Components).

You can install pkg-config, giflib, and libjpeg using Homebrew (MacPorts also works, if you prefer it):

brew install pkg-config giflib libjpeg

On Debian you can install all dependencies by:

sudo aptitude install libcairo2-dev libfreetype6-dev libjpeg8-dev libpango1.0-dev libgif-dev build-essential g++

Run it!

Recursively clone a PeerLibrary repository:

git clone --recursive https://github.com/peerlibrary/peerlibrary.git

This will give you the latest development version of PeerLibrary (development branch). The latest stable version is in the master branch.

And then run a development instance of PeerLibrary:

mrt

And open http://localhost:3000/, which is an address of your local development instance, to start an installation wizard process in which you create an admin user (which has username admin). After you create an admin user, PeerLibrary will reload.

To demo the tool, you will need to populate your instance of the database with publications. Log in as admin and go to Admin dashboard (http://localhost:3000/admin). Click on the Initialize database with sample data button, to initialize the database with the same publications from arXiv. It will fetch metadata, cache a few PDFs and process them. Publications will be searchable at your http://localhost:3000/.

For more information on configuring your installation, see settings.

ArXiv publications

To load and use arXiv publications, open Admin dashboard (http://localhost:3000/admin) and click on Sync arXiv metadata button first and after it loads all the metadata, click Sync arXiv PDF cache button to load all PDFs. After the caching finishes and PDFs are processed you will be able to search and open arXiv publications in PeerLibrary.

arXiv is a huge repository and loading all the publications takes a lot of space (few 100 GBs) and time. You probably do not want to do this. It consumes arXiv resources and costs you money. Use Initialize database with sample data to get a small sample of arXiv publications.

You will need AWS accessKeyId and secretAccessKey which you have to put into your settings.json file. All PDF transfer costs will be billed against this account.

Free Speech Movement publications

To load and use Free Speech Movement publications, open Admin dashboard (http://localhost:3000/admin) and click on Sync FSM metadata button first and after it loads all the metadata, click Sync FSM cache button to load all the TEI textual documents. After the caching finishes you will be able to search and open FSM publications in PeerLibrary.

You will need FSM API appId and appKey which you have to put into your settings.json file.

Troubleshooting

If you are using Mac OS X and are worried that you may have forgotten some dependencies then run the script checkdependencies.sh located in the PeerLibrary root directory. It checks for most of the dependencies required to run the application.

Sometimes when installing dependencies, Meteor will throw the following error:

npm ERR! cb() never called!
npm ERR! not ok code 0

This just means that there was a timeout while downloading a dependency, probably because of a networking issue. Just retry.

If you have not cloned recursively (if you forgot --recursive in git clone --recursive https://github.com/peerlibrary/peerlibrary.git), you will at some point get a such or similar error:

While building package `blob`:
error: File not found: Blob/Blob.js

Or similar errors for other packages, you just have to manually initialize git submodules we are using:

git submodule update --init --recursive

If you are getting an error like:

Error: Cannot find module '../build/Release/canvas'

Then there is an issue compiling the node-canvas dependency. Check if you have all required non-Meteor dependencies installed and retry by removing the whole meteor-pdf.js package and running mrt again:

rm -rf ~/.meteorite/packages/pdf.js/

If you are getting an error like:

Error: Could not locate the bindings file.

Then you upgraded your system and the package which was previously compiled now does not work anymore. You have to force recompilation by, for example, removing the package. Example for segfault-handler package:

rm -rf ~/.meteorite/packages/segfault-handler/

If you are getting Stylus errors like:

error: Stylus compiler error: client/css/_viewer.styl:2

failed to locate @import file variables.styl

You are not running mrt in the top-level directory of PeerLibrary. This is a bug in Meteor.

If you notice that mrt command disappeared is this because you probably updated Meteor. You have to reinstall Meteorite (npm install -g meteorite).

Contributing

PeerLibrary is currently in active development where we are creating basic architecture. Major code refactoring and rewrites are thus common. Nevertheless, you are invited to join the development, but please understand that things might be changing under your feet so it is probably useful to discuss planned contributions in advance.

See the CONTRIBUTING file for more details and ideas.

You can also help with PeerLibrary outreach, promotion, teaching, and community organizing.

peerlibrary's People

Contributors

aculich avatar annaswigart avatar antineutron avatar chilldude avatar giavjeko avatar kevincfz avatar matejjan avatar mitar avatar mpranjic avatar nschloe avatar opalkale avatar speigg avatar spencerhitch avatar sylvanarevalo avatar tiantongq avatar xaho43 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

peerlibrary's Issues

issues installing and updating (OSX)

dev install instructions followed closely, meteor and meteorite installed, PKG_CONFIG_PATH=/opt/X11/lib/pkgconfig , but meteor encounters errors while updating pdf.js. error output below:

[email protected]:~/proj/peerlibrary$sudo mrt

Stand back while Meteorite does its thing

Done installing smart packages

Ok, everything's ready. Here comes Meteor!

[[[[[ ~/proj/peerlibrary ]]]]]

pdf.js: updating npm dependencies -- btoa, canvas, jsdom, xmldom, pdf.js...
No dependency info in bundle. Filesystem monitoring disabled.
=> Errors prevented startup:
Exception while bundling application:
Error: Command failed: npm http GET https://registry.npmjs.org/canvas/1.0.3
at Object.Future.wait (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/dev_bundle/lib/node_modules/fibers/future.js:322:15)
at Object._.extend.execFileSync (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/tools/meteor_npm.js:231:10)
at Object.
.extend.installNpmModule (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/tools/meteor_npm.js:342:10)
at .extend.createFreshNpmDirectory (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/tools/meteor_npm.js:192:12)
at Function.
.each.
.forEach (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/dev_bundle/lib/node_modules/underscore/underscore.js:86:24)
at Object.
.extend.createFreshNpmDirectory (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/tools/meteor_npm.js:191:7)
at Object.
.extend.updateDependencies (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/tools/meteor_npm.js:85:14)
at _.extend.installNpmDependencies (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/tools/packages.js:325:17)
at _.extend.maybeUpdateNpmDependencies (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/tools/bundler.js:422:13)
at .extend.use (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/tools/bundler.js:461:10)
at self.api.use (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/tools/bundler.js:114:21)
at Array.forEach (native)
at Function.
.each.
.forEach (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/dev_bundle/lib/node_modules/underscore/underscore.js:78:11)
at Object.self.api.use (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/tools/bundler.js:110:9)
at _.extend.init_from_app_dir as on_use_handler
at _.extend.use (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/tools/bundler.js:464:11)
at Object.exports.bundle (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/tools/bundler.js:873:12)
at /Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/tools/run.js:697:26
at exports.inFiber (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/tools/fiber-helpers.js:24:12)
- - - - -
npm http 304 https://registry.npmjs.org/canvas/1.0.3
./util/has_cairo_freetype.sh: line 4: pkg-config: command not found
gyp: Call to './util/has_cairo_freetype.sh' returned exit status 0. while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: gyp failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/dev_bundle/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:420:16)
gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:99:17)
gyp ERR! stack at Process._handle.onexit (child_process.js:678:10)
gyp ERR! System Darwin 12.4.0
gyp ERR! command "node" "/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/dev_bundle/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/cwlb/.meteorite/packages/pdf.js/peerlibrary/meteor-pdf.js/6ac181e4bd7c5ed160ff1d7fe0d9e7c66ef3e656/.npm-new-1tdlxog/node_modules/canvas
gyp ERR! node -v v0.8.18
gyp ERR! node-gyp -v v0.8.2
gyp ERR! not ok
npm ERR! [email protected] install: node-gyp rebuild
npm ERR! sh "-c" "node-gyp rebuild" failed with 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the canvas package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get their info via:
npm ERR! npm owner ls canvas
npm ERR! There is likely additional logging output above.

npm ERR! System Darwin 12.4.0
npm ERR! command "/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/dev_bundle/bin/node" "/Users/cwlb/.meteorite/meteors/peerlibrary/meteor/e7aa118cb9c5c6d6b6faacc5d896abf112a585f7/dev_bundle/bin/npm" "install" "[email protected]"
npm ERR! cwd /Users/cwlb/.meteorite/packages/pdf.js/peerlibrary/meteor-pdf.js/6ac181e4bd7c5ed160ff1d7fe0d9e7c66ef3e656/.npm-new-1tdlxog
npm ERR! node -v v0.8.18
npm ERR! npm -v 1.2.2
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /Users/cwlb/.meteorite/packages/pdf.js/peerlibrary/meteor-pdf.js/6ac181e4bd7c5ed160ff1d7fe0d9e7c66ef3e656/.npm-new-1tdlxog/npm-debug.log
npm ERR! not ok code 0

at ChildProcess.exithandler (child_process.js:540:15)
at ChildProcess.EventEmitter.emit (events.js:99:17)
at maybeClose (child_process.js:638:16)
at Process._handle.onexit (child_process.js:680:5)

Please fix the problem and restart.

We have to optimize search results

The way we have search results now is not really good. Client has to do two round trips to the server to get them. First to fetch IDs and then to fetch data. I suggest we publish publications directly through search, but only those fields we need to display the results. In this way publications will get over fast. Additionally, we should add an extra field (which is not stored in the database) which tells in which all search results this publication is in. Then client can filter based on this field to get results for the current search query.

Meteor knows how to merge fields together if same document comes from multiple publications.

Update relative times as time goes on

If we are using relative time when displaying the timestamp, we should be updating it as time goes on. We should find a reasonable Meteor-friendly solution to this. Is there a library for it?

For Django I created this some time ago. See here.

Possible search queries we want to support

Let's list ideas for possible search queries we want to support:

  • My publications
  • People I follow
  • Publications by people I follow
  • Publications I starred
  • Publications people I follow starred (Publications starred by people I follow, Starred publications by my friends)
  • People followed by people I follow
  • Friends of my friends
  • Publications by people following me
  • Publications I commented on
  • Publications I reviewed
  • Publications I read

PDF thumbnails

I can easily create PDF thumbnails, for example of the first page and create PNGs for them. Do we need this? Maybe to shoe them in search results?

Way to show system notifications

We need some way to show user notifications/messages. For example, when things happen in admin dashboard. Or when user is logged in or logged out. Or some other thing later on.

There are some existing libraries I found already packed for Meteor:

But I am not sure if they provide a way for server to push notification to client.

I am not sure if I like popup notifications. They are simple to implement and they don't move things on the page around, but maybe it would be better to have some constant place to show them on the page? Under the header or something.

Or this are two different ways of notifications. One which is just displayed to the user if user has a session open and another which is stored on the server and is shown to the user until the user confirms it (or some timeout). For the latter notifications see #281 (which are used for another type of notifications, user/content notifications).

Support authors search

We want also to support search for authors. So we should design also how search results for authors look like. Probably we want an image of the author displayed for each result. Do we want to support both avatars and profile images? Avatars for comments, profile image for search results?

Cannot run

When trying to run from terminal with mrt, after installing Meteor and Meteorite, get following error message

  branch: https://github.com/peerlibrary/meteor.git#peerlibrary

Installing smart packages


node.js:201
        throw e; // process.nextTick error, or 'error' event on first tick
              ^
Error: ENOENT, no such file or directory '/Users/Stian/.meteorite/meteors/peerlibrary/meteor/ae3ba58fc94cd63ae80a1269ac276da7dccd288c'
    at Object.mkdirSync (fs.js:381:18)
    at Object.mkdirSyncRecursive (/usr/local/lib/node_modules/meteorite/node_modules/wrench/lib/wrench.js:379:12)
    at [object Object]._load (/usr/local/lib/node_modules/meteorite/lib/sources/git.js:125:12)
    at /usr/local/lib/node_modules/meteorite/lib/sources/git.js:73:18
    at /usr/local/lib/node_modules/meteorite/lib/sources/git.js:89:5
    at /usr/local/lib/node_modules/meteorite/lib/sources/git.js:216:5
    at ChildProcess.exithandler (child_process.js:280:7)
    at ChildProcess.emit (events.js:70:17)
    at maybeExit (child_process.js:360:16)
    at Socket.<anonymous> (child_process.js:457:7)

Slow rendering of big PDFs

We should improve how we are render PDFs. Currently we render all pages in advance initially and we should just render them as user navigates to them.

The second issue that we are currently using pdf.js text selection layer which creates many divs. But we are not really using them. So we should remove this text selection layer and store location of them internally in JavaScript to use it for highlighting. But there is no need to have them in DOM.

For search suggest, parse also on the client side

Currently, we are doing search suggestion parsing on the server side. But we can do some preliminary work on the client side, too. Like tokens parsing itself. We can then improve that by doing search queries on the server side to see which found keywords even exist in the database, if they don't we probably parsed something wrong so we do not suggest that. Or as a general rule, we use those search queries to rank the suggestion. So parsing can be done on client side, but ranking with help of the server side.

Mongo .count() just to see if some entry exists is expensive

Mongo .count() just to see if some entry exists is expensive, because it requires to compute precise count and sometimes whole table scan. It is much cheaper to just see if there is one document or not. And transfer no fields of it, just see if it is there.

Search over PDFs by text location

We can even have search over PDFs by text location, because we know where each text fragment is. So if somebody searches for a keyword, we can open PDF directly at the location of that keyword.

Create an admin and monitoring interface

It would be great to have an admin and monitoring interface directly on the web. So we could see how PDFs are downloaded, which are the last one downloaded and similar things. And also to manually trigger refresh/download.

Login and registration should not require moving to a separate page

A nice feature of Meteor is reactivity. So user should be able to login and/or register without us having to move them away from what they are doing at the moment. We can just login them, set the variable, only parts of UI connect to the user would redraw and this is it.

Define public APIs

At some point we should define public APIs for interaction with data. We want to liberate all data.

User authentication

I propose that authentication workflow for users is as follows:

  • the user account can be connected to third-party authentication providers (Facebook, etc.)
  • in this case profile data is populated, if this was not already done so, username is not chosen based on this data and can be set once after that
  • account can be connected to multiple third-party authentication providers
  • account can be registered on PeerLibrary, in this case password is set on the account and username can be chosen immediately
  • e-mail address is not mandatory to be confirmed, but password recovery and other notifications are not send to it until this is done
  • so e-mail address confirmation support is needed
  • e-mail reset is supported if e-mail address is confirmed (we might not prevent this, because password recovery is also a way to confirm e-mail address, also many workflows currently are sending e-mails in a similar manner: password recovery, account enrollment after invitation)
  • accounts can have avatar, chosen among those provided by third-party authentication providers, gravatar, or uploaded to PeerLibrary
  • account can be temporary disabled by the user, all user content linked to the account should become unavailable
  • account can be permanently deleted by the user, all user content linked to the account should be deleted (or at least marked as such)

Store versions of all changes in the database

We should make a simple wrapper around MongoDB interface which would on every change store the diff into special collection. It can work so that on client it immediately applies the change, but on the server it stores the diff into a special collection and then apply.

With each diff we can also store the user and timestamp.

Probably we should store every so and so also full version of the object in that special collection. This could be done probabilistically. So for every change, with 0.1 probability we store whole object.

Objects should never be deleted. Just marked as deleted.

Objects should allow merging objects together. This could happen when we discover that two authors are in fact one.

We could provide insert and update and remove methods directly on objects.

Connect with European open access sites

People's names not standardized

Publications.authors have foreNames and lastName. Users.profile has firstName and lastName. Comments.author has fullName.

We should standardize this.

Remove insecure package

Currently we are using insecure package. We should not really allow changes of all data from the client. So we should add checks.

We should start using audit-argument-checks package. If there are any issues with packages we are using, we should make pull requests for them upstream.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.