prereview / rapid-prereview

An application for rapid, structured reviews of outbreak-related preprints

PREreview

A platform for reviewing preprints.

Requirements

• Node.js of version >=12
• npm

Structure

Composed of 3 different parts:

• A React-based frontend that provides a standalone webform tool to submit tickets.
• A Koa-based backend that renders & serves the frontend and exposes an API used by the frontend.

These parts are located here in this repository:

src/backend  # The backend components
src/common   # Common code and assets
src/frontend # The React frontend

Configuration

PREreview is configured via variables either specified in the environment or defined in a .env file (see env.example for an example configuration that may be edited and copied to .env).

The backend parses the following configuration variables:

PREREVIEW_LOG_LEVEL       # Logging level (default: error)
PREREVIEW_HOST            # The host PREreview runs on (default: localhost)
PREREVIEW_PORT            # The port to bind to (default: 3000)
PREREVIEW_ADMIN_USERNAME  # The administrative user (default: 'admin')
PREREVIEW_ADMIN_PASSWORD  # The administrative password
PREREVIEW_DB_HOST         # Postgres database host (default: localhost)
PREREVIEW_DB_PORT         # Postgres port (default: 5432)
PREREVIEW_DB_DATABASE     # Postgres database name (default: prereview)
PREREVIEW_DB_USERNAME     # Postgres user (default: prereview)
PREREVIEW_DB_PASSWORD     # Postgres password
PREREVIEW_DB_POOL_MIN     # Postgres minimum connections (default: 0)
PREREVIEW_DB_POOL_MAX     # Postgres max connections (default: 10)
PREREVIEW_DB_TIMEOUT      # Postgres connection timeout (default: 0)

Additionally, we use the semi-standard NODE_ENV variable for defining test, staging, and production environments as well as log4js for setting logging verbosity.

Deployment

Standalone

First, clone this repository and from the root of the resulting directory install dependencies:

npm install

Then, build all components:

npm run build

Create the database:

npm run db:migrations

And start the running processes (with necessary environment variables if not defined in .env):

npm run start

(use npm run start:dev to run in development mode)

Additionally, components can be built or started individually using for example npm run build:backend, npm run start:worker, etc.

Docker

You can deploy this tool using Docker. There is an included docker-compose.yml file that will allow you to run it in a production configuration. First, clone the repo and from this directory run docker-compose:

docker-compose up --build -d

This will build the docker container from the current repository, download the official Postgres docker image, and configure them both (the -d flag will detach from the current shell so that you can leave it running, but you can omit it in order to leave the log output attached).

If this is the first time you've run it on this system, you'll want to run the database migrations to initialize the database:

docker-compose run prereview npm run db:migrations

By default, it runs on http://localhost:3000, but you can place it behind a proxy such as Nginx in order to provide TLS support and other features.

License

PREreview is an open-source software project licensed under the MIT License by PREreview.

rapid-prereview's Issues

Admin level features (moderation)

• Allow admins to retract reviews not respecting the CoC
  This can only happen for the free form questions. If we detect that a free form question violates the CoC, do we keep the review data as part of the summary stats (the yes/no questions can't be in violation by design) or do we totally discard the full review? Another way to see that is to decide on the granularity of the review moderation process. Do we moderate at the answer level or do we moderate at the review (as a whole) level ?
• Allow admins to ban personas not respecting the CoC
  Do we keep the review data of banned personas as part of the review summary stats (barplot)?

extension: update popup on history pushState and replaceState

See https://stackoverflow.com/questions/4570093/how-to-get-notified-about-changes-of-the-history-via-history-pushstate

window.onpopstate is not fired with pushState or replaceState.

title (head)

can do with react-helmet

add logging

Use https://www.npmjs.com/package/express-pino-logger

Automatic tweet when new rapid review is posted

use touch events for shell on mobile

Drag and drop over a PDF object is currently broken in Chrome for Mac

See https://bugs.chromium.org/p/chromium/issues/detail?id=984891&q=drag%20object&colspec=ID%20Pri%20M%20Stars%20ReleaseBlock%20Component%20Status%20Owner%20Summary%20OS%20Modified
Issue is active and should be fixed promptly per the comments.

If not, a possible workaround is to resize the <object> so that the PDF is not below the shell when the user starts to drag and to resize it back to 100vh after the drop happened.

Rapid PREreview form

Update: first round of mockups is available in the designs directory (be sure to download the PDF to view all the 3 screens as the github preview doesn't seem to always render the first one).

Structured fields

type	required	used in faceted search	label & options
keywords	no	no	keywords
yes/no/n.a/unsure	yes	yes	Do the findings support the conclusion
yes/no/unsure	yes	yes	Would I recommend this
multi yes/no/n.a/unsure	yes	yes	Relevant to: policy clinic ...
yes/no/n.a/unsure	yes	yes	Novelty of the findings
yes/no/n.a/unsure	yes	yes	Is there a basis for future work
multi yes/no/n.a/unsure	yes	yes	Needs specialized attention: stats methods model ethics data quality
multi yes/no/n.a/unsure	yes	yes	Availability data code not applicable
yes/no/n.a/unsure	yes	no	Does the paper discuss limitations

Free form fields

type	required	label
textarea (~200 words)	no	Technical comment on methods, data, limitations
textarea (~200 words)	no	Editorial comment on novelty, importance, relevance

Sources:

• https://prereview.slack.com/archives/CNCG9UC15/p1568990434000600
• 

Add ability for users to download data from the UI

(in addition to API access)
This could be combined with the share menu (to grab the permalink etc.)

list of infectious diseases

Needed for faceted search.
Candidate: https://en.wikipedia.org/wiki/List_of_infectious_diseases

Modify Rapid CoC prior to launch

@dasaderi to modify here in a new coc branch and label changes with "TODO @sballesteros".

• Specify that content will be moderated after it's posted. Community members will be able to flag potential violations and admins will evaluate and decide if the content needs to be removed and/or the user needs to be blocked.

• Specify that the site is for outbreak-related content only.

extension popup icons

We need 2 icons to replace the R from the screenshot above:

• one in default state (the extension in not active) <- when user navigate pages irrelevant to Rapid.
• one in active state <- when user is on a page that can be reviewed.

In the active state the icon will be complemented by a badge with a count (number of reviews + requests) using the setBadgeText and setBadgeBackgroundColor methods.

See https://developer.chrome.com/extensions/browserAction setIcon section for the details on format / resolution etc.

User notifications (email)

Updated per discussion with @dasaderi and @SamanthaHindle on slack

User notifications (via email)

• when a user review has been moderated (note that this is triggered only after moderators validated user reports and not each time a user reports a potential issue).
• when a user has been blocked (lose write access to the platform) by a moderator.
• when a review is added to a preprint for which the user had requested feedback (via a request for reviews).

Implementation

Backend

• Get a default email address from ORCID (see https://members.orcid.org/api/tutorial/read-orcid-records )
• Allow user to update that email (or set it in case where no emails could be found in the public ORCID record)
• Add logic to verify the email address (only for emails not imported from ORCID). Notifications will only be sent to emails that are verified. Verification will be done by sending an email with a "magic" link to the user. If the user receives the email and clicks on the link the email address will be considered verified.
• Add ability to set notification preferences (see frontend section below)
• Send the emails with sendgrid on Azure. Emails will be sent as plain text only (so unstyled).

Frontend

• UI to display and update the email address (part of the settings). Note that the email address is never displayed publicly and is only used for notification purposes.
• UI to keep track of the verification status of an email
• UI for the notification preference settings (YES / NO) to assess whether or not notifications should be sent when new reviews are added to a preprint for which the user requested feedback

use react-pdf as a fallback on mobile (or when browser doesn't have a plugin to display PDFs)

See https://github.com/wojtekmaj/react-pdf

List of preprint servers for which the extension should activate

@dasaderi @majohansson do you have a list of preprint servers that you think will be used for Rapid / Outbreak science ?
I think we will add a feedback mechanism in the extension, but to start, i would like to have a hard coded list of the preprint servers for which the web extension should be active (i.e allow users to add /request reviews).

To provide more details: we inject the shell on recognized domains using a list like:

  "content_scripts": [
    {
      "matches": ["*://*.arxiv.org/*"],
      "js": ["content-script.js"],
      "css": ["content-script.css"],
      "run_at": "document_end"
    }
  ]

(see https://developer.chrome.com/extensions/content_scripts#declaratively if you are curious)

My guess is that we can start with 10-20 domain that we (you!) know (arxiv, biorxiv etc.), and crowdsource the rest.
If you can add the domain of those preprint servers that would be great! If not I can do some research.

What main "service" does Rapid PREreview provide

!! work in progress (draft)

Rapid PREreview focuses on providing the best infrastructure to request / provide / analyze feedback (structured reviews) on existing preprints relevant to the outbreak community.

The feedback should be of use to:

1. the outbreak community (academics)
2. workers, editors, journalists (visualization etc.)

Rapid PREreview does not focus on:

• coordinating research effort / data analysis / calling for research during emergency situations
• becoming / being a preprint server

See also:

• https://twitter.com/outbreaksci (feed on new preprint manuscripts and ProMED posts related infectious disease outbreaks)
• https://github.com/outbreaksci/rapid-reviews (google form as example of structured review template)

extension popup

From discussion with @halmos on slack:

[Section when user is on a Reviewable Pages]
- X Reviews / X Requests
- Add Review (clicking maximize the shell + toggle on right tab)
- Add Request (clicking maximize the shell + toggle on right tab)
----
[Preprints by score section]
- title + number of review / request
(+ pagination to view beyond)
----
[PREreview Global Section]
- PreReview Home
- My Account
- Login/Logout (optional based on auth stuff)

redirect to preprint when user is asked to login when preprint info has already been entered

right now login redirect to /home or /settings based on profile completion. We should add a special case so that if a user had started to search a preprint he/she is redirected to where he/she was prompted to login.

do not refetch search result on navigation within the app if query hasn't changed

API key

• CreateApiKeyAction
• API: HTTP Authorization header Authorization: ApiKey xxxx see https://stackoverflow.com/questions/33385497/why-should-you-base64-encode-the-authorization-header and https://stackoverflow.com/questions/32157983/unable-to-decode-base64-from-req-header-authorization
• UI (new section in profile settings)
• document how to POST RequestRapidPREreviewAction using the API key (basis of third party integration of Rapid)

Best starting point(s) to engage users to start writing / reading prereviews in the context of the Rapid PREreview project

Context

We want to assess the best starting point(s) to engage users to start writing / reading prereviews in the context of the Rapid PREreview project (see #4 ).

We currently have 3 (non mutually exclusive) approaches in mind:

1. "Homepage approach": User starts from the PREreview preprint discovery page or a dedicated page of the PREreview and/or Outbreak Science websites (for instance a page allowing the user to select the article object of the review with a URL, citation or DOI).
2. "Extension approach": User starts from a browser extension that the user can download from the PREreview website and/or Outbreak Science website (or browser extension stores).
3. "Disqus approach": User starts directly from a preprint service website

User behavior

To get started we would like to focus on a high level discussion centered on how users currently:

1. Discover new preprints (e.g., preprint service homepage?, google scholar? twitter?)
2. Share (and refer to) discovered preprints (e.g., share link to the PDF? share preprint "canonical" homepage / URL / DOI ?)
3. Discover comments (and/or) reviews made about preprints (e.g. bioarxiv uses Disqus)
4. Share (and refer to) comments (and/or) reviews made about preprints (e.g. email notifications from users followed on Disqus ?, comment / review URL or DOI)

And if these behaviors are expected to be different in the Rapid PREreview context.

User acquisition effort

We would also like to discuss the user acquisition effort associated with each approach:

• effort needed to get the user to adopt a new homepage
• effort needed to get the user to install a web extension
• effort needed to get preprint services to integrate a Rapid PREreview script (Disqus approach)

Appendix

To provide some common vocabulary for the discussion we will consider several UI patterns:

• "Shell"
  
• "Extension popup icon"
  
• "Right panel overlay"
  
• "System notification"
  

switch icon to force activate extension

Right now the extension is only activated based on a fixed list of preprint server. That should be fine for most cases, however if that were to be a problem, we could add a Switch toggle to allow user to force activate the extension on pages not covered by our list. We would store those extra pages in local storage so the user only has to do that once.
That would allow the user to immediately use the extension without waiting for us to patch the list etc.

Add public API documentation

secure express app

Use https://github.com/helmetjs/helmet

favicons

Probably the same icon as the extension popup icon ?

Admin notification of a new Rapid Review

It would be very useful for admins to get a notification of a new review. I'd be happy with an email sent to [email protected].

This is both because we can go and moderate, and also because we can share on social media. Also see Issue #49.

@majohansson, what do you think?

Switch to custom domain

outbreaksci.prereview.org

• CNAME: https://docs.microsoft.com/en-us/Azure/app-service/app-service-web-tutorial-custom-domain.
• SSL cert: https://docs.microsoft.com/en-us/Azure/app-service/configure-ssl-certificate#create-a-free-certificate-preview

rename to Outbreak Science Rapid PREreview everywhere and reviews to rPREreviews

per @dasaderi and @majohansson request on slack

• "Outbreak Science Rapid PREreview" everywhere
• reviews are: "rPREreviews" with first r in italic r (rapid)

Internationalization and localization

Restricted to the UI: does not applies to translating user generated reviews.

shell on mobile: the 100vh drama

See https://css-tricks.com/the-trick-to-viewport-units-on-mobile/

setup pm2 on azure

Remaining items to go to production

• switch to production domain (outbreaksci.prereview.org)
• switch to official ORCID API instead of our development mock
• open source the repo (and double check license)
• port extension to Firefox
• publish chrome extension to chrome web store
• publish firefox extension to firefox web store
• update email addresses to latest
• update to latest design for trending score display
• delete the current seed (with fake content / users) and replace with a new one without fake reviews (or start blank or with just request for reviews submitted by us ?)
• finish admin and moderation features
• update Code of Conduct to reflect latest moderation proposal (the community helps to flag content)
• update facet panel and questions to latest ?
• test on chrome
• test on mobile chrome
• test on Firefox
• test on mobile Firefox
• test on Safari
• test on mobile Safari
• test on Edge

DnD support with touch events

Maybe using https://github.com/LouisBrunner/dnd-multi-backend/tree/master/packages/react-dnd-multi-backend ?

Make extension work on our own domain

prefix all CSS classes and ID or use a global reset

We need to protect our content script styles from the page own CSS.
We can either add a prefix to all our classes (probably simplest solution) or try a global reset betting on the fact that extension CSS is injected last (to be verified across browsers).

Handle click outside for side panel on homepage

Clicking outside should close side panel on mobile

prevent users to report reviews violating the CoC if they are not logged in

Data model, API and choice of database

Data model

Everything is in JSON-LD.

Profiles (internal, for operational purposes)

const user = {
  '@context': 'https://rapid.prereview.org',
  '@id': 'user:orcid', // the ORCID of the user
  '@type': 'Person',
  name: 'Romain Gary', // from ORCID (ORCID is source of truth)
  givenName: 'Romain', // from ORCID (ORCID is source of truth)
  familyName: 'Gary', // from ORCID (ORCID is source of truth)
  // etc.

  // The roles (persona). For roles, Rapid PREreview is the source of truth
  // A user can have / manage different persona (roles)
  // Roles can be public or anonynous. Anonymous roles can be
  // made public at any time
  hasRole: [
    // Public role (linked to the ORCID)
    {
      '@id': 'role:romain-gary', // visible display name (user selected and globally unique within Rapid PREreview)
      '@type': 'PublicReviewerRole',
      startDate: '2018-01-01T00:00:00.069Z',
      gravatar: {
        '@type': 'ImageObject',
        contentUrl: 'http://example.com/romain-gary.jpg' // probably base64 encoded to avoid need for blob store
      }
    },
    // Anonymous role (not linked to ORCID or public role)
    {
      '@id': 'role:shatan-bogat', // visible display name (user selected and globally unique within Rapid PREreview)
      '@type': 'AnonymousReviewerRole',
      startDate: '2019-10-02T08:14:50.069Z',
      gravatar: {
        '@type': 'ImageObject',
        contentUrl: 'http://example.com/shatan-bogat.jpg'
      }
    },
    // Anonymous role made public (ended)
    {
      '@id': 'role:emile-ajar', // visible display name (user selected and globally unique within Rapid PREreview)
      '@type': 'AnonymousReviewerRole',
      startDate: '2018-01-01T00:00:00.069Z',
      endDate: '2019-10-02T08:14:50.069Z',
      gravatar: {
        '@type': 'ImageObject',
        contentUrl: 'http://example.com/emile-ajar.jpg'
      }
    }
  ]
};

Note: after discussion with @dasaderi and @majohansson on slack, for now we will follow the approach of PREreview where there is only 1 persona (by default private / anonymous) and the user can only make that persona public later on. Once the persona is public there is no way back to private.

const registerAction = {
  '@context': 'https://rapid.prereview.org',
  '@type': 'RegisterAction',
  agent: 'user:orcid'
};

const createRoleAction = {
  '@context': 'https://rapid.prereview.org',
  '@type': 'CreateRoleAction',
  agent: 'user:orcid',
  object: 'user:orcid',
  payload: {
    '@id': 'role:emile-ajar',
    '@type': 'AnonymousReviewerRole'
  }
};

const updateRoleAction = {
  '@context': 'https://rapid.prereview.org',
  '@type': 'UpdateRoleAction',
  agent: 'user:orcid',
  object: 'role:roleId',
  payload: {
    gravatar: {
      '@type': 'ImageObject',
      contentUrl: 'http://example.com/emile-ajar.jpg'
    }
  }
};

const deanonymizeRoleAction = {
  '@context': 'https://rapid.prereview.org',
  '@type': 'DeanonymizeRoleAction',
  agent: 'user:orcid',
  object: 'role:roleId'
};

Reviews and Request for reviews (public and replicated)

const reviewAction = {
  '@id': 'action:reviewActionId',
  '@type': 'RapidPREreviewAction',
  agent: 'role:roleId', // can be an anonymous role
  actionStatus: 'CompletedActionStatus',
  startTime: '2019-10-02T08:14:50.069Z',
  endTime: '2019-10-02T08:14:50.069Z',
  object: 'preprint:doi',
  resultReview: {
    '@id': 'review:reviewId',
    '@type': 'RapidPREreview',
    dateCreated: '2019-10-02T08:14:50.069Z',
    about: {
      '@type': 'OutbreakScienceEntity',
      name: 'zika'
    },
    reviewAnswer: [
      {
        '@type': 'YesNoAnswer',
        parentItem: {
          '@type': 'YesNoQuestion',
          text: 'Is this a good data model ?',
          suggestedAnswer: [
            {
              '@type': 'Answer',
              text: 'Yes'
            },
            {
              '@type': 'Answer',
              text: 'No'
            },
            {
              '@type': 'Answer',
              text: 'Not applicable'
            },
            {
              '@type': 'Answer',
              text: "I don't know"
            }
          ]
        },
        text: 'Yes'
      },
      {
        '@type': 'Answer',
        parentItem: {
          '@type': 'Question',
          text: 'A question asking for a free text answer'
        },
        text: {
          '@type': 'rdf:HTML',
          '@value': '<p>hello HTML</p>'
        }
      }
    ]
  }
}

const requestReviewAction = {
  '@context': 'https://rapid.prereview.org',
  '@id': 'action:requestReviewActionId',
  '@type': 'RequestForRapidPREreviewAction',
  agent: 'role:roleId', // can be an anonymous role
  actionStatus: 'CompletedActionStatus',
  startTime: '2019-10-02T08:14:50.069Z',
  endTime: '2019-10-02T08:14:50.069Z',
  object: 'preprint:doi'
};

Preprint (internal, for indexing purpose)

const preprint = {
  '@context': 'https://rapid.prereview.org',
  '@id': 'preprint:doi',
  '@type': 'ScholarlyPreprint',
  url: 'https://example.com/homepage',
  identifier: 'doi',
  doi: 'doi',
  name: 'Preprint title',
  preprintServer: {
    '@type': 'PreprintServer',
    name: 'bioRxiv',
    url: 'https://www.biorxiv.org/'
  },
  encoding: [
    {
      '@type': 'MediaObject',
      encodingFormat: 'application/pdf',
      contentUrl: 'http://example.com/preprint.pdf'
    },
    {
      '@type': 'MediaObject',
      encodingFormat: 'text/html',
      contentUrl: 'http://example.com/preprint.html'
    }
  ],
  datePosted: '2019-10-02T08:14:50.069Z',
  dateScoreLastUpdated: '2019-10-02T08:14:50.069Z',
  score: 21, // this is a time sensitive quantity so need to be updated frequently (we only keep updating the ones that are greater than a value (as score -> 0 as time flows) ...
  potentialAction: ['action:reviewActionId', 'action:requestReviewActionId']
};

API

POST /action

GET /resolve?id=doi

Used to return metadata about a preprint see #9

GET /action/:actionId

GET /action?q=search

GET /preprint/:preprintId

GET /preprint?q=search

GET /user/:userId

GET /question/:questionId

GET /role/:roleId

Display stats about a role

GET /feed

Changes feed is only for the public data (reviews and request for reviews)

Database

Main goal is to provide a super fast and smooth faceted search experience at the lowest possible operational complexity (and cost).

Candidates:

• CouchDB [+ Redis] (if we need replication)
• PostgreSQL + Redis (Redisearch)
• Redis as primary database with Redisearch (see https://www.youtube.com/watch?v=SGEG5q4LT9s)

Add a red dot or smtg to the user badge in header (and shell) when user profile is not complete

Smtg subtle but visible enough to catch user attention + highlight the profile settings menu item and add a CTA there to incentivize user to fill the missing profile info

add production domain to ORCID

callback URL is:
https://outbreaksci.prereview.org/auth/orcid/callback
(in addition to the azure one https://rapid-prereview.azurewebsites.net/auth/orcid/callback)

data for chrome and firefox webstores

• icon (@halmos) must respect the exact width and height. See screenshot below and https://developer.chrome.com/webstore/images#icons

• detailed description (@dasaderi or @majohansson ) see screenshot below for instructions

• provide info to chrome and firefox stores and resubmit the application (@sballesteros)

---

Reviewer comments:

Chrome

Your item did not comply with the following section of our Program Policies:
"Spam and Placement in the Store"
Item has a blank description field, or missing icons or screenshots, and appears to be suspicious.

Firefox

Jack Thompson wrote:
Please expand this add-on listing. Describe the purpose and features/interface changes/actions of this add-on in more detail and/or have images demonstrating the features. For suggestions on how to create a better listing see https://developer.mozilla.org/Add-ons/Listing .
Thank you.

Delete a rapid review (not a CoC violation)

Likely we will have to deal with users who submitted the review as a mistake. We as admin need a way to delete content that is not then labled as CoC violation.

@sballesteros does this require development separate from what we already have?

Add "Give feedback" button to beta

• @dasaderi to make feedback form to link

• @sballesteros and @halmos to add "Give feedback" button to platform by December 3rd

• add beta somewhere to make it obvious it's a beta.

Should we have it so that users that try it don't actually end up adding rapid reviews until it's fully launched?

structured review templates ownership

!! work in progress (draft)

Who can create / update structured review templates ?

• only outbreak science / Rapid PREreview
• any users
• special users

How many structured review templates do we expect ?

[extension] Firefox doesn't load content scripts on PDF pages

See https://bugzilla.mozilla.org/show_bug.cgi?id=1454760
It works fine in chrome.
Hopefully Firefox will fix that.

MVP

Update: first round of mockups is available in the designs directory (be sure to download the PDF to view all the 3 screens as the github preview doesn't seem to always render the first one).

Note: in what follows all the authentication / user account / profile creation & display is done by PREreview.

In terms of "release strategy" main idea would be to release this work under a rapid.prepreview.org subdomain under an experimental/beta flag to get user feedback asap, iterate on that feedback and then integrate the good parts to PREreview

Main page

The goal of the main page is to establish Rapid PREreview as the place where the outbreak science community comes to request, get and provide rapid feedback on their preprints.

The page is centered around:

• a searchable list of preprints for which users have provided or expressed a desire to get feedback through Rapid PREreviews
• clear call to actions to:
  • create a new Rapid PREreviews
  • get feedback on existing preprint content (request for Rapid PREreviews)
  • install the web extension to do the same thing even more easily (see section further down)

List

To start as simple as possible, the first implementation of the list will sort items by date of last Rapid PREreview created.

Later we could refine that with a score better suited to provide visibility to preprints with a high demand for feedback. Tentative definition:

score = (v+r) / (t+1)^g

where,
v = number of votes of an item
r = number of Rapid PREreviews of an item
t = time since request submission or first Rapid PREreview (unit (hours, days, weeks) to be determined)
g = tuning factor (to be determined)

Displayed data / controls for each item:

• preprint title
• preprint server
• preprint DOI
• number of Rapid PREreviews (and/or reviewer name (or anonymous alias) with link to their PREreview profile)
• number of upvotes (request for Rapid PREreviews)
• visualization of the aggregated data collected from the structured reviews
• date of first & last review made
• call to action to add a Rapid PREreview
• call to action to request a Rapid PREreview or express the desire to see more reviews (upvote)

Search options:

• Facets (filter by):
  • preprints with requests for Rapid PREreviews (votes)
  • preprints with Rapid PREreviews
  • Preprint server
  • Dates of last posted review (last week, last month, etc.)
  • Structured data collected by the Rapid PREreview creation form (see #6 for the list of facets)
• Full text search:
  • Author name / username / anonymous alias of the Rapid PREreview creator
  • Rapid PREreview textual content (if any)
  • preprint title
• Other indexes
  • preprint DOI
  • preprint URL

When users perform a search the end of the list will re-iterate the call to actions to start a new Rapid PREreview or request for Rapid PREreview in case the desired content wasn't found.

Users opting to view details about an item are taken to the Rapid PREreview display page (see section further down)

Note: this list (and associated search indexes) is only generated from the data collected during the rapid PREreview creation process to minimize complexity and avoid having to maintain a sync engine / index of all the different preprint servers content. However, PREreview already provides and maintains an index of all the different preprint servers so we could merge with PREreview at a later stage. In the same way, if the maintenance of the index is difficult, PREreview could adopt the simpler approach described here.

Create new Rapid PREreview call to action

Users clicking on the call to action are asked to provides a DOI (or URL) of the preprint to review. At that point we try to get as much metadata as we can from this information (see #9) and transition the user to the Rapid PREreview creation and display page (see section further down). As a fallback, if we couldn't get metadata about the preprint the user will be asked to enter the DOI and title manually.

(if user is not logged in, user is sent through the PREreview login workflow and redirected to the page on login success)

Request for Rapid PREreviews call to action

Logged in users can:

• add entries by providing a DOI (or URL) using the same workflow as described above
• upvote existing entries (see score definition above)

Install web extension call to action

(see section further down for description)

Rapid PREreview creation and display page

This page is designed to work as a "fallback" in cases when users haven't / cannot / don't want to install the web extension (see dedicated section below). It is however fully functional (and styled).

The rendering logic depends on the Content Security Policy set by the preprint server hosting the content to review:

• if framing is allowed, an iframe displaying the preprint content is rendered and complemented by:
  • a panel (or shell) containing the list of existing Rapid PREreviews
  • a shell containing the Rapid PREreview creation form (see #1 for screenshot of a "shell")
• if framing is not allowed (e.g., Content-Security-Policy: frame-ancestors 'none') then a UI with a full window version of the list of existing Rapid PREreviews and the Rapid PREreview creation form is displayed (and complemented by metadata about the preprint (title etc.)).

The list of existing rapid PREreviews will be complemented with a summary view (interactive data visualization) allowing the user to quickly visualize aggregated results.

In both case a (permanently dismissible) call to action to install the Rapid PREreview browser extension is displayed.

When the user is not logged in (or is logged in but has already posted a Rapid PREreview), the UI to create new Rapid PREreviews is not displayed (a call to log in is added when the user is not logged in).

Note: this page is relatively similar to the one provided by PREreview with the main difference that the UI is designed to be able to also work in an extension context where we do not have control on the host page. It will be relatively easy to backport the structured review functionality to PREreview (allowing users to select the Rapid PREreview template from PREreview) or to modify the PREreview page to use a similar overlay UI and leverage the web extension described below.

Web extension

Note: development of the web extension will only start after the other parts of the MVP have been completed.

The goal of the web extension is to bring the features of Rapid PREreview where / when the user need it (therefore reducing the effort needed to write or request Rapid PREreviews). The extension is designed so that if a user visit a preprint, he/she is informed (in a non intrusive way) that he/she can Rapid PREreview it immediately (and without context switch) or ask for it to be reviewed (see upvotes and scores above).

Popup icon

the popup icon is part of the browser and lives next to the browser URL and as such is visible at all time

The icon is "highlighted" (different color / icon etc.) when:

• user is on a preprint content page that has been reviewed by members of the Rapid PREreview community (in this case the icon is complemented by a badge indicating the number of Rapid PREreviews)
• user is on a preprint content page that could be reviewed with a rapid PREreview
• user is on a preprint content page object of a "request for rapid PREreview" (in this case the icon is complemented by a badge indicating the number of requests (upvotes))

Popup UI

Clicking on the popup icon opens a popup window/menu (located right below the popup icon) containing (when relevant):

• An invitation for the user to start to write a Rapid PREreview for the page currently visited by injecting new overlay UI to the page (see content script section below)
• An invitation for the user to request for Rapid PREreview for the page currently visited
• The list of preprints for which users have requested Rapid PREreviews sorted by score (see definition above)
• A feedback section to improve the quality of the extension and allow users to easily:
  • report URLs where the popup icon should have activated but didn't
  • report URLs where the popup icon should not have activated but did

Content script

features injected by the extension to the webpage currently visited by the user

The content script will share its code with the Rapid PREreview creation and display page (see section above) and inject 2 UI elements (working as overlay) to the page visited:

Rapid PREreview editor

Rapid PREreview creation form displayed in a shell

Existing Rapid PREreviews panel

Panel (or shell) containing a list of existing Rapid PREreviews for the visited page content

Note: the web extension could easily be generalized to PREreview if proven useful (and we will track usage / adoption data).

See also:
#1, #3, #4, #6

---

History:

• Rewrite of the main page section based on feedback from @halmos and @georgiamoon on slack (see https://prereview.slack.com/archives/CNCG9UC15/p1568777139003000)
• Clarification of the intents / motivations of the main page and web extension
• Updated link to form to #6
• Simplified list sorting for first version per @majohansson slack comment
• Added link to mockups

Add checkbox to agree to CoC when first log in

If possible to do before December 3rd.

Probabilistic facets

Let's consider a preprint with 10 Rapid PREreviews for the question:
"is the data available?" (possible answer being yes, no, not applicable (n.a)) - see #6 for the current questions.

In the faceted search we want to be able to filter by preprints with data available but we may not have a definitive answer. We could have the following answers:

• 7 yes
• 2 no
• 1 n.a

Our job is to best represent that and provide the best search interface. For the search interface maybe our facet should be (better terms to be found):

• likely (> 60% yes)
• unlikely (< 40% yes)
• unclear (in between 40% and 60% yes)
  (we could be as granular as required with the percentile and add more: e.g., very likely etc..)

Then our query becomes:
"find all the preprints likely to have the needed data available".

The unclear facet is interesting as it can provide an incentive for reviewers to add reviews to remove uncertainties (or check if the divide is justified).

[extension] scope the reach ui attribute selectors

we overwrite some of them so that could clash if the host page where the content-script is injected also uses @reach/ui.
We can use normal classes instead of the attribute selectors.
We would need to exclude those classes from the postcss-plugin-namespace plugin as some reach component are rendered in portals that are typically appended to the body so outside the realm of our scoping.

Getting preprint metadata from unique identifier (DOI or OAI)

Context

arXiv preprints (and maybe others) do not have DOI but instead work with the Open Archives Initiative (see http://www.openarchives.org/)
We need a way to grab the metadata on the fly from a unique identifier entered by the user.

Getting the metadata

DOIs

We query:

• https://api.crossref.org/works/ (See https://github.com/CrossRef/rest-api-doc)
• https://develop.openaire.eu/api.html#pubs

arXiv ID:

See:

• https://arxiv.org/help/oa
• https://support.crossref.org/hc/en-us/articles/213679866-OAI-PMH (note that the GetRecord verb is only available to Metadata Plus subscribers)

Others

Are there other cases that we should consider ? if so what are they?