olaven / paperpod

https://paperpod.fm

This project is currently inactive. If you're interested regardless, please reach out :).

Paperpod

Sponsors

Please consider sponsoring this project. This will allow me to give you perks in return for your kindness!

Assets

• Design Workspace in Figma
• Any drawings from here
• Or icons from here

Developer's getting started

Setup

In addition to cloning the source code, you'll most likely want access to an

• AWS account. This account must have
  • an S3 bucket called "paperpod" and have
  • access to AWS Polly.
• a Stripe account with the resources outlined in stripe.tf.

Afterwards, create a .env file containing:

ADMIN_USERNAME=<SOME_ADMIN_USERNAME>
ADMIN_PASSWORD=<SOME_ADMIN_PASSWORD>
GATEWAY_PORT=8080
API_PORT=8081
AUTHENTICATION_PORT=8082
AUTHENTICATION_INTERNAL_PORT=8083
WEB_PORT=8084
DOCS_PORT=8085
METABASE_PORT=8086
JWT_SECRET=<SOME_SECRET_STRING>
PGDATABASE=paperpod
PGPORT=5432
PGUSER=<SOME_USERNAME>
PGPASSWORD=<SOME_PASSWORD>
AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY>
AWS_SECRET_ACCESS_KEY=<AWS_SECRET_ACCESS_KEY>
STRIPE_PUBLISHABLE_KEY=<STRIPE_PUBLISHABLE_KEY>
STRIPE_API_KEY=<STRIPE_API_KEY> # must have permissions to deal with payments and products
ULTRAHOOK_API_KEY=<ULTRAHOOK_API_KEY>
LOG_LEVEL=trace
# below only relevant for infra development:
STRIPE_COUPON_CODE=<STRIPE_COUPON_CODE>
DO_TOKEN=<DO_TOKEN>
GH_TOKEN=<GH_TOKEN>

Working with the code

This project relies heavily on Yarn Workspaces and Docker Compose. A basic understanding of how those work is recommended.

• All commands are run from the root of the project
• To run a yarn command (i.e. add a dependency) to a package, simply run yarn <PACKAGE> <COMMAND> <ARGUMENTS>, e.g. yarn mobile add lodash.
• To boot and develop the web services, run docker compose up
• To view logs from any service, use docker compose logs -f <SERVICE_NAME>, e.g. docker compose logs -f authentication.
• To stop everything, run docker compose down

Finding documentation

The documentation is still a bit lacking, but most if it is available in this and other README.md files in this repository. Furthermore, API docs are available at paperpod.fm/docs.

Recommendations

The project structure is built with docker compose in mind. Docker Compose is great, but the commands can get long and tedious. My recommendation is to add these aliases to your environment, for an easier time:

alias du="docker compose up -d";
alias dp="docker compose ps";
alias dr="docker compose restart";
alias dd="docker compose down";
alias db="docker compose build";
alias de="docker compose exec";
alias dl="docker compose logs -f";

paperpod's Issues

Extension has basic login-component

• Input for username/password
• Basic input validation
• Does call auth endpoint
• Shows a message depending on success/error

make sure the link tags link to the feed itself.

paperpod/packages/converter/src/helpers/rss.ts

Line 47 in 85fc1ff

tag("link", "LINK TO FEED"),

Make sure infrastructure is disposable

There's some catch-22 situations with droplet creation / password.

1. destroy
2. re-create
3. gather what goes wrong
4. figure out fix

refresh medchanism for JWT-tokens

If the token expires (e.g. a user spends 15 minutes on the site), there's no way to get a new token.
In other words, the only way to use the page after token expiry is to refresh the page and log in again.

Browser extension should provide login-menu when needed

Documenting endpoints

"*" pronounced litteraly

Tested in Norwegian and English.
"*" is pronounced "asterix". A possible solution would be to swap this for a longer pause somehow.

Use logger with LOGLEVEL

remove utils to accomodate the fact file paths in s3 are generated on the spot, and not somethng I have to generate

paperpod/packages/common/src/server/utils/article-utils.ts

Lines 1 to 19 in 67922ef

	import querystring from "querystring"
	import { models } from "../..";
	export const getFilename = (article: models.Article) =>
	querystring.escape(
	article.owner_id + article.original_url
	)
	export const parseFilename = (filename: string, owner: models.User) => {
	const unescaped = querystring.unescape(filename);
	const [_, original_url] = unescaped.split(owner._id)
	const [owner_id] = unescaped.split(original_url);
	return {
	original_url,
	owner_id
	}
	}

Use to insert breaks after "*"-points and "." SSML

See: https://cloud.google.com/text-to-speech/docs/ssml

Browser extension should grab the current url on click

Manage Domain Records

with terraform.
Map to manager droplet.

Infrastructure should get Gateway port from env

Port (currently 80) is hardcoded in terraform. This is reduntant, as it is already stored in secrets and consumed by actions.
Terrafom should accept this as an env var.

RSS feed endpoint should be implemented

Enable backups for droplet

Through terraform

Add linting in git hook

Handle user resubscribe

A user that resubscribes should

• have access as before
• get shown a nice message in the day after resubscribing :)

Block api endpoints if not subscribing

Feature: PDF parsing

It should be possible to add PDF files.
This is a very common way of distributing text that
several people have requested.

This will prove especially useful for students or people reading a lot in their job,
where web articles are probably less common than outside of school/work.

Manage Certificates

See if it is feasible to manage certificates through terraform and do loadbalancer link

Article properties should not be optional

Most article properties are optional right now.
This made some development easier, but it is not wize in the long run.
I think most of the properties on Article should be compulsory.

Where that cannot be guaranteed (e.g. converter functions), Partial<Article> should be
used instead of Article.

Articles should have `publication_timestamp`, `added_timestamp`, `description`, and `author`

These fields will greatly improve the metadata that can be added to the RSS-feed (see #16).
All but added_timestamp can be extracted from unfluff.
added_timestamp can just be calculated when persisting.

Landing page should be on subdomain

i.e. landingpage.paperpod.fm

This is the configuration when paperpod.fm was used. Presumably change @ with landingpage or similar to update.

Setup Stripe Checkout

See this tutorial, using pre-built checkoug flow

Port 80 should redirect to 443

The current setup only runs on port 443.
This breaks some links, defaulting to 80.

The server should listen on port 80 and redirect to 443 / Upgrade the connection.

Store audio in S3 to increase character limit from 3k to 100k

'With up to 3000 characters you can listen, download, or save immediately. For up to 100,000 characters, your task must be saved to an S3 bucket.'

Extension crashing on login

uncaught exception: 200 when creating session

Database should use custom db user / db

currently using default db / default db user.

replace translate withAWS? https://docs.aws.amazon.com/comprehend/latest/dg/API_DetectDominantLanguage.html

paperpod/packages/converter/src/helpers/audio.ts

Line 2 in 2fd5cef

//import * as Translate from "@google-cloud/translate"

Login/Signup error should show message

• in web
• in extension

Create full CI deployment with Terraform/Actions

Some groundwork has been done in #88.

• figure out how to pass outputs from terraform to github actions
• update the existing deploy-flow in such a way that it deployes the docker stack to a digitalocean droplet

Publish extensions

Publish outside of the chrome/firefox stores. This allows for installation directly
from the paperpod application and also allows for control over the publishing
process.

• chrome external extension
• firefox self distribution

fix type ignoring in article routes. May be fixed by #13.

paperpod/packages/api/src/routes/article-routes.ts

Line 24 in 007353c

{ original_url: link, owner_id: user._id }

Provide more information in Article

Suggestions:

• Article duration
• Image / other visual context (might be hard, perhaps favicon could be a decent substitute :D )
• Category
• Stream url

Browser extension should be usable on firefox

Drop this for now if it turns out to be a lot of work.

Test coverage on web

• Add same test coverage tresholds to web as to other modules
• Implement tests to pass that treshold

Board Test

Creating this issue to see if the Developmnt board catches it.

Update small icon to Pa

In memory instance of postgres when running tests.

Or at least not depending on a database already existing.

Extension stores login-state

• figure out how to do this -> Chrome Storage
• implement

Experiment with serif fonts

To mimick paper

Research Spotify

Is it possible to add RSS to spotify?
/ paperpod? Alternatively, find another solution?

Converting long articles: Error: 3 INVALID_ARGUMENT: 5000 characters limit exceeded.

Error: 3 INVALID_ARGUMENT: 5000 characters limit exceeded.

sync token from extension using background scripts

state can be synced using storage. See #122

Extension should succeed with saving

Fix deprecated `uuid` from `faker`

Login with enter

Use Loom to explain Paperpod flow

Depends on a customer-presentable design

Browser extension should handle POST OK/ERROR with a message

Handle user unsubscribe

• Show a friendly message and point them to subscribe again :-)
• Make sure they can't read from their feed. (return an audio message telling them how to resubscribe?)