mdouchement / standardfile Goto Github PK

Yet Another Standardfile (standardnotes server) Implementation written in Golang

License: MIT License

Dockerfile 0.30% Go 99.70%

standardnotes golang standardfile notes client-library text-editor terminal-ui standard-notes

standardfile's Introduction

Yet Another Standardfile Implementation in Go

This project is maintained for the basic features as encrypted notes because on its own it already takes hours to figure out what is the issue when a breaking change or bug happens (e.g. #87).

People's pull requests that implement or fix extra features (revision, file storage, etc.) will be gladly reviewed and merged.

For any bug linked to the code or its faulty behavior, please take a look to the existing Issues and feel free to open a new one if nothing match your issue
For any question about this project, the configuration, integrations, related projects and so one, please take a look to the existing Discussions and feel free to open a new one

This is a 100% Golang implementation of the Standard Notes protocol. It aims to be portable and lightweight.

Running your own server

You can run your own Standard File server, and use it with any SF compatible client (like Standard Notes). This allows you to have 100% control of your data. This server implementation is built with Go and can be deployed in seconds.

https://hub.docker.com/r/mdouchement/standardfile

Client library

Go to pgk/libsf for more details. https://godoc.org/github.com/mdouchement/standardfile/pkg/libsf

It is an alternative to https://github.com/jonhadfield/gosn

SF client

go run cmd/sfc/main.go -h

Terminal UI client:

Requirements

Golang 1.16.x (Go Modules)

Technologies / Frameworks

Cobra
Echo
BoltDB + Storm Toolkit
Gowid

Differences with reference implementation

Drop legacy support for clients which hardcoded the "api" path to the base url (iOS)

Permalink

Drop the POST request done on Extensions (backups too)

Permalink

This feature is pretty undocumented and I feel uncomfortable about the outgoing traffic from my server on unknown URLs.

Drop V1 support

All stuff used in v1 and not in v2 nor v3

JWT revocation strategy after password update

Reference implementation use a pw_hash claim to check if the user has changed their pw and thus forbid them from access if they have an old jwt.

Here we will revoke JWT based on its iat claim and User.PasswordUpdatedAt field. Looks more safer than publicly expose any sort of password stuff. See internal/server/middlewares/current_user.go

Session use PASETO tokens instead of random tokens

Here we will be using PASETO to strengthen authentication to ensure that the tokens are issued by the server.

Not implemented (yet)

2FA (aka verify_mfa)
Postgres if a more stronger database is needed
A console for admin usage

License

MIT

Contributing

All PRs are welcome.

Fork it
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create new Pull Request

standardfile's People

Contributors

Stargazers

Watchers

Forkers

chroming cherya techmechanic1 shagr4th squalus crusader99 jayknyn uche40

standardfile's Issues

config for the official web client

Sorry this is not an issue but a question.

What is the best way to pre-configure the official web client (/) to use this backend (/api) when served under the same url?

I can see the client has this config by default:
https://github.com/standardnotes/app/blob/main/packages/web/index.html

<body
  data-default-sync-server="<%= env.DEFAULT_SYNC_SERVER %>"
  data-default-files-host="<%= env.DEFAULT_FILES_HOST %>"
  data-enable-unfinished-features="<%= env.ENABLE_UNFINISHED_FEATURES %>"
  data-web-socket-url="<%= env.WEBSOCKET_URL %>"
  data-purchase-url="<%= env.PURCHASE_URL %>"
  data-plans-url="<%= env.PLANS_URL %>"
  data-dashboard-url="<%= env.DASHBOARD_URL %>"
  data-dev-account-email="<%= env.DEV_ACCOUNT_EMAIL %>"
  data-dev-account-password="<%= env.DEV_ACCOUNT_PASSWORD %>"
  data-dev-account-server="<%= env.DEV_ACCOUNT_SERVER %>"
  >
  <script>
    window.defaultSyncServer = document.body.dataset.defaultSyncServer || "https://api.standardnotes.com";
    window.defaultFilesHost = document.body.dataset.defaultFilesHost;
    window.enabledUnfinishedFeatures = document.body.dataset.enableUnfinishedFeatures === 'true';
    window.websocketUrl = document.body.dataset.webSocketUrl;
    window.purchaseUrl = document.body.dataset.purchaseUrl;
    window.plansUrl = document.body.dataset.plansUrl;
    window.dashboardUrl = document.body.dataset.dashboardUrl;
    window.devAccountEmail = document.body.dataset.devAccountEmail;
    window.devAccountPassword = document.body.dataset.devAccountPassword;
    window.devAccountServer = document.body.dataset.devAccountServer;
  </script>
</body>

which in runtime shows me this (I built it myself with no special env):

    <script>
      window.defaultSyncServer = "https://api.standardnotes.com";
      window.defaultFilesHost = "https://files.standardnotes.com";
      window.enabledUnfinishedFeatures = false;
      window.websocketUrl = "wss://sockets.standardnotes.com";
      window.purchaseUrl = "https://standardnotes.com/purchase";
      window.plansUrl = "https://standardnotes.com/plans";
      window.dashboardUrl = "https://standardnotes.com/dashboard";
    </script>

Improve milliseconds timestamps

This file https://github.com/mdouchement/standardfile/blob/master/pkg/libsf/time.go is no longer needed.
It can be replaced by:

keys cannot contain : character

I downloaded the binary file from the release page, and run with the command below

./standardfile-linux-amd64 init -c standardfile.yml

the content of standardfile.yml was copied from here

# Address to bind
address: "localhost:5000"
# Disable registration
no_registration: false
# Database folder path; empty value means current directory
database_path: ""
# Secret key used for JWT authenthentication (before 004 and 20200115)
secret_key: jwt-development
# Session used for authentication (since 004 and 20200115)
session:
  secret: paseto-development
  access_token_ttl: 1440h # 60 days expressed in Golang's time.Duration format
  refresh_token_ttl: 8760h # 1 year

but get the error message

Error: (2, 1): parsing error: keys cannot contain : character
Usage:
  standardfile init [flags]

Flags:
  -c, --config string   Configuration file
  -h, --help            help for init

2021/09/04 09:50:37 (2, 1): parsing error: keys cannot contain : character

some infomation that may be helpful

go version: 1.17 linux/amd64
platform: ubuntu 20.04
kernel: 5.11.0-27-generic

Unable to use with latest client sources

Hello !

Latest standardnotes client can't connect anymore to this wonderful server
Probably because of this: standardnotes/snjs@1bc19b7

New routes (/v2/***) to implement ?

Add docker image scheduled build for latest tags

https://github.com/mdouchement/standardfile/blob/master/.github/workflows/docker.yml#L4

Maybe using https://api.github.com/repos/mdouchement/standardfile/releases/latest

Question: standardfile protocol

https://standardfile.org/ seems down. May I know where can we find the protocol spec?

unix socket support

Looks like it is not possible to listen on unix socket

Error message:

Error: could not run server: listen tcp: address /var/snap/notes/current/notes.socket: missing port in address

Could you help?

Subscription does not work on android

I updated my install to the newest version (0.11.0), I adapted the config file to activate subscription features.

On the desktop standardnotes apps, it works, I have access to themes, different types of notes and so on.

On the android app, it does not work. I don't have access to any of the subscription features.

I tried disconnect, update the app, reinstall the app ... nothing seems to work.

have a nice day and thank you for this amazing piece of software.

1 PC client has subscription features enabled, 2 other clients only have basic features

The other 2 clients, only basic features allowed, are a Mac and an Android phone with the latest client. Is this by design?

invalid login credentials (on registration)

I am trying to test notes web v3.8.18 against latest sf and I am getting invalid login credentials as soon as I register (I do not even have a chance to enter login/password).
Request:

{
 "password":"***",
 "email":"***",
 "ephemeral":false,
 "identifier":"***",
 "pw_nonce":"***",
 "version":"004",
 "origination":
 "registration",
 "created":"1631372134116",
 "api":"20200115"}

Server says:

[401] POST /api/v1/users (326) 7.372964ms

Support incoming feature Session

https://github.com/standardnotes/syncing-server/blob/004-new/SESSIONS.md

Unable to login with the macOS app

I've just setup a docker compose file to test this out. It's working with the web app. I can create a user, log in, write a note and log out. When I log back in it is still there.

When I try to log using the macOS app I get "Unknown error". Any pointers please?

EDIT:

I forgot to add in my original post, thank you heaps for creating this! The SN stack seems very needlessly complicated ... case in point, this project!

Delete tag sync problem - sync_conflict and the web client will keep retrying

To repro

Add a new tag in browser A, then delete the tag.
login and sync from browser B
Go back to browser A, refresh
The web app will start to POST v1/items endlessly

Example request:

{"items":[{"content_type":"Tag","created_at_timestamp":0,"created_at":"2022-10-09T06:28:08.438Z","deleted":true,"updated_at_timestamp":0,"updated_at":"2022-10-09T20:36:46.083Z","uuid":"970fb9d9-abaf-4671-b81b-cfc7b5ed03e8"}],"sync_token":"MjoxNjY1MzQ4OTY3MjY5Njg3NTI3","limit":150,"api":"20200115"}

response:

{
    "retrieved_items": [],
    "saved_items": [],
    "conflicts": [
        {
            "server_item": {
                "uuid": "970fb9d9-abaf-4671-b81b-cfc7b5ed03e8",
                "created_at": "2022-10-09T06:28:08.438Z",
                "updated_at": "2022-10-09T20:36:50.337854034Z",
                "user_uuid": "fc44c92b-0b70-4e18-954b-dfdcc4b5323a",
                "items_key_id": "",
                "content": "",
                "content_type": "Tag",
                "enc_item_key": "",
                "deleted": true
            },
            "type": "sync_conflict"
        }
    ],
    "sync_token": "MjoxNjY1MzQ4OTY3Mjk3MDk1MTA2",
    "cursor_token": ""
}

Looks like it is from

standardfile/internal/server/service/sync_20190520.go

Line 147 in e925f8a

Type: "sync_conflict",

, I am not familiar with the protocol, so not sure if this is by design. But basically this makes the web app sending requests crazily and waste a lot of resource on both client and server side.

Support POST /v1/items/check-integrity

POST /v1/items/check-integrity

{
    "integrityPayloads": [
        {
            "uuid": "a19606e7-5838-4c7f-9641-93ffdcacf3c2"
        }
    ]
}

https://github.com/standardnotes/standalone does not support yet this feature.

Setup without docker is broken

Tried to run server without docker (with nginx-unit go setup) and it's not working, I can't figure out why now

Got 401 on every register/login attempts with response

{"error":{"message":"Invalid login credentials.","tag":"invalid-auth"}}

Docker setup running perfect on same machine with same ingress nginx. Maybe im missing some envs or something?

Use Knoa to handle dynamic JSON

Some fields of StandardNotes' API uses dynamic JSON.
Golang is a typed language so it's not made to handle this easily.

https://github.com/ivancorrales/knoa

Note revisions

standardnotes/syncing-server#60

import database

I am migrating from https://github.com/tectiv3/standardfile which uses sqlite database, is it possible to import existing data?
It has the following structure:

CREATE TABLE IF NOT EXISTS "items" (
    "uuid" varchar(36) primary key NULL,
    "user_uuid" varchar(36) NOT NULL,
    "content" blob NOT NULL,
    "content_type" varchar(255) NOT NULL,
    "enc_item_key" varchar(255) NOT NULL,
    "auth_hash" varchar(255) NOT NULL,
    "deleted" integer(1) NOT NULL DEFAULT 0,
    "created_at" timestamp DEFAULT CURRENT_TIMESTAMP NOT NULL,
    "updated_at" timestamp DEFAULT CURRENT_TIMESTAMP);

CREATE TABLE IF NOT EXISTS "users" (
    "uuid" varchar(36) primary key NULL,
    "email" varchar(255) NOT NULL,
    "password" varchar(255) NOT NULL,
    "pw_func" varchar(255) NOT NULL DEFAULT "pbkdf2",
    "pw_alg" varchar(255) NOT NULL DEFAULT "sha512",
    "pw_cost" integer NOT NULL DEFAULT 5000,
    "pw_key_size" integer NOT NULL DEFAULT 512,
    "pw_nonce" varchar(255) NOT NULL,
    "pw_auth" varchar(255) NOT NULL,
    "pw_salt" varchar(255) NOT NULL,
    "created_at" timestamp DEFAULT CURRENT_TIMESTAMP NOT NULL,
    "updated_at" timestamp DEFAULT CURRENT_TIMESTAMP);

Registering via Standard Notes client (v3.5.11) fails with "No password cost provided"

Commenting this condition from the server code eliminates the error and results in a seemingly working account:

standardfile/internal/server/auth_handlers.go

Lines 75 to 77 in 22dfcd5

    
           if params.PasswordCost <= 0 { 
        
           	return c.JSON(http.StatusUnauthorized, sferror.New("No password cost provided.")) 
        
           }

Loads subscription's features from the configuration file

The PR #90 provides the initial support of subscription's features requested by #74.

The project mdouchement/standardfile does not intend to update this list of features. In order to ease the updates of the features by everyone using this project, let's move the JSON payload to the configuration file along an explanation on how to extract it from official projet.

The new implementation should read the JSON exported from the https://github.com/standardnotes API:

https://docs.standardnotes.com/self-hosting/subscriptions/

It will update the IDs inside the JSON to match the user's ones like it's done in #90.

The configuration option should change from enable_subscriptions: false to:

suscription_feature_set: |
  {
    "meta": {
      "auth": {
        "userUuid": "some-user-uuid-v4",
        "roles": [
          {
            {
              "uuid": "8047edbb-a10a-4ff8-8d53-c2cae600a8e8",
              "name": "PRO_USER"
            },
            {
              "uuid": "8802d6a3-b97c-4b25-968a-8fb21c65c3a1",
              "name": "BASIC_USER"
            }
          }
        ]
      }
    }
  },
  and so one...

Then the subscriptions controller should be activated only when suscription_feature_set != "".

Swagger

server not reachable

I'm running standardfile in docker on a synology and can't connect to the server.

my standardfile.yml

address: "localhost:5000"
no_registration: false
database_path: "/data/database/"
secret_key: ...
session:
  secret: ...
  access_token_ttl: 1440h # 60 days expressed in Golang's time.Duration format
  refresh_token_ttl: 8760h # 1 year

I'm running the container as user 1029:100 and both the yml and the database are available inside the container:

uid=1029 gid=100(users)
/ $ ls -l etc/standardfile/
-rwxr-xr-x    1 1029     users          536 Jun  3 14:42 standardfile.yml
/ $ ls -l data/
drwxr-xr-x    1 1029     users           30 Jun  2 19:52 database

The log seems fine as well:

Routes:
[...]
   GET /version
2022/06/08 09:51:04 Server listening on localhost:5000
   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.7.2
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on 127.0.0.1:5000

however, when sending any request to the port I published (8084:5000) , e.g. a GET request to {serverip}:8084/version, I only receive a Error: Failure when receiving data from the peer without any logs.

Any idea what the issue might be?
Thanks a lot :)

Account creation issue

Hi there!

First up: love the project. Backend makes a super fast alternative to the official run (which runs very slowly on my own services when compared to this one). However, one small issue - on account creation, it immediately goes to an "expired session" forcing me to log out, and then back in again. Not a big deal by any means, but still something worth looking at in future :)

[libsf] could not unseal item Note: EncryptedItemKey: invalid secret format

Something seems to have changed in the official client that breaks libsf.

Drop old API format

Use:

/v1/*
/v2/*

New error format

{
    "meta": {
        "auth": {
            "userUuid": "452e3655-6f31-4bc6-8f99-ab24be85df3f",
            "roles": [
                {
                    "uuid": "8802d6a3-b97c-4b25-968a-8fb21c65c3a1",
                    "name": "CORE_USER"
                }
            ]
        },
        "server": {
            "filesServerUrl": "http://localhost:3125"
        }
    },
    "data": {
        "error": {
            "message": "The email you entered is already taken. Please try again."
        }
    }
}

meta can be ignored

Support workspaces

Unable to login with latest server and Android app

I'm using standardfile v0.7.1 and Android Standard Notes 3.26.14 (latest version on f-droid). I expected #77 to fix this, but I get a 401 error from the new v2/login-params API.

Jul 10 12:00:48 xxxxx standardfile[828]: [401] POST /v2/login-params (149) 1.110931ms

[CVE] Change JWT package

Use https://github.com/golang-jwt/jwt now due to the CVE on the old package GHSA-w73w-5m7g-f7qc

Echo project has already changed the package https://github.com/labstack/echo/blob/master/CHANGELOG.md#v450---2021-08-01

Subscriptions

The developers of StandardNotes have provided an official way to enable the 'Pro' subscription features on self-hosted instances:

https://docs.standardnotes.com/self-hosting/subscriptions/

Is it possible to build this into your server?

Admin console

https://github.com/containous/yaegi

Is your implementation still up-to-date with the latest StandardNotes changes?

Update standardfile/pkg/libsf to support new API version

Terminal UI client

Issue with session refresh

Refreshing the session
Error: could not refresh session: Invalid login credentials.

secret key not found

Hey,

I'm trying to install this on my local machine, but it refuses to start.

I first initialized the db with:
standardfile init -c /etc/standardfile.yaml

which worked. Then I tried to start the server with:

standardfile server -c /etc/standarfile.yaml.

Which gave me the following output:

Error: secret_key not found
Usage:
standardfile server [flags]

Flags:
-c, --config string Configuration file
-h, --help help for server

2022/07/26 13:28:14 secret_key not found
main.glob..func3
/home/runner/work/standardfile/standardfile/cmd/standardfile/main.go:126
github.com/spf13/cobra.(*Command).execute
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:990
github.com/spf13/cobra.(*Command).Execute
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:918
main.main
/home/runner/work/standardfile/standardfile/cmd/standardfile/main.go:51
runtime.main
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/proc.go:250
runtime.goexit
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571

What did I miss ? Sorry if it is a dumb question.

Add clarification on JWT and PASETO parameters in config file

Hi, I'm sorry to open an issue request about this because this isn't really an issue... it's just that it's unclear to me whether I should change any of the JWT and PASETO parameters on the standardfile.yml file before deploying the server.
I couldn't find a wiki for this project or any other documentation, so my apologies in advance if it's all explained somewhere else.

Thanks!

Continuous network requests with idle clients (since v0.6.6)

I tried a fresh v0.10.0 install today on a Raspberry. Things went well and the latest official macOS and iOS clients seem to work as expected.

But, while any of the clients are open, even when sitting idle, I noticed that the server is continously handling requests, many times a second:

[...]
[200] POST /v1/items (313) 2.28679ms
[200] POST /v1/items (313) 3.001393ms
[200] POST /v1/items (313) 2.505491ms
[200] POST /v1/items (313) 4.199917ms
[200] POST /v1/items (313) 2.634193ms
[200] POST /v1/items (313) 2.369752ms
[200] POST /v1/items (313) 2.479732ms
[200] POST /v1/items (313) 2.474639ms
[200] POST /v1/items (313) 2.477399ms
[200] POST /v1/items (313) 3.468239ms
[200] POST /v1/items (313) 3.779013ms
[200] POST /v1/items (313) 2.581231ms
[200] POST /v1/items (313) 2.841747ms
[200] POST /v1/items (313) 2.532047ms
[200] POST /v1/items (313) 2.870042ms
[200] POST /v1/items (313) 2.769191ms
[200] POST /v1/items (313) 2.668656ms
[200] POST /v1/items (313) 3.201558ms
[200] POST /v1/items (313) 2.557176ms
[200] POST /v1/items (313) 2.33653ms
[200] POST /v1/items (313) 5.08261ms
[200] POST /v1/items (313) 6.139988ms
[200] POST /v1/items (313) 2.498935ms
[200] POST /v1/items (313) 2.268641ms
[...]

This happens until the clients are closed. When the iOS client is switched to background, the requests keep going, but at a slower pace. When the server is stopped and restarted, it picks up right away when giving the macOS client focus. IOW this happens all the time while a client is connected.

The bandwidth consumed is a continuous 30-35KB/sec upload and twice that as download to the client, per client.

This effect is present back in v0.6.6, but doesn't happen in v0.6.4.

Diff between the two versions:
v0.6.4...v0.6.6

To rule out a client issue, I also tested this with an account on the official server and everything was normal in that case, with network activity only while editing notes, and no traffic while idle.

	if params.PasswordCost <= 0 {
	return c.JSON(http.StatusUnauthorized, sferror.New("No password cost provided."))
	}