galoymoney / galoy

bitcoin banking infrastructure

Galoy

💡 Get help

Q&A or Mattermost 💬

TLDR

Galoy is an opinionated Bitcoin banking platform.

This repo represents the main API that brings all functionality together.

Responsible Disclosure

Found critical bugs/vulnerabilities? Please contact our team by email at [email protected] .

Get Started

Want to try it out and contribute? Check out the dev documentation to deploy locally with a docker-compose script.

If you have questions, you can join our Workspace

For an overview of all relevant repositories checkout awesome-galoy.

Galoy-Backend features

• GraphQL API:
  • Public API following industry best practices
  • Schema for end clients
    • Reference
    • Documentation for the Blink hosted instance
  • Schema for admin activities
    • Reference
• Authentication:
  • Dedicated REST API to obtain an authentication token by logging in with:
    • phone by receiving a code via SMS or Whatsapp using Twilio
    • email by receiving a code via email
  • OAuth is integrated to be used in the Dashboard (example: Blink Dashboard)
  • Account scoped API keys can be created in the Dashboard with custom permissions and expiry
• Internal ledger:
  • Records all account activity via double entry accounting
  • Supports several fiat currencies
  • CSV based export of all accounting data
• Contact list for frequent transaction partners
• Price
  • Sub-second price data polled from the largest exchanges to record USD value at settlement
  • Historical price data can be queried for display for different time frames
• Send / Receive BTC payments
  • External settlement via lightning or onchain
  • Automatic internal settlement when payer & payee are on the same galoy instance
  • Fees can be applied for sending / receiving for all settlement methods
  • Support for tipping via dedicated web-frontend
  • Include memos to payments
  • Advanced onchain functions via Bria
    • advanced fee estimation
    • automatic transaction batching
    • automated CPFP to avoid stuck payments
    • PSBT based remote signing
• Lightning Network
  • Support for clearnet and Tor
  • Support for invoices with and without a specified amount
  • Route probing to pre-display an accurate fee and mitigate attacks based on expensive routing
  • Static Channel Backup backup to S3 and Google Cloud storage
  • Automatic liquidity management
    • Balancing of channels between the internal nodes
    • Submarine swaps
• Custodial storage of all user assets
  • Limited funds stored in hot-wallet (keys kept on servers)
  • Threshold based rebalancing to cold-storage (keys stored on offline hardware devices)
• Security:
  • Velocity check based on user verification level
  • Spam protection for sharing memos
  • Configurable 2fa for payments (in progress)
  • DDoS prevention
    • via rate limiting infront of critical APIs
    • via geetest CAPTCHA
• Resilience
  • Databases (PostgreSQl, MongoDB and Redis) are run by default in high availability/resilience mode. If one pod/node goes down, there is an automatic failover on another pod/node.
• Production ready
  • Supports horizontal scaling and highly available deployments via k8s
  • Client side load balancing across multiple LND nodes
  • Out-of-the-box dashboards for KPIs deployed to Grafana showing metrics exported via Prometheus
  • Quick response times thanks to the pagination of large data sets
  • Returning error codes for full translation capability of the frontend
  • Instrumentation enabled for real-time insights into production runtime (opentelemetry / honeycomb)
• User onboarding (optional)
  • Gamification via user quiz that pays out sats
  • Map of in-network merchants
• Notifications
  • Mobile clients can receive notifications of balance changes in real-time
  • Optional notification of balance for active end users

Local Development Setup

Running the Galoy software locally can be done in a variety of ways, but this abbreviated section will focus on a single method for getting your environment ready to run the stack. For more information and options on running Galoy locally, see the development environment documentation.

(1) Choose a Supported Platform

Currently developing on MacOS and Linux is supported.

(2) Install Dependencies

Install dependencies on your chosen platform.

• 1) nix with flakes enabled
• 2) docker from Docker Desktop or Docker Engine
• 3a) direnv version >= 2.30 installed
• 3b) direnv hooked into your shell

For nix, we highly recommend using the Determinate Nix Installer.

For docker, the Docker Desktop version corresponding to your native architecture should be used.

For direnv, you can install it with your package manager of choice. However, if you're unsure which installation method to use or your package manager does not provide a compatible version, you can use nix itself (e.g. nix profile install nixpkgs#direnv).

We recommend using the upstream docs for hooking direnv into your shell, but here is an example on how to do it on a system where zsh is the default shell. In this example, the following is added to the end of ~/.zshrc.

if [ $(command -v direnv) ]; then
   eval "$(direnv hook zsh)"
fi

(3) Enter the Repository Directory

All commands need to be run from the nix environment. Since direnv is installed and hooked into your shell, you can cd into the repository and nix will bootstrap the environment for you using the flake.

Please note: you may notice a large download of dependencies when entering the repository for the first time.

(4) Running the Stack

We use buck2 to run the stack, run and build individual services and libraries, perform lints and tests, etc.

_Before continuing, you should stop any locally running services to avoid conflicting ports with the stack. Some of the services that will run include, but are not limited to the following: PostgreSQL, OpenTelemetry, MongoDB.

Check if you are ready to run the stack before continuing.

buck2 run dev:healthcheck

You may notice some checks related to resource limits. On macOS and in WSL2 in particular, we recommend significantly increasing the file descriptor limit for buck2 to work as intended (e.g. ulimit -n 10240). Please note: the new file descriptor limit may not persist to future sessions.

Once ready, we can build relevant services and run the entire stack locally.

buck2 run dev:up

Once Tilt starts, you can check on the status of all services by accessing the UI through the given port on your local host (e.g. http://localhost:10350/). Every service should eventually have a green checkmark next to them, which ensures that they are in "ready" states.

(5) Test account credentials

For account login on regtest, you can input any random mobile number and then use 000000 as the OTP.

(6) Troubleshooting in Tilt

If some services fail to start, you can restart them on the Tilt dashboard.

(7) Tearing Down the Stack

The following command will stop all running services and containers. It will also remove the containers and, consequentially, the data held in them.

buck2 run dev:down

Run integration tests with Galoy as a dependency?

Take a look at the Quickstart if you want to take it for a spin.

galoy's Issues

on_us notification not equal to normal tx?

being able to request a payment

from Mike:

Would there be a way for one Galoy wallet user to send another user an invoice in the app? Could a business send them an invoice and they would be notified in the app and then have the choice to pay it, save it, or delete it?

I think it could be something that sets the wallet apart so that people want to use it so they can receive payment requests. Not sure if longer term there might be a way to make it an industry standard that other wallets might adopt to be able to participate

Maybe first time they receive an invoice it comes with a warning and asks if they want to accept invoices from that user. Maybe make the invoice sender pay a small fee (maybe $1) for first invoice sent to an account (maybe even refund this fee if the invoice is accepted (or paid). Not sure how hard that would be in your end but just some ideas

Message for send to username should be shared

add language in settings. send language from the OS to send notification in the right language

replace our custom monitoring chart with standard kube-prod-runtime

I just discovered this project: kubeprod

It's basically deploying grafana/prometheus/alert-manager + all the things we need to do to replace some google cloud specific tools (elastic search, fluentd, etc).

so instead of re-eventing the wheel, we should definitively use kubeprod. it would also make it straightforward to deploy on other cloud provider.

populate change-cause when rolling out a new graphql version

$ kubectl rollout history deployment/graphql-server -n mainnet
deployment.apps/graphql-server 
REVISION  CHANGE-CAUSE
66        <none>
67        <none>
68        <none>
69        <none>
70        <none>
71        <none>
73        <none>
74        <none>
75        <none>
76        <none>
77        <none>

the fact there is no change-cause make it hard to roll out previous version easily.
the cause could be a circleci message with a date + commit or something like that.

int have upperbound limit too low for graphql?

"Variable "$value" got invalid value 3616672277; Expected type Int. Int cannot represent non 32-bit signed integer value: 3616672277"

https://console.cloud.google.com/logs/query;pinnedLogId=2020-11-13T16:26:17.438910790Z%2Fryxcfi2amkf239f68;query=resource.type%3D%22k8s_container%22%0Aresource.labels.cluster_name%3D%22testnet-cluster%22%0Aresource.labels.namespace_name%3D%22mainnet%22%0Aresource.labels.container_name%3D%22graphql-server%22%0AjsonPayload.level%3E%3D%2240%22;timeRange=PT3H?project=galoyapp

make server on "maintenance mode" if lnd / bitcoind is not up

users could still be able to see their balance and do on_us transaction even if our node is down

one way to do it would be to see if lnd-container is up using a kubectl call.

use public/private key pair instead of symmetrical key for JWT_Secret

we should be using a private/public key pair, so that the client can check the signature of the JWT token being generated. This may prevent an attack in the middle attack where a client could otherwise be connected to a forged backend (for some reason like a DNS attack).

add username to invoice (on #31)

Try merging lightning/onchain qrcode for receiving payment

See how this would react on other wallets

mongodb logs issue?

there a lots of "connection ended", "connection accepted" in the logs

and also slow query

using:

resource.type="k8s_container"
resource.labels.cluster_name="testnet-cluster"
resource.labels.namespace_name="mainnet"

highlight on jsonPayload.msg

cashback for business

do a more proper implementation, with max amount per business.

the previous implementation (now removed because a bit hacky) was integrated into Lightning.ts. This is where the logic were.

A better implementation is to run a cronjob dedicated to the cashback, such that it could do more check/analytics.

One way to relaunch the cashback program is to have a cap amount per business. And maybe also have a reduce velocity per day. For instance:

max cashback per business: - 50k sats
max daily cashback: - 5k sats
cashback disbursed daily, and only if payment is from non-business to business user.

make user id a random number

currently, it contains a timestamp. which gives an idea about when a user has been created. this is not great for privacy reasons.

eventually, uid will be shared/included in the invoice.

add on_us_receipt and on_us_payment types

Generate more insights into our payments

I have been watching a video no stackdriver yesterday and it makes me think we should change our approach for the transaction per day analysis.

If we use a log based approach, we could have additional KPI:

• success rate. instead of being able to just know how much successful payment has been made on a particular timeframe, we could compared it to failed payment
• total payment, not just succesful one.
• group payment error by type

we may need to provide a bit more log output from the app, to have clear key-pair values when a payment succeed and when a payment fails.

It might also remove the need to have a special plugin for mongodb that seems not trivial to install automatically.

Auth: need captcha?

there is currently no restriction on the register api. this could lead to some attack of mass account registration, but also, as current auth is done with text, mass account billing from twilio.

TODO: looks at Captcha solution and see if this is what makes sense, in the context of a mobile app.

improve documentation for graphql api for other developpers

https://www.getpostman.com/collections/312bf8bc55b9e93249fe

a lot of onchain_payment have pending: true despite being confirmed

db.getCollection('medici_transactions').find({"type": "onchain_payment", "pending": true}).count() --> 356
db.getCollection('medici_transactions').find({"type": "onchain_payment"}).count() --> 914

looking at the log, it seems this is because the trigger Pod didn't receive the notification from lnd.

similar to the cleanup function for Lightning when a payment has been sent (updateUsersPendingPayment), another cron job should run daily to make sure sent on-chain transaction with 6 blocks confirmation are marked pending = false.

missing fees/ revenues in the accounting books

are missing (at least):

• fee accounting for loop server
• miner fee from cold storage to hot wallet

add Thunderhub

Alex mentioned that Thunderhub could be better than RTL for rebalancing.

https://github.com/apotdevin/thunderhub

add standard exporter

• mongodb: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-mongodb-exporter
• redis: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-redis-exporter
• nginx: https://github.com/nginxinc/nginx-prometheus-exporter (not helm ?)
• graphql: https://github.com/slanatech/swagger-stats

timeout for onchain payment

yet the payment succeeded

 {
  "insertId": "13md9jagdtswdv9rr",
  "jsonPayload": {
    "topic": "payment",
    "token": {
      "uid": "5f1b76e7035c825d8615fb56",
      "network": "mainnet",
      "currency": "BTC",
      "iat": 1604522667
    },
    "onUs": false,
    "memo": "Cold Card Hardware Wallets ",
    "hash": "6ccbf12d775938568b5f466084b5674eae0b706f8898348d8c7e5201b035213b",
    "hostname": "graphql-server-596cd4d4bb-6587r",
    "balance": 27066259,
    "pending": true,
    "usd": 1161.73990311,
    "fee": 33197,
    "amount": 7383900,
    "pid": 29,
    "currency": "BTC",
    "id": "28f92404-cad6-451d-ac17-120ef934de2e",
    "module": "graphql",
    "body": {
      "query": "mutation onchain($address: String!, $amount: Int!, $memo: String) {\n        onchain {\n          pay(address: $address, amount: $amount, memo: $memo) {\n            success\n          }\n        }\n      }",
      "variables": {
        "memo": "Cold Card Hardware Wallets ",
        "amount": 7383900,
        "address": "1LZ2SnKPwCJZMT3XHh285NLSVLxFoBvWFd"
      }
    },
    "feeUsd": 5.19964611,
    "sats": 7417097,
    "type": "onchain_payment",
    "msg": "successfull onchain payment",
    "success": true,
    "transactionType": "payment",
    "level": 30,
    "address": "1LZ2SnKPwCJZMT3XHh285NLSVLxFoBvWFd",
    "protocol": "onchain"
  },
  "resource": {
    "type": "k8s_container",
    "labels": {
      "container_name": "graphql-server",
      "pod_name": "graphql-server-596cd4d4bb-6587r",
      "cluster_name": "testnet-cluster",
      "location": "us-central1-a",
      "namespace_name": "mainnet",
      "project_id": "galoyapp"
    }
  },
  "timestamp": "2020-11-07T02:28:32.277488241Z",
  "severity": "INFO",
  "labels": {
    "k8s-pod/app": "graphql-server",
    "k8s-pod/kube-monkey/enabled": "enabled",
    "k8s-pod/pod-template-hash": "596cd4d4bb",
    "k8s-pod/kube-monkey/identifier": "graphql-server"
  },
  "logName": "projects/galoyapp/logs/stdout",
  "receiveTimestamp": "2020-11-07T02:28:36.159189247Z"
}

look
1/ if there is a systematic issue with on-chain payment
2/ how graphql timeout are being logged (it seems no log at all?)

use pool for new channel opening

I have just thought about our flow of capital. Currently, it's "getting out", which is exactly what we would do if we were providing liquidity in pool.

So it will probably make sense that instead of opening a channel our own, that we would lease liquidity. the effect would be that we will be ending up having new channel with the liquidity on our side, but with the side effect that we would earn a yield on each new channel that we are opening.

make it possible to "send all" amount for onchain payment

currently, when making a payment, the fees are being added to the amount sent.

for instance, if someone want to send 10,000 sats, and the fees is 2,000, the total will be 12,000 sats.

the issue to do a send all is that, when the base amount increase, the fee might changes (fee are being calculated dynamically by lnd).

so for instance, if someone has 15000 sats. he would need to have maybe 13000 sats in the amount, and if the fees are 2000 sats then this would send all. but maybe for 13,000 sats the fees would be 2050 because one utxo is being used. and for 12950, only 2000 would be necessary because a single utxo would be needed.

so empirically people are doing a dichotomy to find the right amount to swipe their balance. I believe the right way to manage is to have a send_all flag where instead of the fees being added, it should be deduced from the total amount sent.

this would have an impact on the react native side. but first the API should be available for consumption to the mobile wallet.

page issue for nginx error

When nginx return 4xx or 5xx errors, no page is being sent

the payment shows a failed but actually succeeded

tried with Phoenix wallet + Galoy on the same android phone

Add mock test for gcs scb upload

issues with Twillio / Error 30008 Unknown error

websocket (for ATM integration, +lnpage?)

Eric from Athena is currently using our API with graphql and doing a poll every 2 mins.

This would be a much better experience by pushing the on_chain transaction directly to his server.
the cash would be expense directly instead of having to wait for 2 min to receive the notification that the payment has been received.

Right now it shows instantly on the wallet but the machine only acknowledges the payment later.

I think if we have a WebSocket for events, this may also be helpful for the lnpage payment receipt?

At least for the ATM, the payment we would need are the on_chain_on_us, and on_chain_payment (?)

explore lnurl payment/call back options

there is an example in the last lightning labs newsletter: https://lightninglabs.substack.com/p/lightning-pool-is-open-for-business

the data from the QRCode at the bottom is: LNURL1DP68GURN8GHJ7MRWWPSHJTNRDUHKG6TNW3EX7TTJDA6HGETJ9AKXUATJDSKHW6T5DPJ8YCTH8AKXU5RP09GXZUNPD4EN6ETEFFAXZ3EE09JYVWF3VDKHW620D9YNYN28F4M5U7NRD9XYXJNTV9VYUVRRD5UKVCNDGE69556FXEYK6MR5T9TKGMZFD9MKJCJ8X4G4JKRVGAV4S4N2TFV9Y42E2A3KJNMFFFX9VKRGDDYKJAMFV3VYUMRRDSUHWK2CD3NXY46KXPS5WWTTF94X76TZGU6NZCMDWA5KV5F9XDZZ2V6YYENXZATRV46976RPWD5R6ENRW30KGSJXVF5YX5R4XPS56KPJ23CHXVJ6X9MXU3TJXVS3AT2F

need to add spread for USD/BTC conversion

issue with lntxbot

join t.me/lightningwallet
you'll have an invoice to pay to join the channel.
try paying with the wallet

graphql-server says:

{"level":50,"time":1600293671054,"pid":28,"hostname":"graphql-server-7c869b85fc-hr629","err":{"type":"Error","message":"there is no payable route for this payment","stack":"Error: there is no payable route for this payment\n    at /functions/src/Lightning.ts:264:15"},"msg":"lightning payment error"}

I believe the same issue will be from anyone using lntxbot.

Integrate jaeger

notification push for onchain transaction not working?

price needs to refresh faster

From @hunterglobal :

I denominated the payment in USD value...
But when I clicked "Send" (roughly 3 seconds later), it appears the payment was fixed in sats -- leading to a slightly different USD value displayed
This may be perceived as a bit "weird" by everyday users -- just something to be aware of, if not already on your mind

This is because send USD amount is translated into sats with the price in memory in the wallet.
But the price in the backend might already be different. and when showing the receipt, the price in USD shown correspond to what is in memory in the backend, not in the wallet.

price in the wallet can have a lag with price in the backend.

dealerWallet have a floating sat balance

'BTC:Liabilities:Customer:5f8a1e6c2f410ed166a1b9ab': -378669.81700220774,

add monitoring for channel balance (and rebalance as necessary)

latest.apk link to an old version

I just downloaded the apk and unbundled it. And the version shows as 1.2.2 and not the last version 1.2.3

nginx 499 error

an example from the log

it seems this is because the client closes the connection. I don't know if this is something this could be optimized in the front end, or this is because the app was shutdown/send to background/crashed

a filter to see all the relevant logs

trigger memory leak?

https://console.cloud.google.com/kubernetes/deployment/us-central1-a/testnet-cluster/mainnet/trigger/overview?project=galoyapp

being able to regenerate auth token periodically

No notifications received for incoming payments or for daily balance

I am unable to receive notifications. Also tried paying myself from WOS and it's the same irrespective of whether the app is open or not. Will look at logs to see what's going on.

Separate gcs mongo backup into testnet and mainnet folders

specter certificate issue

(https://specter.mainnet.galoy.io/)

onchain transaction confirmed notification received twice

with 5 min interval, on testnet, when sending an onchain payment

payment failure with wallet that open channels on the fly

(note: this is a old ticket and many things has changed since we had this issue, so description might not be up to date)

I tried many wallets now that the getFee has been deployed to mainnet.

It seems Breez still doesn't work unless the channel has already been opened. (WOS does work)
Phoenix now works (it didn't the last time I tried), but there was already a pre-existing channel on my wallet. Might
make sense to try with a new Phoenix wallet to see where we stands.

backend to update the business location/map

we currently run a script to update the businesses on the bitcoin beach map. This script ingest a .csv.
we should have a way to update the business on the map in real-time directly by the people (we can call them "editor" for this issue) on-site with a dedicated backend.

I guess this could either expand on lnpage or have a different repo/url altogether because it would be a different audience (although there would be overlap, editors would also typically users).

Anyhow, I think we should use the same architecture as lnpage: frontend in react, backend using our graphql.

note: this would be the beginning of a more comprehensive backend to help the users (for lost credentials, if there are issues with some transactions, etc)

Macaroon/TLS are visible in the logs

Looking at the CircleCI logs (which would be available publicly once it's open-source) we can see the credentials. We need a way to hide it.

add ways to secure the account with 2fa

Add a google authenticator option to secure the account. This should be optional.
(But recommended above a certain amount, this part should probably be handled on the mobile side)

As we are a payment application for daily transaction (and not a brokerage app where people would store million of dollar), we don't necessary want to enforce 2fa for every payment. there should be a setting where 2fa is required if payment/velocity is above x $amount (ie : $100/24 hours).