Welcome to Tradle serverless / Tradle MyCloud! You'll find everything you need to configure and launch your own Tradle instance here.
If you're developer, you'll also see how to set up your local environment, deploy, and develop your own chatbots.
- Orientation
- Setup
- Local Playground
- Development
- Destroy
- Troubleshooting local deployment
- Troubleshooting remote deployment
- Scripts
- Project Architecture
Jump down the rabbit hole
First, install some tools
Make sure you have git installed. If you're on OS X, you already have it.
To enforce https access to github run this command:
git config --global url."https://".insteadOf git://
The following are the versions used by the Tradle dev team:
- Node.js@14.17.5 - this is the version used by Amazon for AWS Lambda. Yes, you can use the latest Node.js instead, but keep this in mind if you hit errors.
- [email protected] - comes with 14.17.5
Docker is used during the build process, as well as in the local playground. Docker Compose is used for container orchestration and networking
- Docker
a. OS X
b. Window
c. Linux - Docker Compose
Make sure you can run docker as non-root. On Linux, you can do this by adding your user to the docker group with: sudo gpasswd -a $USER docker
- Install
- create a new IAM user with AdministratorAccess
- Configure your credentials:
aws configureoraws configure --profile <profileName>. This will set up your AWS credentials in~/.aws/
jq: a great command line JSON parser (On OS X, you can brew install jq)
typescript: This project uses TypeScript, which needs to be compiled to JavaScript prior to use.
Install: npm i -g --save-exact [email protected]
Note: Depending on your local setup you may need install with sudo
Note: if you don't care about playing locally and want to skip ahead to deploying Tradle MyCloud to the cloud, skip this section
- awslocal. aws-cli wrapper for querying localstack. (On OS X, install with
[sudo] pip install awscli-local)
Note: with npm i command that follows below the serverless and nodemon commands are installed relatively. You can use npx serverless or
npx nodemon to execute them respectively.
Clone this project. The rest of setup below takes place in the cloned repository's root folder.
# install dependencies
npm installBy default, aws cli operations will run under the profile named default
If you ran aws configure --profile <profileName> and not aws configure, open vars.json and add a property:
{
...
"profile": "<profileName>"
...
}Note: if you don't care about playing locally and want to skip ahead to deploying Tradle MyCloud to the cloud, skip this section
Goal: set up an environment where we can talk to the chatbot that comes in the box, and see how we can develop our own.
Check if the environment variable TMPDIR is set, and if not set it (better add it to ~/.bash_profile or ~/.bashrc)
# make sure you have docker running
docker psThe first time you start the playground, Docker will pull the necessary images from Docker Hub, which can take a while, depending on which century your internet connection is from.
npm startNow open your browser to http://localhost:55555. If 55555 is already your favorite port for something else, you can change the port in ./docker/docker-compose-localstack.yml.
If you don't see your local provider, click the red menu button on the Conversations screen, choose "Add Server Url" and add http://localhost:21012
| Profile | Conversations | Chat |
|---|---|---|
 |
 |
 |
After you chat with the bot a bit, open up GraphiQL at http://localhost:21012 and play with the API:
# http://localhost:21012
#
# sample query:
{
rl_tradle_ProductRequest {
edges {
node {
_author,
_time,
_link,
requestFor
}
}
}
}You can also browse the database via the DynamoDB Admin at http://localhost:8001
When you deploy to the cloud, GraphiQL will be available at https://xxxxxxx.execute-api.us-east-1.amazonaws.com/dev/graphql
The endpoints for localstack are enumerated in their docs (or see ./src/test/localstack.json). To query them using the AWS cli, specify an additional --endpoint option, e.g.:
aws dynamodb list-tables --endpoint http://localhost:4569
aws s3 ls --endpoint http://localhost:4572- To change the region/name/domain/logo of your deployment, edit
./vars.json. Then runnpm run build:yml. See./default-vars.jsonfor a list of variables you can override. - If you'd like to write your own bot, for now the easier way to do it is directly in your cloned tradle/serverless repo. Check out the built-in bot in: ./in-house-bot/index.js.
First, make sure Docker is running
# make sure docker is running
docker psAutopilot
# 1. compile typescript -> javascript
# 2. test
# 3. rebuild native modules with AWS Linux container
# 4. deploy to cloud
npm run deploy:safeManual
# compile typescript -> javascript
tsc
# gen resources in cloud emulator and test
npm run gen:localresources && npm test
# rebuild native modules with AWS Linux container
npm run rebuild:lambda
# deploy to cloud
npm run deployDeployment can take ~5-10 minutes.
Once everything's deployed, open your browser to https://app.tradle.io. On the Conversations page, click the red button, and choose Add Server URL. Paste in your API endpoint (it looks like https://xxxxxxx.execute-api.us-east-1.amazonaws.com/dev/)
See tradleconf, a command line tool for configuring styles, plugins, custom models, etc. of a deployed Tradle MyCloud.
npm run info # or run: sls info
# Service Information
# service: tradle
# stage: dev
# ...
# endpoints:
# ..
# ANY - https://example.execute-api.us-east-1.amazonaws.com/dev/tradle/graphql
# ANY - https://example.execute-api.us-east-1.amazonaws.com/dev/tradle/samples
# ..This project uses TypeScript, which compiles to JavaScript. If you're changing any *.ts files, or if you run git pull be sure you have tsc -w running on the command line, which will watch for changes and rebuild your sources.
You can start debugger mode by adding --inspect or --inspect-brk after the npm start script:
npm start --inspect-brkIf you modify serverless-uncompiled.yml, run npm run build:yml to preprocess it. Before running tests, re-run npm run gen:localresources
To override variables in the yml without picking a fight with git, create a vars.json file in the project root. See default-vars.json for which variables you can override.
After modifying vars.json, run npm run build:yml
Note: running tests messes with your locally emulated resources. After running tests, run npm run reset:local before running npm start
# run tests on local resources
npm run test
# browse that data via graphql
npm run test:graphqlserver
# GraphiQL is at http://localhost:21012
# DynamoDB Admin is at http://localhost:8001Thanks to serverless-offline, changes made to the codebase will be hot-reloaded, which makes development that much sweeter...but also slower. To disable hot-reloading, add this in vars.json:
serverless-offline:
# disable hot-reloading
skipCacheInvalidation: true
# copy these from default-vars.json unless you want custom ones
host: ...
port: ...Use tradleconf
Sometimes you want to wipe the slate clean and start from scratch (usually by age 25 or so). The following command will wipe out all the AWS resources created in your deployment. Obviously, use with EXTREME caution, as this command executes with your AWS credentials (best use a separate account).
To destroy your remote stack, resources, data, etc., use tradleconf
To destroy your local resources, use npm run nuke:local, or npm run reset:local to destroy + reinit
npm run nuke
# a series of y/n prompts ensues,
# ensuring you're committed to the destruction of all that is holyNote: this is ONLY for troubleshooting your local development environment and NOT your remote deployment
Symptom 1:
# Error: connect ECONNREFUSED 127.0.0.1:4569
# ...Cause: localstack is not up.
Fix: npm run localstack:start
Symptom 2:
# ResourceNotFoundException: Cannot do operations on a non-existent table
# ...Cause: you haven't generated local resources (tables, buckets, etc.)
Fix: run npm run gen:localresources
Symptom 3:
...bucket does not existCause: you probably ran tests, which f'd up your local resources
Fix: npm run reset:local
Symptom: tests fail, you don't know why
Cause: to be determined
Fix: npm run reset:local # delete + regen local resources (tables, buckets, etc.)
Symptom 4:
Serverless command "<some command>" not foundCause: your serverless.yml is corrupted. build:yml probably failed the last time you ran it.
Fix: fix serverless-uncompiled.yml, make sure build:yml completes successfully before retrying
Symptom 5:
still havent connected to local Iot broker!Cause: something in redis upsets mosca, but what exactly is TBD
Fix: npm run fix:redis
Symptom 6
The log is going nuts but the mobile/web client can't seem to communicate with your local MyCloud
Cause: if you have multiple clients connected at once (e.g. mobile, simulator, multiple browser tabs), your machine probably just can't handle it. If you've got Dev Tools open and are debugging your lambdas, that exacerbates things. This is due to the fact that locally, the serverless environment is simulated by invoking each lambda function as if it's waking up for the first time in a docker container. It needs to require() everything from scratch, then run itself, then die. This is memory/computation expensive.
Fix: turn off the debugger, don't use more clients than your machine can handle. Yes, locally, this might only be a 2-5!
Symptom 7
Credentials Error --------------------------------------
Missing credentials in config
Cause 1: your AWS cli is not configured with your credentials
Fix: see AWS cli
Cause 2: you may be using a global installation of serverless rather than the project-local one. If you're running Tradle locally via npm scripts, this should be taken care of for you. If you're running sls / serverless commands directly, make sure to use the project-local one in node_modules, e.g.: ./node_modules/.bin/sls offline start
Symptom 8
npm install fails with Authentication failed for 'https://github.com/tradle/models-corporate-onboarding.git/' (or some other private repository it fails to pull).
Cause 1: you don't have access to the repository
Fix: check to see if you can clone that repository directly, into some other folder. If you can't, request access from Tradle
Cause 2: your git credentials have expired, or are not being properly cached
Fix: set up caching for your git credentials (varies depending on your operating system), and then check to see if you can clone that repository directly, into some other folder.
Cause 3: npm is having trouble with dependencies with git:// urls.
Fix: open ~/.gitconfig on your machine, and add this block:
[url "https://"]
insteadOf = "git://"
Symptom 9
"namespace":"global:http","msg":"request failed","level":"ERROR","details":{"method":"POST","port":4569,"path":"/","host":"10.0.0.127"Cause: docker isn't running, or if it is, localstack isn't Fix: see fix for Symptom 1
Symptom 1
After deploying to AWS, CloudWatch logs shows:
module initialization error TypeError
Cause: a native module in your dependency tree was not compiled for the Amazon Linux Container
Fix: npm run rebuild:lambda and re-deploy
Keep in mind that deployment keys in S3 are based on the current git commit, so you'll need to re-commit before deploying, otherwise AWS CloudFormation will not re-deploy your lambdas with new code.
If the issue persists, you may have unknowingly introduced a new native dependency. Run ./src/scripts/list-native-modules.sh and see if there's anything missing in the native_modules var in ./src/scripts/rebuild-native.sh. If there, is, update native_modules and repeat the above fix.
Keep in mind that code bundle S3 keys are based on the current git commit hash, so you'll need to create a new git commit before pushing, e.g.: git commit --allow-empty -m "chore: bust deployment cache"
start DynamoDB and S3 in a Docker
stop local DynamoDB and S3
restart local DynamoDB and S3
update docker images
generate local DynamoDB tables and S3 buckets
generate local tables, buckets, identity and keys
delete local tables, buckets, identity and keys
delete and recreate local resources (tables, buckets, identity)
lint, run tests, rebuild native modules for the AWS Linux Container used by AWS Lambda, and deploy to AWS
start up two UIs for browsing local data:
- a DynamoDB Admin interface
- GraphiQL
starts up GraphiQL for querying remote data
- warm up all functions with:
sls warmup run - warm up a subset of functions with
sls warmup run -f [function1] -f [function2] -c [concurrency] - estimate cost of warm ups:
sls warmup cost
This project uses the Serverless framework. serverless.yml file is thus the main configuration file for the cloud architecture you'll be deploying: tables, buckets, IaM roles, lambda functions, logs, alarms, pictures of kittens, etc.
You can set up a local playground, with most of the functionality of the cloud one right on your machine. To make this possible, this project uses localstack for simulating DynamoDB and S3 locally, and serverless-offline + mosca for simulating AWS's APIGateway and IoT broker, respectively.
./
serverless-uncompiled.yml # npm run build:yml turns this into:
# -> serverless-interpolated.yml
# -> serverless-compiled.yml
# -> serverless.yml
vars.json # your provider's name/domain/logo, as well as dev env opts
src/ # typescript code, some shell scripts
*.ts
scripts/ # command line scripts, and utils
bot/ # bot engine
in-house-bot/ # currently co-located in-house-bot bot implementation
test/
lib/ # transpiled JS codeBelow you'll find the description of the various architecture components that get created when the stack is deployed.
you'll typically see table names formatted per a combination of the serverless and tradle convention, tdl-[service]-ltd-[stage]-[name] e.g. the events table is tdl-tradle-ltd-dev-events on the dev stage
events: immutable master logbucket-0: mutable data store for data, seals, sessions, etc.
ObjectsBucket: stores the payloads of all messages sent/received to/from users, as well as objects created by business logic, e.g. tradle.Application (to track application state)SecretsBucket: if I told you, I'd have to kill you. It stores the private keys for your MyCloud's identity.PrivateConfBucket: public/private configuration like: identity, styles, and bot plugin configuration filesFileUploadBucket: because Lambda and IoT message-size limits, any media embedded in objects sent by users is first uploaded hereLogsBucket: exactly what you thinkServerlessDeploymentBucket: stores past and current MyCloud deployment packages
Note: subject to change as lambdas are split out or collapsed together
jobScheduler: lambda that fans out scheduled tasks (e.g. warming up other lambda containers, retrying failed deliveries, sending pending transactions to the blockchain, polling the blockchain for confirmations, etc.)genericJobRunner: lambda that executes tasks fanned out byjobSchedulerpreauth(HTTP): generates temporary credentials (STS) for new connections from users, attaches the IotClientRole to them, creates a new session in thepresencetable (stillunauthenticated). Generates a challenge to be signed (verified inauth) *auth(HTTP): verifies the challenge, marks the session as authenticated *oniotlifecycle(IoT): manages the user's Iot session, attempts to deliver queued up messages depending on the user's announced send/receive positioninbox(HTTP): receives batches of inbound messages (typically from other MyClouds)info(HTTP): gets the public information about this MyCloud - the identity, style, logo, country, currency, etc.bot_oninit: initializes the MyCloud node - generates an identity and keys, saves secrets and default configuration files to respective buckets. Should really be namedinitoroninit, but good luck getting AWS to rename something.onmessage: processes inbound messages, then hands off to synchronous business logiconresourcestream: replicates changes to immutable events table, hands off to asynchronous business logicgraphql: your bot's built-in graphql API that supports existing Tradle models and custom ones you add.cli: command line lambda used for various admin tasks
* Note: the purpose of authentication is to know whether to send the user queued up messages. Inbound messages don't require pre-authentication, as they are all signed and can be verified without the need for a session's context.
- client (Tradle mobile/web app) calls
/preauth(preauthlambda) and gets a temporary identity via AWS STS. It also gets a challenge to sign. - client calls
/authwith the signed challenge. At this point MyCloud deems it safe to send the client any queued up messages, and will start doing so. - client subscribes to AWS Iot topics restricted to its temporary identity's namespace. This allows it to receive messages, acks and errors from MyCloud. MyCloud receives these Iot lifecycle events (connect, disconnect, subscribe) in Lambda, and updates the client's session information (
iotlifecyclelambda). - the client and MyCloud can send each other messages via AWS Iot.
Note: you don't need this unless you change the templates in in-house-bot/templates/raw
To prerender templates (primarily to inline css), run npm run prerender:templates
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent