Coder Social home page Coder Social logo

mayan's Introduction

mayan

A port of maya in NodeJS

CircleCI

Install

npm install -g @zenginehq/mayan

Using

$ mayan

mayan <command>

Commands:
  mayan build [plugin]    Build plugin
  mayan deploy [plugin]   Deploy plugin
  mayan watch [plugin]   Watch plugin(s) and deploy on changes
  mayan publish [plugin]  Publish plugin
  mayan register [plugin]  Register plugin in API
  mayan init [plugin]  Initialize plugin locally

Options:
  --env          Environment name
  --verbose      Display verbose debug output                          [boolean]
  --help         Show help                                             [boolean]
  --show-hidden  Show advanced options                                 [boolean]

For more information, RTFM at https://github.com/ZengineHQ/mayan

Key Features

  • Write frontend plugins using ES6 features such as arrow functions, let/const, destructuring, rest/spread, etc
  • Write frontend plugin styles using SCSS
  • You can also keep on using vanilla js and css if that's your thing
  • Support for frontend plugin modules directly in the main plugin package.json as regular dependencies (the old src dir approach also works)
  • Follows symlinks for @zenginehq modules for a great local development experience
  • Ability to --skip-build when deploying and publishing and --skip-deploy when publishing
  • Minifies CSS, JS and HTML when prod or production environment used
  • Supports maya-pre-build and maya-post-build package.json scripts for both backend services and frontend plugins
  • mayan watch [plugin] will watch your frontend plugin directories and deploy on changes for faster development process (recommended: use --frontend flag). Takes all the same commands as deploy.
  • coming soon register command to create/update plugins using the Zengine API and maintain your maya.json ids updated
  • coming soon init command to initialize a local dev environment either from a fresh git clone (registers plugin if necessary) or from scratch (programatically invokes yeoman generator and then registers and publishes plugin)

Directory Structure

/my-plugin-repo
├── backend/ --> backend service code
├── plugins --> frontend UI code
│   └── name-of-plugin --> can have several of these per repo
│       ├── node_modules
│       ├── src
│       │   ├── example-main.css
│       │   ├── example-main.html
│       │   └── example-main.controller.js
│       ├── package.json
│       ├── package-lock.json
│       └── plugin-register.js
├── maya_build/ --> build artifacts are kept here
├── maya.json --> config file for various environments/multiple plugins
├── maya.example.json --> save non-sensitive config info in version control
├── README.md
└── .gitignore --> include maya.json here because of sensitive config info

Running Backend Services Locally

You can expose your local backend services to the internet using mayan watch --proxy. Configure the proxy server by placing a proxy_settings key at the top level, the plugin level, and/or the backend service level. These settings inherit up the "chain," with the service-level overriding all else. Any option could feasibly be placed at any level, but obviously some options more naturally reside in certain locations. The only caveats to that would be:

  1. subdomain, authtoken, ngrokPort belong at the top level because these only get referenced once to set up the ngrok tunnel

  2. port belongs at the service level (leaving this out will default to 3000 and increment upwards to avoid using the same port for multiple services)

Once you run mayan against a backend service the --proxy flag, your generated ngrok url will be displayed in the terminal and automatically copied to your clipboard. You can then use this url in your test webhook configurations or send requests to it from your frontend plugin.

stdin commands

While your backend services are running, you can send these commands to mayan using stdin in the same terminal where mayan is running:

.exit will shut down all processes gracefully (same as ctrl+C)

[backend service name] typing the name of a service will trigger a reload of that service, similar to saving a file in that service's watched directories.

ex: to reload the calculate service, type "calculate" and hit enter

webhook-update [webhook ID] will use the access token provided in your current environment to fetch the url of the webhook you specified with the second argument, then update that webhook's url to point to your ngrok tunnel, thus routing all of that webhook's traffic to your local service. NB: this command requires the --proxy flag

ex:

$ mayan w --sd --proxy
# output from command indicating services are running
https://6e12515eb18c.ngrok.io --> copied to clipboard!
Inspect this connection in your browser at http://127.0.0.1:4040

Listening on http://:::3000/workspaces/:workspaceId/:pluginNamespace/:pluginRoute

Listening on http://:::3001/workspaces/:workspaceId/:pluginNamespace/:pluginRoute

# user input while service is running
webhook-update 123456
# output in response to user's command
Successfully updated webhook url to:
https://6e12515eb18c.ngrok.io/workspaces/123/nameSpace/endpoint?query=param

alias: wu

ex:

wu 123456
Successfully updated webhook url to:
https://6e12515eb18c.ngrok.io/workspaces/123/nameSpace/endpoint?query=param

Or, for scheduled webhooks:

$ mayan w --sd --proxy
# output from command indicating services are running
https://6e12515eb18c.ngrok.io --> copied to clipboard!
Inspect this connection in your browser at http://127.0.0.1:4040

Listening on http://:::3000/workspaces/:workspaceId/:pluginNamespace/:pluginRoute

Listening on http://:::3001/workspaces/:workspaceId/:pluginNamespace/:pluginRoute

# user input while service is running
scheduled-webhook-update 123456
# output in response to user's command
Successfully updated webhook url to:
https://6e12515eb18c.ngrok.io/workspaces/123/nameSpace/endpoint?query=param

alias: swu

ex:

swu 123456
Successfully updated webhook url to:
https://6e12515eb18c.ngrok.io/workspaces/123/nameSpace/endpoint?query=param

maya.json format

{
  "environments": {
    "dev": {
      "api_endpoint": "api.wizehive-dev.com -> you probably don't need this field",
      "access_token": "your token here -- keep out of source control",
      "plugins": {
        "name-of-directory": { // assumes a frontend code repository at ./plugins/name-of-directory
          "id": 123,
          "namespace": "my-cool-plugin",
          "route": "/my-cool-plugin", // deprecated legacy property (invalid in version 2+)
          "version": 2, // either a new or migrated plugin (leave property off for deprecated legacy process)
          "services": {
            "my-cool-service": {
              "id": 321,
              "route": "/my-cool-route",
              "proxy_settings": { // service-level proxy settings
                "x-zengine-webhook-key": "reallylongstringofcharacters",
                "port": 3008 // hardcoded specific port, used whether you're running --proxy or not
              }
            }
          },
          "proxy_settings": { // plugin-level proxy settings
            "x-firebase-url": "https://some-url-im-not-comfortable-unveiling.firebaseio.com/",
            "x-firebase-secret": "correspondingsecretforfirebase"
          }
        }
      }
      "default": true // if no --env (-e) is provided, this environment is assumed
    },
    "prod": {
      "api_endpoint": "api.zenginehq.com -> default value, you probably don't need this",
      "access_token": "your token here -- keep out of source control",
      "plugins": {
        "name-of-directory": {
          "id": 456,
          "namespace": "my-cool-prod-plugin",
          "route": "/my-cool-prod-plugin",
          "version": 2,
          "service": {
            "id": 789
          }
        }
      }
    }
  },
  "proxy_settings": { // top-level proxy settings
    "subdomain": "", // If you have a paid ngrok account
    "authtoken": "", // If you have a paid ngrok account
    "ngrokPort": 0 // 0 will be ignored and default to 5050
  }
}

With the advent of Zengine Plugins 2.0, any new or migrated plugins should specify version 2 in the frontend configuration, so mayan knows which build process to use.

Docker

Running mayan from a Docker image allows for a common plugin build environment. Instead of using npm to globally install mayan, clone this repo, then build and run the Docker container using the commands below.

Build

docker build -t mayan .

Usage

The included mayan.sh script will run the mayan container with the necessary parameters.

/path/to/mayan/mayan.sh build

You can make a symlink to the mayan.sh script to make it easier to access from your plugin directories.

ln -s $(pwd)/mayan.sh /usr/local/bin/mayan.sh

MacOS SSH Config

MacOS may include the line UseKeychain yes in ~/.ssh/config, which is not a supported config in the container. To prevent this error, add the line IgnoreUnknown UseKeychain above it.

Contributing

Fork

git clone [email protected]:ZengineHQ/mayan.git

Install dependencies

npm install

Test and lint

npm test
npm run eslint

Install binary locally

npm link

Adding new commands

Follow this approach/strategy to organize commands

Releasing

This uses release-it to do releases. It will handle creating the release tag, updating the README and creating the github and npm releases.

To do a dry run:

npx release-it -d

To do the real thing:

npx release-it

mayan's People

Contributors

tehpsalmist avatar evert0n avatar wizehive-chrismanjoine avatar roberttmoon-wizehive avatar dmcgrathwizehive avatar tedkulp avatar dependabot[bot] avatar wdmny avatar aparks avatar joebello avatar

Stargazers

 avatar Alex Weber avatar  avatar

Watchers

 avatar Anthony Putignano avatar  avatar James Cloos avatar David McNelis avatar  avatar Juan Scarton avatar  avatar  avatar  avatar  avatar

mayan's Issues

mayan should check for updates and prompt the user to upgrade

it's a common practice amongst node cli apps to check the latest available version on startup and if they are out of date prompt the user to upgrade.

we can decide whether or not we want this to be mandatory, ie: don't allow the user to run an older version if a newer one is available

Backup command

A couple of times now, I had to do a backup of the current deployed plugin (to be able to rollback if needed)
Would be helpful if mayan could do that backup for me in a single command line

Most time I want to do it is when a plugin is deployed to production but hasn't been previous managed by Git or maya/mayan

Can't build invalid html

I ran into an issue trying to build the html for bulk email and got the following output:

mayan build --frontend

Building frontend bulk-email/788
 ✔ Running pre-build scripts
 ✔ Installing node_modules
 ✔ Installing dependencies
 ✔ Building Javascript files
 ✔ Building SCSS/CSS files
 ✖ Building HTML files
   → </script>
   Running post-build scripts
   Cleaning up
mayan build [plugin]

Build plugin

Positionals:
  plugin  Plugin name                                    [string] [default: "*"]

Options:
  --env          Environment name
  --help         Show help                                             [boolean]
  --show-hidden  Show advanced options                                 [boolean]
  --frontend     Only build frontend plugins          [boolean] [default: false]
  --backend      Only build backend services          [boolean] [default: false]

TypeError: Cannot convert object to primitive value
    at PluginError._messageDetails (/Users/parksa/Documents/zengine/plugins/mayan/node_modules/gulp-htmlmin/node_modules/plugin-error/index.js:102:24)
    at PluginError._messageWithDetails (/Users/parksa/Documents/zengine/plugins/mayan/node_modules/gulp-htmlmin/node_modules/plugin-error/index.js:81:22)
    at PluginError.toString (/Users/parksa/Documents/zengine/plugins/mayan/node_modules/gulp-htmlmin/node_modules/plugin-error/index.js:133:14)
    at new Error (<anonymous>)
    at promise.then.catch.err (/Users/parksa/Documents/zengine/plugins/mayan/lib/util.js:78:13)
    at <anonymous>

This is because of the minification step; the gulp plugin uses this minifier, which as they say in the README:

HTMLMinifier can't work with invalid or partial chunks of markup.

I was able to modify the mayan source locally to skip the minification process so it would generate the plugin.html file, and got lucky and found the invalid closing tag at the very end of the file. However, this isn't always the case, so I could imagine it normally being pretty hard to debug where the issue is without more verbose output. The same complaint is brought up here, and the suggestion is to use the HTML linter that used to be part of the minifier.

config.json File

During the build process, if there is an existing config.json file, but no config-env.json file, mayan should default to the config.json file. Right now, if there is no config-env.json file, no config.json file will be included in the build.

Backend service deployment issues when using mayan to deploy wigeon-plugins

This has came out when checking why a stage-branch-based custom version of the SP plugin was having issues with the generation of the user consent report on a stage workspace while investigating and incident reported by a customer using the production version of the SP plugin. On the production side there was a bug on that process, on the custom plugin there was a deployment issue caused for little differences between Maya and Mayan.

This generation process uses the proxy backend service to make an Ajax request to wigeon.

The returned error when trying to use the backend service was:

{
    "status": 500,
    "error": "System error processing plugin service"
}

when looking into the published code of the backend service to track the issue I noticed that there was a config.json file with the right settings but into the plugin's code it was referenced as config-local.json

double checked with the old Maya and confirmed that was the regular behavior when copying config files (set the name to config-local.json during deployments)

 echo dist/config-local.json config-local.json | xargs -n 1 cp config-$MAYA_ENV.json

When tried to make a publish using Maya the new reports backend service (added to support progress reports feature) failed to build. As any new plugin, Mayan deployed and published this new backend service without any issue.

Investigated why this issue hasn't came out on production releases and found that David is still using Maya and not Mayan for that task.

the proxy backend service was originally used only by Review Portal plugin. When created the user consent report for GDPR compliance it was used to allow all plugins (SP,AF,WF,RP) to query Wigeon to generate that report. For this reason this backend service was added to all plugins.

This is not a big issue since we still have Maya. But if we plan to keep mayan evolving and use it with wigeon-plugins we need to address those small differences at some point.

option aliases

I personally like to type as few characters as possible, if I have to type at all, so cutting down mayan publish --yes to mayan p -y would be great. Also, --frontend to -f, etc. I think it's fairly simple with yargs, but I've never done it. Also interested to know if anyone else has strong opinions on this before I look into it.

allow bypassing "are you sure" with -y

this would maintain compatibility with the original maya plus its nice to have

context: when publishing mayan asks if you are sure, that's a good thing. let's add a shortcut to allow this to be skipped though

util.runSequential recursively create errors

Bug with util.runSequential create nested errors

Error: Error: Error: Error: Error: File not found with singular glob: /Users/everton/Projects/zn-plugin-mayan-test/plugins/dummy-frontend-plugin/plugin-register.js

Add check step for commands

Some of the messages are a bit hard to understand the issues it represents, for example if you have a wrong plugin path configured on maya.json and try to deploy it will error with this message

File not found with singular glob: /Users/everton/Projects/zn-plugin-mayan-test/plugins/dummy-frontend-plugin/plugin-register.js (if this was purposeful, use `allowEmpty` option)

Would be better to have a check step to error with a more straight forward error message

In this example saying that the frontend plugin doesn't exists

Cache node modules

Currently the longest and most annoying steps to wait for in the build process are installing node modules... everything else happens almost instantly...

That's expected because we're nuking the build dir and starting from scratch every time...

Ideally there would be some 3rd party tool to help but it could even be as simple as storing a checksum of the package.json "dependencies" key and if its the same re-use the existing built node modules, if available

pbcopy does not work on windows/linux

in mayan/lib/backend/watch.js line 86-90 we use pbcopy to copy the ngrok url to the clipboard

  const copy = require('child_process').spawn('pbcopy')
  copy.stdin.write(url)
  copy.stdin.end()
  console.log(green(`${italic(url)} --> copied to clipboard!`))

this is not available by default on windows.

some potential solutions/workarounds

could catch an error and point users to install:
https://ghuntley.com/blog/pbcopy-pbpaste-for-windows-released

can use windows 'clip' command
https://superuser.com/questions/472598/pbcopy-for-windows

Windows error with PORT command line variable

mayan/lib/backend/watch.js line 49:

executable: createScriptExecutable(`PORT=${service.port} node ${service.script}`)

this syntax for setting environment variables fails with this error on windows:

'PORT' is not recognized as an internal or external command,
operable program or batch file.

where the correct syntax would be

executable: createScriptExecutable(`SET PORT=${service.port} && node ${service.script}`)

could check the runtime and select a different command, or use node's environment variables to select the port.

investigate backend symlinks and related concerns/restrict watch to frontend?

Hey @alexweber , we talked about backend symlinks, and it's occuring to me that nodemon is actually handling them for us (sorry I don't have a specific link, it was a conclusion I drew from a bunch of random SO posts and github issues, as well as my recollection that I relied on this a couple months ago without even thinking about it to make a PR to our backend helper modules). That means no changes are necessary in the backend, and it also means a backend watch command really won't be as good as what nodemon can already do for us.

I am thinking out loud here, but maybe it would be more "user friendly" to just know that the watch command is a frontend aid, and the backend services are best run with nodemon. Perhaps we could/should remove the --frontend flag (because it would be assumed/implied always) for simplicity (watch --backend doesn't work at the moment any way)?

nvmrc version

Version should be the latest stable version - 8.10.0. The current nvmrc version does not support features that are used by the app - specifically fs.readFileSync.

UI overhaul

Hey @AParks I realize this one is a bit of a stretch but humor me for a bit...

So we're using the excellent Listr module for the frontend and backend build process right. Along with the functionality that you've already seen in mayan, Listr actually allows you to define sub-tasks which themselves can be Listr objects and also report status for these individual tasks.

Therefore, I think the utopia setup here would be to use Listr for everything, not just building the plugins (notice how the deploy/publish steps are different).

We could then do something like have a main task tree and each task could have its own sub-tasks:

  • Build plugins
    -- css
    -- js
    -- html
    -- etc
  • Deploy plugins
  • Publish plugins
  • etc

Then the entire interface would be one giant set of Listr tasks reporting progress etc and be really nice and consistent... :)

maya-pre-build on windows

"maya-pre-build": "cp config-$MAYA_ENV.json config.json" on windows does not work $MAYA_ENV is not available

mayan --show-hidden non-zero exit code

Bravo:test-plugin everton$ mayan --show-hidden ; echo $?
mayan <command>

Commands:
  mayan build [plugin]     Build plugin
  mayan deploy [plugin]    Deploy plugin
  mayan init [plugin]      Initialize plugin
  mayan publish [plugin]   Publish plugin
  mayan register [plugin]  Register plugin in Zengine API

Options:
  --version      Show version number                                   [boolean]
  --env          Environment name
  --config       Path to JSON config file
  --help         Show help                                             [boolean]
  --show-hidden  Show advanced options                                 [boolean]

For more information, RTFM at https://github.com/ZengineHQ/mayan

1
Bravo:test-plugin everton$

Should exit with 0

Register Command

When you first create a plugin, there are still some manually things that you need to do that could easily be streamlined through the command line, instead of logging into Zengine and doing through the developer ui.

Register command to create a plugin + backend service in zengine and populate the newly created ids in the maya.json file

Can port this:
https://gist.github.com/alexweber/68b59b1d3c971b81a61de477065cd6eb

Improve error handling for deploy/publish commands

Error handling for deploy/publish isn't finished currently it's just throw and exits

Example when an invalid token is used:

Error: Error: Error: StatusCodeError: 401 - {"error":"invalid_grant","error_description":"The access token provided has expired"}
    at promise.then.catch.err (/Users/everton/Projects/mayan/lib/util.js:78:13)
    at <anonymous>

Working on this one

symlinks are broken

I'm using 1.5.0, and symlinking to a local frontend-config, but it is not copying the directory over. I threw a log into the buildModules.js to prove that it is correctly detecting the symlinked package, but for some reason the copy is not taking place.

@alexweber I know we were looking at changing some things up with the copyDir functions, and I didn't want to reinvent the wheel (also I couldn't remember how you were doing it), so I'm opening up this issue while I try to refactor this aspect of the application.

Init Command

As a plugin developer, I want a single command that does all the initial legwork required when developing a brand new plugin from scratch.

Uses the plugin generator to generator some scaffolding code.
Creates a git repo for the plugin, makes the first commit (as Mayan author)
Runs the register command (which creates the zengine plugin + service, and populates the ids in the maya.json file)

Watch Command

@alexweber mentioned that this topic has been discussed elsewhere, but I couldn't find those conversations, so here's a new issue.

I have a script that I have been using to watch my frontend files and automatically deploy on save. It works great for my purposes.

This is it.

Alex and I also discussed the issue of rate of save/multiple concurrent file saves, which might warrant some sort of debounce effect on the save event, to avoid colliding deploy commands or hitting the API above the rate-limit. I think the more critical issue there would be to prevent the deploy command from executing while another deploy command is in action. As long as only one deploy executes at a time, that should avoid nasty errors.

I have really enjoyed working with the angular-cli (an incredible tool that has improved vastly in the last few months), so I am going to investigate their source code to see how they handle debouncing/throttling, and maybe get some ideas from that.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.