Coder Social home page Coder Social logo

meteor-apm-server's Introduction

APM

This project reduces the original Kadira APM to a single Meteor project and includes template MUP configuration to let you deploy to any remote server.

Most of the original features are working (like Slack alerts), but there is still a lot of work. Feel free to contribute!

Get up and running with MUP

The easiest way to get this server up and running is to use the recommended configuration with MUP, see below.

Setup steps using Docker

You can also use docker to spin up an instance pretty quick:

docker run -d --name meteor-apm-server \
  -p 4000:80 \
  -p 7007:7007 \
  -p 11011:11011 \
  -e PORT=80 \
  -e MONGO_URL=mongodb://[your mongodb url] \
  -e MONGO_OPLOG_URL=mongodb://[your mongodb oplog url] \
  -e ROOT_URL=[e.g. monitoring.yourdomain.com] \
  strictlyskyler/meteor-apm-server:1.1.0

This can be useful for running an instance quickly in your own environment with orchestration.

Setup steps using MUP

Keep in mind when using MUP: your user account on the server must be sudoer without password

  1. Clone this repo and run meteor npm install.

  2. Copy mup-placeholder.js to mup.js. Replace the placeholder entries in the configuration with your configuration.

  3. Copy settings-placeholder.json to settings.json. Change any settings as it suits your project (see Meteor apm settings section below)

  4. Server configuration steps you need to verify prior to deployment:

    a) This setup was tested using a server with at least 512MB of RAM.

    b) Allow public access to your server on the following ports: 22, 80, 443, 7007, 11011.

    c) In order for SSL configuration to succeed, you must set setup your DNS to point to your server IP address prior to deployment. Make sure to point both apm.YOUR_DOMAIN.com and apm-engine.YOUR_DOMAIN.com to the same server IP address.

  5. Run npm run mup-deploy.

  6. Visit your APM UI page at https://apm.YOUR_DOMAIN.com. Login with username [email protected], password admin. CHANGE YOUR PASSWORD FROM THIS DEFAULT.

  7. In the APM web UI, create a new app and pass the settings to your Meteor app (you can copy paste from the UI):

settings.json

{
  ...
  "kadira": {
    "appId": "YOUR_APM_APP_ID",
    "appSecret": "YOUR_APM_APP_SECRET",
    "options": {
      "endpoint": "https://apm-engine.YOUR_DOMAIN.com"
    }
  },
}
  1. Re-deploy your Meteor app, and you should see data populating in your APM UI in seconds.

Server Restarts

The custom nginx proxy configuration does not persist through a server restart. There is no current way to hook this into MUP easily, so you will need to run npx mup deploy again if you need to restart the server. This should not be a problem with normal operation.

Meteor apm settings

metricsLifetime sets the maximum lifetime of the metrics. Old metrics are removed after each aggregation. The default value is 604800000 (1000 * 60 * 60 * 24 * 7 ^= 7 days).

As a baseline, a current Meteor application with ~500 DAL uses 0.7 GB for 7 days of APM data.

public.apmEngineUrl refers to the endpoint URL of your application (this allows to display good hints when creating the application)

Configuration details:

If you want to do custom configuration and server setup, here are items to be aware of:

  1. A mongo replica set is required. This is set up automatically for you when using the template MUP configuration script.

  2. If you are not getting APM data and see a No 'Access-Control-Allow-Origin' header is present console error in your Meteor app, this is due to incorrect nginx proxy configuration. To confirm the issue, ssh into your server (npx mup ssh) and run docker exec mup-nginx-proxy cat /etc/nginx/conf.d/default.conf. Look for the upstream block for apm-engine.YOUR_DOMAIN.com, the entry should look like

upstream apm-engine.YOUR_DOMAIN.com {
    # YOUR_APP
    server SOME_IP:11011;
}

If you see port 80 for that setting, the MUP hook that tries to update this port may be failing.

Changes to original Kadira

  • Reduce to one project
  • Added MongoDB indexes
  • Removed MongoDB shards
  • Remove raw data after processed
  • Use Meteor 1.6 (Node v8)
  • Removed premium packages

ToDo

  • Direct db access of alertsman (apm/server/alertsman/server.js) and remove api (apm/server/api/server.js)
  • Replace invalid links to old kadira docs

meteor-apm-server's People

Contributors

ffxsam avatar jasongrishkoff avatar jehartzog avatar koenlav avatar lmachens avatar markreid avatar msteinmn avatar pcorey avatar ramezrafla avatar rohit2b avatar strictlyskyler avatar vparpoil avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

meteor-apm-server's Issues

Connection successful but nothing displayed

I deployed the APM on a fresh EC2 instance and everything went through without any issues.
I can access the UI and log in, but after I re-deployed my app with the credentials from the UI, I get

"Kadira: Successfully connected" but nothing else.

The UI is still displaying the credentials and not loading the data from the app.

Any idea?

Specify some minimum system requirements in Readme

Although I'm sure scaling will depend on usage, it'd be great to have some kind of minimum requirements specified in the Readme for deploying this.

I'm planning on deploying this myself soon and may have some feedback on this but would like to get the conversation started and see if anyone else has experience running a minimal deploy.

Looking for maintainers

Hi, I am looking for maintainers for this project, because I don't use Meteor in my projects anymore.
If you want to become a maintainer, please answer to this thread.
Thanks!

Kadira Successfully connected but no data

I don't get any data on apm server after connecting my application to the server.
The apm application page still display settings instructions even after I get the message "Kadira: Successfully connected" and "Kadira: completed instrumenting the app" with the keyid and keysecret.
I don't have a firewall blocking connection.

Are you looking for maintainer?

Hello @lmachens
Long time!
Sorry for silence, got distracted with work and life.

What is the status of this amazing project? I recall having challenges editing the code, Kadira's code tends to be high on the spaghetti scale ;)

Let me know what else needs to be done? Are there any other community developments you are aware of we can use as inspiration?

Thanks

UI crashes after a while

After sometime of meteor-apm-server deployment, the UI crashes with 401 error in frontend while engine keeps on collecting the data.

Logs shows something like below:

blocked due missing appId:  /
blocked due missing appId:  /
blocked due missing appId:  /
blocked due missing appId:  /
blocked due missing appId:  /
blocked due missing appId:  /
blocked due missing appId:  /
blocked due missing appId:  /kadira-logo-light.png
blocked due missing appId:  /kadira-logo-light.png
blocked due missing appId:  /kadira-logo-light.png
blocked due missing appId:  /
blocked due missing appId:  /

I tried accessing static resource as well, but same issue even those could not be loaded.

If anybody here, have insight into this, please help.

Documenting issues

@lmachens

Thanks so much for this great effort! We deployed and it's working great.

We want to contribute to improving the APM. So I just listing the issues we faced so that we can start a conversation on what can be done, and what we can do to help

  • We should remove all mention of Kadira anywhere, it confuses users (and the original author had indicated they want their branding removed). This relates to icons (which means design) and to mention of Kadira (I see it namely in the login page)
  • Remove sign up option at the bottom of login page
  • Allow customization of title
  • We faced 2 minor hickups when pushing with MUP: we had to make sure sudo user has passwordless sudo access, and a bug during docker deployment with failed permissions (had to add user to docker group). We should document this in the README steps
  • In production, many (like us) have multiple meteor instances on the same host, so we need to mention the 'hostname' feature in Kadira.connect

And .. we should market it on the Meteor forums as working out of the box. I can add a post there so people know it's available and how it helped us

(more to come)

Excessive disk space usage

Overall everything is running smoothly now, but the last issue I'm seeing is a constant disk space usage that I can't explain yet.

The cleanup.js seems to be removing old data, based on looking at the site, but actual disk space usage continuously rises at something like 500MB per day.

I did a little digging and I was able to find a few things but nothing yet which confirms why:

  • The sizes of all the docker images docker ps -s are all reasonable, under 700 MB.
  • The largest folders on the filesystem are /var/lib/docker/ at 3.7GB and /var/lib/mongodb/ at 1.1 GB.

I think the size of mongodb is expected, but the docker folder seems to be the one that is growing.

I've already removed the console.log commands, and I've verified using mup logs that there isn't excessive logging going on, so I'm not sure what is eating up all the disk space here. I'll follow up if I figure out what's going on.

App collaboration feature doesn't work because users can't register

I was taking a look at this over the weekend, I thought I should open up an issue and ask for opinions before diving too far into a solution...

Right now:

  • Share feature sends an email to the designated email, telling them they've been invited to collaborate on an app.
  • User needs to login and can then accept the invitation, at which point they're given access to the app.
  • User registration is closed; apart from [email protected] there's no way to add new users to the system.

I propose:

  • Share feature sends the notification email
  • User is given the option to either login with an existing account or register a new one with email/password.
  • We use the invitation token (the id field from db.pendingUsers, where collaboration invites are stored) to allow them to register a single user.

Thoughts? I've started tinkering and it won't take much to implement, I could probably get a PR up pretty shortly...

Email alerts are overly active

After setting up the MAIL_URL and trying to setup an alert I kept getting the same alert multiple times (60+ emails), even when setting it to be sent once.

Less Import Error During Deploy

Firstly, thanks very much for your work on this project!

When running ‘npm run mup-deploy’ the following error was appearing during the app build/bundling part of the process:

Building App Bundle Locally
local:jobs: updating npm dependencies -- aws-sdk...
local:kadira-data: updating npm dependencies -- lru-cache...
Errors prevented bundling:
While processing files with less (for target web.browser):
client/0config/bootstrap/custom.bootstrap.import.less:6: Unknown import:
custom.bootstrap.mixins.import.less

I managed to fix the problem by editing the "/client/0config/bootstrap/custom.bootstrap.import.less" file and change this line:

@import "custom.bootstrap.mixins.import.less";

to

@import "{}/client/0config/bootstrap/custom.bootstrap.mixins.import.less";

Nemo64/meteor-bootstrap#64 pointed me in the right direction for the fix.

Please fix

How amazingly awesome this is. Thank you so much for doing this!

Unable to use agent package with meteor-apm-server

I have setup server via docker image

I tried connecting my app after running meteor add meteorhacks:kadira it added two packages to my project

meteorhacks:kadira   added, version 2.30.4    
meteorhacks:meteorx  added, version 1.4.1

Then my application doesn't want to start, is there is a specific meteor-apm-server agent I could use instead?

=> Started proxy.                             
W20200428-12:32:07.569(-4)? (STDERR) /Users/dean/.meteor/packages/meteor-tool/.1.10.2.szrfc1.fymg++os.osx.x86_64+web.browser+web.browser.legacy+web.cordova/mt-os.osx.x86_64/dev_bundle/server-lib/node_modules/fibers/future.js:280
W20200428-12:32:07.605(-4)? (STDERR)                                            throw(ex);
W20200428-12:32:07.605(-4)? (STDERR)                                            ^
W20200428-12:32:07.606(-4)? (STDERR) 
W20200428-12:32:07.606(-4)? (STDERR) TypeError: Cannot read property 'constructor' of undefined
W20200428-12:32:07.606(-4)? (STDERR)     at exposeSubscription (packages/meteorhacks_meteorx.js:61:41)
W20200428-12:32:07.606(-4)? (STDERR)     at exposeLivedata (packages/meteorhacks_meteorx.js:31:5)
W20200428-12:32:07.606(-4)? (STDERR)     at packages/meteorhacks_meteorx.js:175:1
W20200428-12:32:07.606(-4)? (STDERR)     at packages/meteorhacks_meteorx.js:200:4
W20200428-12:32:07.607(-4)? (STDERR)     at packages/meteorhacks_meteorx.js:208:3
W20200428-12:32:07.607(-4)? (STDERR)     at /Users/dean/Projects/swiftdental/.meteor/local/build/programs/server/boot.js:401:38
W20200428-12:32:07.607(-4)? (STDERR)     at Array.forEach (<anonymous>)
W20200428-12:32:07.608(-4)? (STDERR)     at /Users/dean/Projects/swiftdental/.meteor/local/build/programs/server/boot.js:226:21
W20200428-12:32:07.608(-4)? (STDERR)     at /Users/dean/Projects/swiftdental/.meteor/local/build/programs/server/boot.js:464:7
W20200428-12:32:07.608(-4)? (STDERR)     at Function.run (/Users/dean/Projects/swiftdental/.meteor/local/build/programs/server/profile.js:280:14)
W20200428-12:32:07.608(-4)? (STDERR)     at /Users/dean/Projects/swiftdental/.meteor/local/build/programs/server/boot.js:463:13

Thanks

Mongodb constantly at 100%+ of CPU

I've got a 2-core system running on Digital Ocean, and after ~3 days of tracking data, everything totally freezes up with Mongodb stuck at 100% on both cores. I'm using redis-oplog for oplog tailing, but that doesn't seem to help :(

Anyone else experiencing similar?

No 'Access-Control-Allow-Origin' header is present

Thanks for creating this!

I'm using mup to deploy this to a Digital Ocean droplet. My setup is as follows:

I keep encountering this issue: "Failed to load https://kadira.domain.com/simplentp/sync: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://www.domain.com' is therefore not allowed access."

I've tried jiggling with the settings in server/config/security.js -- am I on the right path?

Method '__dynamicImport' not found

Just deployed this to production for the first time and I'm seeing this error come up in the Kadira logs:

message: Method '__dynamicImport' not found [404] 
stack:
Error: Method '__dynamicImport' not found [404]
    at Session.method (packages/ddp-server/livedata_server.js:671:18)
    at packages/lmachens_kadira.js:2617:38
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:15)
    at Session.sessionProto.protocol_handlers.method (packages/lmachens_kadira.js:2616:44)
    at packages/ddp-server/livedata_server.js:559:43

I'm running Meteor 1.6.1 -- not sure if that has something to do with it?

Error: 'very strange! - no entries found'

Hi,
I setup mongoDB in my system manually and I use this Docker command:

sudo docker run -d --name meteor-apm-server --net="host" \
  -p 4000:80 \
  -p 7007:7007 \
  -p 11011:11011 \
  -e PORT=80 \
  -e MONGO_URL=mongodb://localhost:27017/meteor \
  -e MONGO_OPLOG_URL=mongodb://localhost:27017/local \
  -e ROOT_URL=http://url.domain.com \
  strictlyskyler/meteor-apm-server:1.1.0

Kadira-UI run and I changed my password. On my Meteor application I have I20180612-20:23:50.341(2)? Kadira: completed instrumenting the app and I20180612-20:23:50.362(2)? Kadira: Successfully connected. But, the dashboard is always at the Configure Your App page.

And with a Docker logs I have this message:

in the root directory of your application.
Kadira Alertsman started
Fetchman started on port: 7007
starting apm-engine on port 11011
very strange! - no entries found
very strange! - no entries found
very strange! - no entries found
very strange! - no entries found
very strange! - no entries found
very strange! - no entries found
very strange! - no entries found
very strange! - no entries found
very strange! - no entries found
very strange! - no entries found
very strange! - no entries found
very strange! - no entries found
   cleaned up 0 in 7 ms

Any ideas ? Thanks.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.