nanit / kubernetes-graphite-cluster Goto Github PK
View Code? Open in Web Editor NEWStatsD + Graphite cluster on top of Kubernetes
License: MIT License
StatsD + Graphite cluster on top of Kubernetes
License: MIT License
Currently events are not shared between the nodes and uses local sqlite db
Is there any plan to update kube-watch.js scripts in order for the project to be deployable on newer kubernetes?
The scripts currently use version 3.12 of kubernetes-client
, whilst current version is 5.3.0, and they break on k8s v1.9
Would require fetching graphite nodes DNS names and inserting them to carbon relay configuration and graphite query node configuration
I have a Kube 1.6.2 deployed over CentOS VMs using Kubeadm.
When I tried to deploy the carbon/graphite cluster, I got the following errors in the Make progress:
...
...
Step 22 : RUN chmod 0664 /opt/graphite/storage/graphite.db
---> Running in b4297ad8b242
---> e468c1e48708
Removing intermediate container b4297ad8b242
Step 23 : RUN cp /src/graphite-web/webapp/manage.py /opt/graphite/webapp
---> Running in 8454921eed63
---> fc42a56abfc4
Removing intermediate container 8454921eed63
Step 24 : RUN cd /opt/graphite/webapp/ && python manage.py migrate --run-syncdb --noinput
---> Running in b01ad861d148
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 302, in execute
settings.INSTALLED_APPS
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 55, in __getattr__
self._setup(name)
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 43, in _setup
self._wrapped = Settings(settings_module)
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 99, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/opt/graphite/webapp/graphite/settings.py", line 153, in <module>
from graphite.local_settings import * # noqa
File "/opt/graphite/webapp/graphite/local_settings.py", line 246
CLUSTER_SERVERS = [@@GRAPHITE_NODES@@]
^
SyntaxError: invalid syntax
The command '/bin/sh -c cd /opt/graphite/webapp/ && python manage.py migrate --run-syncdb --noinput' returned a non-zero code: 1
make: *** [docker-graphite-master] Error 1
This happens in the carbon-relay pod logs:
NAMESPACE NAME READY STATUS RESTARTS AGE
default carbon-relay-2782403225-cvtk3 0/1 CrashLoopBackOff 5 23m
default carbon-relay-2782403225-xf7nk 0/1 CrashLoopBackOff 9 23m
default carbon-relay-2782403225-zmlf5 0/1 ErrImagePull 0 23m
Log:
kubectl logs carbon-relay-2782403225-xf7nk
+ /set-cluster-nodes.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to kubernetes port 443: Connection refused
+ exec /opt/graphite/bin/carbon-relay.py --debug --logdir=/var/log/carbon start
Traceback (most recent call last):
Starting carbon-relay (instance a)
File "/opt/graphite/bin/carbon-relay.py", line 32, in <module>
run_twistd_plugin(__file__)
File "/opt/graphite/lib/carbon/util.py", line 96, in run_twistd_plugin
runApp(config)
File "/usr/local/lib/python2.7/dist-packages/twisted/scripts/twistd.py", line 23, in runApp
_SomeApplicationRunner(config).run()
File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 376, in run
self.application = self.createOrGetApplication()
File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 436, in createOrGetApplication
ser = plg.makeService(self.config.subOptions)
File "/opt/graphite/lib/twisted/plugins/carbon_relay_plugin.py", line 21, in makeService
return service.createRelayService(options)
File "/opt/graphite/lib/carbon/service.py", line 135, in createRelayService
setupPipeline(['relay'], root_service, settings)
File "/opt/graphite/lib/carbon/service.py", line 86, in setupPipeline
setupRelayProcessor(root_service, settings)
File "/opt/graphite/lib/carbon/service.py", line 175, in setupRelayProcessor
for destination in util.parseDestinations(settings.DESTINATIONS):
File "/opt/graphite/lib/carbon/util.py", line 121, in parseDestinations
return [parseDestination(dest_string) for dest_string in destination_strings]
File "/opt/graphite/lib/carbon/util.py", line 117, in parseDestination
return server, int(port), instance
ValueError: invalid literal for int() with base 10: '@@GRAPHITE_NODES@@'
Would require dynamic fetching of statsd DNS names on statsd proxy entrypoint
I'm interested in leveraging your work. Are you planning on applying an Open Source License?
The values that you've setup in environments and makefile are difficult to manage, consider creating a helm chart https://github.com/kubernetes/helm and list your app on https://kubeapps.com/
I just set up a cluster per the recommendations here and found that I needed to set up the sqlite database for a first run lest specific queries fail such as graphite/metrics/find?query=*
:
# curl graphite/metrics/find\?query=\*
<body style="background-color: #666666; color: black;">
<center>
<h2 style='font-family: "Arial", sans-serif'>
<p>Graphite encountered an unexpected error while handling your request.</p>
<p>Please contact your site administrator if the problem persists.</p>
</h2>
<br/>
<div style="width: 50%; text-align: center; font-family: monospace; background-color: black; font-weight: bold; color: #ff4422;">
</div>
<div style="width: 70%; text-align: left; background-color: black; color: #44ff22; border: thin solid gray;">
<pre>
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 149, in get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 147, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/graphite/webapp/graphite/metrics/views.py", line 150, in find_view
profile = getProfile(request)
File "/opt/graphite/webapp/graphite/user_util.py", line 25, in getProfile
return default_profile()
File "/opt/graphite/webapp/graphite/user_util.py", line 41, in default_profile
'password': '!'})
File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 122, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 465, in get_or_create
return self.get(**lookup), False
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 381, in get
num = len(clone)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 240, in __len__
self._fetch_all()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1074, in _fetch_all
self._result_cache = list(self.iterator())
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 52, in __iter__
results = compiler.execute_sql()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 848, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 95, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/sqlite3/base.py", line 323, in execute
return Database.Cursor.execute(self, query, params)
OperationalError: no such table: auth_user
</pre>
</div>
</center>
Luckily, it wasn't too difficult to figure out how to set up the schema for a first run:
PYTHONPATH=/opt/graphite/webapp django-admin.py migrate --settings=graphite.settings --run-syncdb
I'm going to propose that this be done immediately following the creation of the database file or inside a file missing or zero length test in the entry point.
It's not clear what the default Graphite user/pass is, and the default that Graphite documents doesn't work. What is the default for the Graphite administrative panel user/pass?
Currently the setup has only been tested on AWS.
It would be nice to have it tested and adjusted to other k8s configurations as well.
I set up RBAC support over at bittorrent@ca441f7 along with a change to use EBS as our EFS volumes didn't have enough io.
Is there any interest in formatting the RBAC portions of that commit into a PR? If so, I would be happy to do so.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.