jp / ambari Goto Github PK
View Code? Open in Web Editor NEWChef cookbook to install Ambari server and Ambari agent, then deploy a Hadoop cluster using the Blueprints API
License: Apache License 2.0
Chef cookbook to install Ambari server and Ambari agent, then deploy a Hadoop cluster using the Blueprints API
License: Apache License 2.0
For ambari v2.1 the attribute rhel_7_repo is set already but the case is missing so the repo is not selected making the recipe fail.
Fix:
case node['platform']
when 'redhat', 'centos', 'amazon', 'scientific'
case node['platform_version'].to_i
when 5
yum_repo = node['ambari']['rhel_5_repo']
when 6
yum_repo = node['ambari']['rhel_6_repo']
when 7
yum_repo = node['ambari']['rhel_7_repo']
end
remote_file '/etc/yum.repos.d/ambari.repo' do
source yum_repo
not_if { ::File.exist?('/etc/yum.repos.d/ambari.repo') }
end
Julien; while trying ambari to deploy applications throughout the cluster we encountered an issue with the version of snappy; it's a known problem for rhel7/hdp, see http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.1.0/bk_releasenotes_ambari_2.1.0.0/content/ambari_relnotes-2.1.0.0-known-issues.html
resource_management.core.exceptions.
Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install snappy-devel' returned 1.
Error: Package: snappy-devel-1.0.5-1.el6.x86_64 (HDP-UTILS-1.1.0.20)
Requires: snappy(x86-64) = 1.0.5-1.el6
Installed: snappy-1.1.0-3.el7.x86_64 (@anaconda/7.1)
snappy(x86-64) = 1.1.0-3.el7
Available: snappy-1.0.5-1.el6.x86_64 (HDP-UTILS-1.1.0.20)
snappy(x86-64) = 1.0.5-1.el6
A fix can be to add following to the agent recipe:
yum_package 'remove snappy 1.1.0-3.el7' do
package_name 'snappy'
version '1.1.0-3.el7'
action :remove
end
I realize that this may not be the best solution as the version is hardcoded and did not test other platforms or ambari versions.
Paul
Is there a way to pass in external database configuration for ambari-server to use instead of defaulting to the embedded postgres? If not, this would be in incredibly useful feature
Ran into an issue today, using this cookbook to expand an old cluster that was provisioned with Ambari 2.0.0.
The repos under the case switch https://github.com/jp/ambari/blob/master/attributes/default.rb#L17 for Ambari 2.0.0 had been switched to the repos for 2.0.1 in #4ed33fb.
This meant any node provisioned with that code and added to an existing cluster would have an inconsistent version on the Ambari Server (2.0.0) and the newly-provisioned Ambari Agent (2.0.1) and fail to register.
INFO 2015-10-29 18:56:40,612 NetUtil.py:60 - Connecting to https://ip-10-0-1-168.ec2.internal:8440/connection_info
INFO 2015-10-29 18:56:40,724 security.py:93 - SSL Connect being called.. connecting to the server
INFO 2015-10-29 18:56:40,816 security.py:55 - SSL connection established. Two-way SSL authentication is turned off on the server.
ERROR 2015-10-29 18:56:40,860 Controller.py:143 - Cannot register host with non compatible agent version, hostname=ip-10-0-1-161.ec2.internal, agentVersion=2.0.1, serverVersion=2.0.0
INFO 2015-10-29 18:56:40,860 Controller.py:350 - Registration response from ip-10-0-1-168.ec2.internal was FAILED
Anyways, I ran into this today, saw the bug, and fixed it in 9876342.
It seems like supermarket release is lagging by almost a year. By any chance you could update it?
Thanks for maintaining this!
Some of the files do have a comment at the top stating the license is Apache 2.0, but it would be much clearer if there were a LICENSE file at the root level of the repo with the entire Apache 2.0 license, and if the README specified the license as well. Right now, the README has a section called "License and Authors" but doesn't specify a license; it only lists an author.
Thanks!
Hello all!
I am using Centos 7, almost a pristine minimal installation, with few changes only for hostname and NTP server.
I have Python 2.7 installed, as it is required by Ambari on this system.
I have successfully deployed an Ambari Server using this cookbook. However when I try to deploy the agents using the other recipe I find that chef fails to execute the resource 'service[ambari-agent]', the error I pasted below is shown and then thousands of lines where the system tries to chown for root:root everything under /proc directory, leading to a system where almost anything can work.
` * service[ambari-agent] action start
================================================================================
Error executing action `start` on resource 'service[ambari-agent]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '255'
---- Begin output of /sbin/service ambari-agent start ----
STDOUT: Verifying Python version compatibility...
Using python /usr/bin/python
Checking for previously running Ambari Agent...
Starting ambari-agent
Verifying ambari-agent process status...
ERROR: ambari-agent start failed. For more details, see /ambari-agent.out:
====================
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 387, in <module>
main(heartbeat_stop_callback)
File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 363, in main
controller = Controller(config, server_hostname, heartbeat_stop_callback)
File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 107, in __init__
self.heartbeat_idle_interval_min = int(self.config.get('heartbeat', 'idle_interval_min')) if self.config.get('heartbeat', 'idle_interval_min') else self.netutil.HEARTBEAT_IDLE_INTERVAL_DEFAULT_MIN_SEC
File "/usr/lib/python2.6/site-packages/ambari_agent/AmbariConfig.py", line 172, in get
raise err
ConfigParser.NoOptionError: No option 'idle_interval_min' in section: 'heartbeat'
====================
Agent out at: /ambari-agent.out
Agent log at: /ambari-agent.log
STDERR: mkdir: cannot create directory '': No such file or directory`
Any Idea why this is happening?
Hi,
Ambari Server's configuration is stored in /etc/ambari-server/conf/ambari.properties. For example, to enable ldap authentication requires changing numerous settings in this file. Would be great for ambari.properties to be configurable by the server.rb recipe. Many thanks.
Hey JP,
I am automating a single-node install in Kitchen. I've never used the blueprints recipe before and I'm not fully clear on how to do that from the README.md.
README.md - https://github.com/jp/ambari#blueprints-attributes
Recipe - https://github.com/jp/ambari/blob/master/recipes/blueprints.rb
I'm conversant in Ambari blueprints and have been using them in some capacity for the last year or so. We've just never automated this step through Chef before.
I'm looking at the code and the docs and I am not sure how to use the following two attributes correctly:
node['ambari']['blueprints']['blueprint_json'] = JSON description of the Blueprint. See a single-node example here : https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-ExampleBlueprint
node['ambari']['blueprints']['cluster_json'] = JSON template for the Cluster. See a single-node example here: https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-ExampleClusterCreationTemplate
Do those attributes need to be defined as file paths within the cookbooks? Are we creating these .json files as cookbook templates, then converting to .json in the recipe? I guess I'm looking for an example of how this works.
So let's say I have the following cluster_creation.json and single_node_blueprint.json. Where would I put those files within the cookbook, and what would I configure as these two attributes?
Happy to submit a patch for this once I understand it better.
Regards,
Joe Reid
I've exported my ambari config to json. When running the ambari::blueprints chef recipe and importing, curl fails to post the blueprint with error: "execute[Init Blueprints] action runsh: 1: Syntax error: "(" unexpected"
From the json, I have copied from the brace after "blueprint_json": to the corresponding closing brace and attached here: json_file.txt
I can't upload .json files to this website, so had to rename to .txt. If you open, remove line-breaks in text using [http://www.textfixer.com/tools/remove-line-breaks.php], and at shell prompt type: curl --user admin:<password> -H 'X-Requested-By:ambari-cookbook' --data '<paste_text>' <your-ambari-server>:8080/api/v1/blueprints/my_blueprint
Similar error occurs: -bash: syntax error near unexpected token `('
Interestingly, if you curl post this json as a file rather than text it works fine: at shell prompt type: curl --user admin:<password> -H 'X-Requested-By:ambari-cookbook' --data @<json_file> <your-ambari-server>:8080/api/v1/blueprints/my_blueprint
Is this a bug? If so, saving the ['ambari']['blueprints']['blueprint_json'] value to a file and curl-posting this instead of the text to ambari seems the solution.
My ambari config not particularly complex, so I expect others will export and experience this issue.
Hi,
Thanks for this cookbook, but on Ubuntu 15.04,
I'm having this error on the agent:
ERROR 2015-10-12 14:39:49,408 NetUtil.py:78 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
ERROR 2015-10-12 14:39:49,408 NetUtil.py:79 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
When I curl the server port 8440, I'm getting:
curl: (60) SSL certificate problem: self signed certificate
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
Any suggestion ?
Hey guys!
Ambari 2.0 has been out for awhile now. There are some pretty significant changes, so I think it's probably a good idea to support multiple versions.
I have a forked version supporting 2.0 I've been running for about a week now. I'll put together a pull request if there is interest in that.
Regards,
Joe Reid
Hey there!
I ran into error No resource or method named
not_if' for Chef::Recipe "setup_package_manager
when attempting to use this cookbook with the ChefDK (0.4.0) and Chef (12.2.1).
================================================================================
10.0.1.78 Recipe Compile Error in /var/chef/cache/cookbooks/hdp/recipes/ambari_server.rb
10.0.1.78 ================================================================================
10.0.1.78
10.0.1.78 NoMethodError
10.0.1.78 -------------
10.0.1.78 No resource or method named `not_if' for `Chef::Recipe "setup_package_manager"'
10.0.1.78
10.0.1.78 Cookbook Trace:
10.0.1.78 ---------------
10.0.1.78 /var/chef/cache/cookbooks/ambari/recipes/setup_package_manager.rb:45:in `from_file'
10.0.1.78 /var/chef/cache/cookbooks/ambari/recipes/server.rb:20:in `from_file'
10.0.1.78 /var/chef/cache/cookbooks/hdp/recipes/ambari_server.rb:1:in `from_file'
10.0.1.78
10.0.1.78 Relevant File Content:
10.0.1.78 ----------------------
10.0.1.78 /var/chef/cache/cookbooks/ambari/recipes/setup_package_manager.rb:
10.0.1.78
10.0.1.78 38: yum_repo = node['ambari']['rhel_5_repo']
10.0.1.78 39: when 6
10.0.1.78 40: yum_repo = node['ambari']['rhel_6_repo']
10.0.1.78 41: end
10.0.1.78 42: remote_file "/etc/yum.repos.d/ambari.repo" do
10.0.1.78 43: source yum_repo
10.0.1.78 44: end
10.0.1.78 45>> not_if do
10.0.1.78 46: ::File.exists?("/etc/yum.repos.d/ambari.repo")
10.0.1.78 47: end
10.0.1.78 48: when "suse"
10.0.1.78 49: remote_file "/etc/zypp/repos.d/ambari.repo" do
10.0.1.78 50: source node['ambari']['suse_11_repo']
10.0.1.78 51: end
10.0.1.78 52: not_if do
10.0.1.78 53: ::File.exists?("/etc/zypp/repos.d/ambari.repo")
10.0.1.78 54: end
Versions:
chef (12.2.1, 12.0.3)
chef-dk (0.4.0)
chef-provisioning (1.0.1, 0.18)
chef-provisioning-aws (1.0.4, 0.2.1)
chef-provisioning-azure (0.1)
chef-provisioning-fog (0.12)
chef-provisioning-vagrant (0.8.1)
chef-vault (2.4.0)
chef-zero (4.2.1, 3.2.1, 1.5.6)
cheffish (1.1.2, 0.9.2)
chefspec (4.2.0)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.