appscale / appscale-tools Goto Github PK
View Code? Open in Web Editor NEWA set of command-line tools that can be used to interact with AppScale.
License: Other
A set of command-line tools that can be used to interact with AppScale.
License: Other
Noticed on an old deployment: doing appscale status
(with or without -v
) can return success:
AppScale is up. All 0 nodes are loaded
Even though the deployment has been torn down. In this case the login node IP has been re-used and is ping-able, but SSH into it with creds in .appscale is not possible.
The advanced deployment support provided by AppScale enables a potentially large number of deployments to be run. However, we have not exhaustively tested these combinations. Therefore, we want to test certain deployments (see below) and warn users if they run something we don't regularly test. For the purposes of the following, these deployments are supported:
Python 2.7.9 enables certificate verification by default for http clients. This causes an issue when the SOAPpy client tries to pass parameters to an AppController. Python throws an SSLError exception (certificate verify failed) since the AppController is using a self-signed certificate.
One (not recommended) fix would be to globally disable verification.
I think a better fix would be to pass a custom SSLContext to SOAPpy, but I don't think it supports that feature at this time. Replacing _create_default_https_context
with a function that creates a custom context is a possible temporary workaround.
However, the CN on the self-signed certificate (appscale.com) won't match the hostname that the SOAPpy client requests (the node's ip address). I don't know of a good way to fix this. Telling SOAPpy to set a Host: appscale.com
header might work, but I'm not sure if SOAPpy supports this. Generating a certificate for each node (with the CN set to the node's ip address) might also work, but I'm not sure if that would have other undesirable consequences.
I can start working on a pull request once a decision is made on how to address this issue.
Right now the tools write both a YAML and JSON file that contain metadata about the cloud deployment, and should be copying these files to the head node in AppScale. We do copy the YAML file, but don't copy the JSON file, so we should copy it as well. Not copying the JSON file means that when the user tries to upload an app via the web UI, they get this error:
Message about your submission:
/usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:92:in `initialize': No such file or directory - /root/.appscale/locations-appscalecgb6.json (Errno::ENOENT)
from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:92:in `open'
from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:92:in `read_file'
from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:800:in `read_nodes_json'
from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:757:in `get_all_public_ips'
from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:156:in `update_locations_file'
from /usr/local/appscale-tools/bin/../lib/appscale_tools.rb:377:in `upload_app'
from /usr/local/appscale-tools/bin/appscale-upload-app:12
The error returned is misleading because the ami exists. However, the command failed because the ec2 tools were not setup right on the machine. So we need to check if the local installation of ec2 tools works well.
ubuntu@ip-10-110-94-46:~/keys$ appscale up
The machine image you specified, ami-0936bd60, was not found when querying ec2-describe-images. Please specify a machine image that does exist and try again.
/usr/local/appscale-tools/bin/../lib/../lib/vm_tools.rb:166:in validate_machine_image' from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:1326:in
validate_run_instances_options'
from /usr/local/appscale-tools/bin/../lib/appscale_tools.rb:466:in `run_instances'
from /usr/local/appscale-tools/bin/appscale-run-instances:15
We noticed that your AppScale deployment failed because of a InfrastructureException error. Is it ok if we gather the logs from your AppScale deployment to your local machine, to aid in debugging this problem? (Y/N) N
Not copying over logs - quitting!
This is after I've started AppScale, shut it down, and now trying to restart it:
About to start AppScale 1.6.6 over a non-cloud environment.
New secret key is ****************************Y9ry
Log in to your head node: ssh -i /root/.appscale/appscale.key [email protected]
{"table"=>"cassandra", "hostname"=>"192.168.55.208", "keyname"=>"appscale", "keypath"=>"appscale.key", "replication"=>"1", "autoscale"=>"true", "ips"=>"", "appengine"=>"1", "group"=>"appscale"}
Head node successfully created at 192.168.55.208. It is now starting up cassandra via the command line arguments given.
Generating certificate and private key
Starting server at 192.168.55.208
Please wait for the controller to finish pre-processing tasks.
Please wait for AppScale to prepare your machines for use.
The location file did not contain a secret tag.
/usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:763:in get_from_yaml' from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:818:in
get_secret_key'
from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:171:in update_locations_file' from /usr/local/appscale-tools/bin/../lib/appscale_tools.rb:485:in
run_instances'
from /usr/local/appscale-tools/bin/appscale-run-instances:15
We noticed that your AppScale deployment failed because of a AppScaleException error. Is it ok if we gather the logs from your AppScale deployment to your local machine, to aid in debugging this problem? (Y/N) N
Not copying over logs - quitting!
Currently, if the user's app does not have an app.yaml or appengine-web.xml, they will get the following error message during a deploy: "coercing to Unicode: need string or buffer, NoneType found".
When running over Eucalyptus, if a user creates too many keys or security groups (as can happen by running AppScale over and over again), when Eucalyptus will run out of network tags. When the tools call run-instances
in this scenario, the following error message is returned:
RunInstancesType: Failed to allocate network tag for network: arn:aws:euca:eucalyptus:0000000000:security-group/boogroup/: no network tags are free
Currently the tools do not realize this is an error message, and assume that the run-instances
command succeeded, and proceed to wait for the instance to start (which never happens). Need to catch this error message and abort execution.
If users attempt to use version X of the tools with version Y of AppScale, the tools should complain about the mismatch and abort. Currently, they do not, and erroneously report instead about the database versions mismatching.
When specifying "127.0.0.1" as an IP address for a node in the AppScalefile, the tools do not give a useful explanation about why the "up" failed.
Need to investigate under what conditions this happens, and does not happen. Effects appscale-tools/lib/agents/ec2__agent.py when using configure_instance_security(). Problem commonly found when using appscale-create-image tool
When running "appscale up" or "appscale-run-instances" for the first time, the SSH key is copied to the machine. However, if the user has not set a root password before doing so, he/she is unable to complete the process. Users must be instructed to create a root password before deploying AppScale.
The new administrator password must be at least six characters long and can include non-alphanumeric characters.
Enter your new password:
Enter again to verify:
<nothing shows up here until AppScale starts which might be 1-2 minutes>
As it is, the tools omit the list of configured handlers when making Admin API calls, requiring the controller to parse the version source. It would be better to keep them together with the rest of the version configuration options.
The database that your Google App Engine applications will be backed by.
Defaults to 'cassandra', but 'hypertable' is also supported.
table : 'cassandra'
Optionally, since we only support one datastore right now, do not provide this option at all.
At the moment, we just support the "module" keyword.
Here is an example bad layout mixing advance deployments and simple deployments.
ips_layout :
controller: 128.111.55.206
servers:
Issue #89 revealed that on some Xen deployments, writing the locations.yaml file would result in binary characters being present at the top of the file, which would cause reading the file to fail. Pull request #91 fixed this problem by removing anything before the initial ---
at the top of the YAML file, but we still aren't sure why the binary would be there in the first place. Need to investigate why and address this properly.
Currently, the default rake task runs the old unit tests (in test/test_*.rb
). Running rake
therefore does not run our current unit tests, which are of the form test/tc_*.rb
. Need to change accordingly, remove old gem requires that we don't use anymore, and test it out.
When a private network is setup, AppScale should know the public IP redirect should be routed to.
Running latest version of tools on OS X Mountain Lion with python 2.7
Traceback (most recent call last):
File "/usr/bin/appscale", line 21, in
import version_helper
ImportError: No module named version_helper
Should it say sudo bash osx/appscale_install.sh
instead of sudo bash osx/appscale_build.sh
, as the latter doesn't exist?
Also, should 2.2.0 be marked as the latest release? Current "Latest Release" is still 2.0.0, not even 2.1.0.
Thanks!
Currently we use the --test
flag to enable users to skip keying in a cloud admin username and password. However, it defaults to one (unsecure) username and password. Would like to extend this to take in the username and password from environment variables.
Back in the days of Euca2, we introduced a --iaas flag, which was a shorthand form of --infrastructure euca. We would use this flag by default, since the Euca2ools worked on both Eucalyptus and EC2. However, as of Euca3, this is no longer the case, so using --iaas on EC2 will not work. Therefore, disable this feature and make the user tell us what cloud they're on.
On line 424 of ec2_agent.py, the script checks to see if the given keyname is being used by any terminated instances. However, the error messages does not say that. It says to check SSH/"Key Pairs" screen an delete the existing key. Actual message:
'SSH keyname {0} is already registered. Please change the "keyname" specified in your AppScalefile to a different value, or erase it to have one generated for you.'
There are three problems with that error message. 1) It's not correct and 2) the script creates that Key Pair earlier in the process which made me keep deleting it over and over again when the real issue is that a terminated instance still had the same key. 3) The startup process has automatically hardcoded that "random" keyname at the end of the AppScalefile without telling the user that it has done so.
Here's what I suggest instead for a message:
SSH keyname {0} is already registered to a terminated instance. Either wait for the terminated instance to be removed or remove the keyname that was automatically added to the end of your AppScalefile.
Currently, we re-find and re-parse the application configuration files for every element we need. It would be nice if we had a single class to represent a configuration resource.
This error shows up even if you use "--help" or "help".
$ ./appscale-tools/bin/appscale help
Traceback (most recent call last):
File "./appscale-tools/bin/appscale", line 130, in
appscale.help()
File "./appscale-tools/bin/../lib/appscale.py", line 156, in help
raise UsageException(self.USAGE)
custom_exceptions.UsageException:
Usage: appscale command []
Available commands:
init: Writes a new configuration file for starting AppScale.
up: Starts a new AppScale instance.
ssh: Logs into a virtual machine in a currently running AppScale deployment.
status: Reports on the state of a currently running AppScale deployment.
deploy: Deploys a Google App Engine app to AppScale.
undeploy: Removes a Google App Engine app from AppScale.
remove: An alias for 'undeploy'.
tail: Follows the output of log files in a currently running AppScale deployment.
logs: Collects the logs produced by an AppScale deployment.
destroy: Gracefully terminates the currently running AppScale deployment.
down: An alias for 'destroy'.
clean: Forcefully terminates all services in a cluster AppScale deployment.
help: Displays this message.
Doing an advance placement of datastore slaves has the tools complain about not having enough db nodes. saying the number of db nodes is less than the requested replication.
The node index does not get updated after a new node is added to an existing deployment.
msrestazure
requires keyring
which requires SecretStorage
. SecretStorage stopped maintaining Python 2.7 starting from its 3.0.0 version. To fix it we have explicitly require SecretStorage<3.
When running over a virtualized cluster, the terminate script leaves Erlang processes hanging around. This causes RabbitMQ to not start up on subsequent runs.
When the tools are installed via gem, they do not install the euca2ools or ec2 tools (which they rely on for cloud deployments). Need to change this so that installing via gem also checks to see which tools exist, and only installs the ones that are needed.
root@lucid64:~/first_try# appscale up
Traceback (most recent call last):
File "/root/appscale-tools/bin/appscale", line 46, in <module>
appscale.up()
File "/root/appscale-tools/bin/../lib/appscale.py", line 193, in up
subprocess.call(add_keypair_command)
File "/usr/lib/python2.6/subprocess.py", line 480, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
For public port forwarding. Useful for doing a private IP deployment and then having a single public IP for access.
Originally Java apps had an optional flag called <threadsafe>
, but at some point this flag became mandatory. Since we don't check for its existence, users can give us an app without it and we will blindly accept it, causing the app to fail when AppScale tries to start up the Java AppServer.
Need to check for this tag, ensure that it's value is either true
or false
, and abort if it is not.
Currently the tools do not validate the argument passed to --machine
, so if a user passes in a machine image that doesn't exist, the tools try to start up a machine with that ID and hang forever. Need to verify that the machine image actually exists before trying to start up a machine with its ID.
When I run bash appscale-tools/debian/appscale_build.sh
, It seems that there's requests version conflict in the script
Installed /usr/local/lib/python2.7/dist-packages/appscale_tools-3.1.0-py2.7.egg
Processing dependencies for appscale-tools==3.1.0
Searching for requests>=2.7.0
Reading https://pypi.python.org/simple/requests/
Downloading https://pypi.python.org/packages/5b/0b/34be574b1ec997247796e5d516f3a6b6509c4e064f2885a96ed885ce7579/requests-2.12.4.tar.gz#md5=acdb48888a9d3c7309da12fc7f83fedb
Best match: requests 2.12.4
Processing requests-2.12.4.tar.gz
Writing /tmp/easy_install-SQy_XG/requests-2.12.4/setup.cfg
Running requests-2.12.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-SQy_XG/requests-2.12.4/egg-dist-tmp-JaMIKU
warning: no files found matching 'test_requests.py'
creating /usr/local/lib/python2.7/dist-packages/requests-2.12.4-py2.7.egg
Extracting requests-2.12.4-py2.7.egg to /usr/local/lib/python2.7/dist-packages
Adding requests 2.12.4 to easy-install.pth file
Installed /usr/local/lib/python2.7/dist-packages/requests-2.12.4-py2.7.egg
error: requests 2.12.4 is installed but requests!=2.12.*,>=2.0.0 is required by set(['adal'])
Unable to complete AppScale tools installation.
If the app-engine.xml does not exists, the following stacktrace happens. The error message at interface is just: "coercing to Unicode: need string or buffer, NoneType found".
Take a look in appengine_helper.py:233 to add a better info message :) It happens if the app-engine.xml is not found.
stacktrace : Traceback (most recent call last):
File "/usr/local/appscale-tools/bin/appscale", line 87, in <module>
appscale.deploy(sys.argv[2])
File "/usr/local/appscale-tools/bin/../lib/appscale.py", line 449, in deploy
return AppScaleTools.upload_app(options)
File "/usr/local/appscale-tools/bin/../lib/appscale_tools.py", line 561, in upload_app
app_id = AppEngineHelper.get_app_id_from_app_config(file_location)
File "/usr/local/appscale-tools/bin/../lib/appengine_helper.py", line 165, in get_app_id_from_app_config
app_config_file = cls.get_config_file_from_dir(app_dir)
File "/usr/local/appscale-tools/bin/../lib/appengine_helper.py", line 233, in get_config_file_from_dir
elif os.path.exists(cls.get_appengine_web_xml_location(app_dir)):
File "/usr/lib/python2.7/genericpath.py", line 18, in exists
os.stat(path)
TypeError: coercing to Unicode: need string or buffer, NoneType found
exception : TypeError
locale : en_US
tools_version : 1.14.0
platform : Linux-3.2.0-32-generic-x86_64-with-Ubuntu-12.04-precise
message : coercing to Unicode: need string or buffer, NoneType found
runtime : CPython
<app_id> for the app id gives an odd error back:
sh: cannot open appid: No such file
./../lib/../lib/common_functions.rb:936:in `size': No such file or directory - /tmp/mgAsLDmHg3/<appid>.tar.gz (Errno::ENOENT)
from ./../lib/../lib/common_functions.rb:936:in `warn_on_large_app_size'
from ./../lib/../lib/common_functions.rb:998:in `get_app_info'
from ./../lib/../lib/common_functions.rb:904:in `get_app_name_from_tar'
from ./../lib/appscale_tools.rb:393:in `upload_app'
from ./appscale-upload-app:12
I didn't know that... oops.
When it will be supported?
When the tools start AppScale, they begin by starting an AppController on one node, who then starts AppScale on its own node and elsewhere. However, if this starting process fails (e.g., if the message to god to start isn't received successfully), then the tools may hang. Need to properly catch a timeout here and retry accordingly.
For distributions which are not lucid.
I created a formula, which successfully installs appscale-tools. However when I ran script, for example appscale init cluster
, I faced problems that templates couldn't be found etc.
So I'm here to ask if it would be good to place them under share/appscale
or some other directory and then they would be available from /usr/local/share/appscale/
.
My current version of formula: https://github.com/samitheberber/homebrew/blob/appscale-tools/Library/Formula/appscale-tools.rb
Currently the tools assume that the first node in the system can be logged into via ssh without any problems. This isn't true in cloud deployments if the first node's IP / DNS address is the same as something already in the user's ~./ssh/known_hosts
file, as SSH will complain about a possible man-in-the-middle attack and not allow the tools to access the machine via SSH or SCP. Need to correctly handle this error condition.
HttpError 400 when requesting https://www.googleapis.com/compute/v1beta15/projects//global/firewalls?alt=json returned "Invalid value for field 'resource.network': 'https://www.googleapis.com/compute/v1beta15/projects//global/networks/appscaleb2dfd327b2424db9ad1aa085cf184680'. Cross-project references for this resource type are not allowed.">
appscale-remove-app --appname guestbook
on a single or multiple node cluster fails. Just hangs. Either via web interface or this commandline.
On some Xen deployments, the AppScale tools writes a locations.yaml file that has a lot of binary at the beginning of it. This causes attempts to read the locations.yaml file to fail, and thus causes appscale-run-instances
to abort.
When AppScale deploys, the tools print out the contents of the message it sends to the first AppController in the system. In EC2 and Eucalyptus deployments, these contents include the user's credentials and are thus leaked to the terminal, which causes security problems if the user sends this output (typically as a log) to others for help. Need to obscure these credentials.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.