appscale / appscale-tools

A set of command-line tools that can be used to interact with AppScale.

AppScale GTS

AppScale GTS is an open source serverless platform for building and running scalable web and mobile applications on any infrastructure.

The platform enables developers to focus solely on business logic in order to rapidly build scalable apps, cleanly separating it from deployment and scaling logic. It allows operations to provide a consistent, tunable environment that can simplify running and maintaining apps on multiple infrastructures. The business will benefit from faster time-to-market, reduced operational costs, maximized application lifetime, and the flexibility to integrate with new or existing technologies.

AppScale GTS is open source and modeled on Google App Engine APIs, allowing developers to automatically deploy and scale unmodified Google App Engine applications over public and private cloud systems and on-premise clusters. It currently supports Python, Go, PHP and Java applications. The software was developed by AppScale Systems, Inc., based in Santa Barbara, California, and Google. In 2019, the company ended commercial support AppScale GTS, however the source code remains available in this GitHub Repo.

Why Use AppScale GTS?

The goal of AppScale GTS is to provide developers with a rapid, API-driven development platform that can run applications on any cloud infrastructure. AppScale GTS decouples application logic from its service ecosystem to give developers and cloud administrators control over application deployment, data storage, resource use, backup, and migration.

I Want ...

• to contribute
• to learn more

Documentation

• Users
  • Automated Data Persistence
  • Multinode Deployments
  • Making AppScale Scale
• Developers
  • Datastore Topics
  • Advanced Topics
• Architecture
  • AppScale's core components
• FAQs and Troubleshooting

Community and Support

Join the Community Google Group for announcements, help, and to discuss cloud research.

appscale-tools's Issues

Tool should validate that app names only have valid characters

<app_id> for the app id gives an odd error back:

sh: cannot open appid: No such file
./../lib/../lib/common_functions.rb:936:in `size': No such file or directory - /tmp/mgAsLDmHg3/<appid>.tar.gz (Errno::ENOENT)
    from ./../lib/../lib/common_functions.rb:936:in `warn_on_large_app_size'
    from ./../lib/../lib/common_functions.rb:998:in `get_app_info'
    from ./../lib/../lib/common_functions.rb:904:in `get_app_name_from_tar'
    from ./../lib/appscale_tools.rb:393:in `upload_app'
    from ./appscale-upload-app:12

'appscale status' returns 'is up' for a terminated cloud deployment

Noticed on an old deployment: doing appscale status (with or without -v) can return success:

AppScale is up. All 0 nodes are loaded

Even though the deployment has been torn down. In this case the login node IP has been re-used and is ping-able, but SSH into it with creds in .appscale is not possible.

Terminate script fails to kill Erlang processes

When running over a virtualized cluster, the terminate script leaves Erlang processes hanging around. This causes RabbitMQ to not start up on subsequent runs.

Add Unit Tests for bad layouts

Here is an example bad layout mixing advance deployments and simple deployments.
ips_layout :
controller: 128.111.55.206
servers:

• 128.111.55.218
• 128.111.55.203

Unable to find secret tag

This is after I've started AppScale, shut it down, and now trying to restart it:

About to start AppScale 1.6.6 over a non-cloud environment.
New secret key is ****************************Y9ry
Log in to your head node: ssh -i /root/.appscale/appscale.key [email protected]
{"table"=>"cassandra", "hostname"=>"192.168.55.208", "keyname"=>"appscale", "keypath"=>"appscale.key", "replication"=>"1", "autoscale"=>"true", "ips"=>"", "appengine"=>"1", "group"=>"appscale"}
Head node successfully created at 192.168.55.208. It is now starting up cassandra via the command line arguments given.
Generating certificate and private key
Starting server at 192.168.55.208
Please wait for the controller to finish pre-processing tasks.

Please wait for AppScale to prepare your machines for use.
The location file did not contain a secret tag.
/usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:763:in get_from_yaml' from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:818:inget_secret_key'
from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:171:in update_locations_file' from /usr/local/appscale-tools/bin/../lib/appscale_tools.rb:485:inrun_instances'
from /usr/local/appscale-tools/bin/appscale-run-instances:15

We noticed that your AppScale deployment failed because of a AppScaleException error. Is it ok if we gather the logs from your AppScale deployment to your local machine, to aid in debugging this problem? (Y/N) N
Not copying over logs - quitting!

Tools do not validate ami/emi in cloud deployments

Currently the tools do not validate the argument passed to --machine, so if a user passes in a machine image that doesn't exist, the tools try to start up a machine with that ID and hang forever. Need to verify that the machine image actually exists before trying to start up a machine with its ID.

'appscale up' does not work on a clean image

root@lucid64:~/first_try# appscale up
Traceback (most recent call last):
  File "/root/appscale-tools/bin/appscale", line 46, in <module>
    appscale.up()
  File "/root/appscale-tools/bin/../lib/appscale.py", line 193, in up
    subprocess.call(add_keypair_command)
  File "/usr/lib/python2.6/subprocess.py", line 480, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
    errread, errwrite)
  File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Investigate why YAML.dump puts binary at top of file

Issue #89 revealed that on some Xen deployments, writing the locations.yaml file would result in binary characters being present at the top of the file, which would cause reading the file to fail. Pull request #91 fixed this problem by removing anything before the initial --- at the top of the YAML file, but we still aren't sure why the binary would be there in the first place. Need to investigate why and address this properly.

Error in running appscale.bash

When I run bash appscale-tools/debian/appscale_build.sh, It seems that there's requests version conflict in the script

Installed /usr/local/lib/python2.7/dist-packages/appscale_tools-3.1.0-py2.7.egg
Processing dependencies for appscale-tools==3.1.0
Searching for requests>=2.7.0
Reading https://pypi.python.org/simple/requests/
Downloading https://pypi.python.org/packages/5b/0b/34be574b1ec997247796e5d516f3a6b6509c4e064f2885a96ed885ce7579/requests-2.12.4.tar.gz#md5=acdb48888a9d3c7309da12fc7f83fedb
Best match: requests 2.12.4
Processing requests-2.12.4.tar.gz
Writing /tmp/easy_install-SQy_XG/requests-2.12.4/setup.cfg
Running requests-2.12.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-SQy_XG/requests-2.12.4/egg-dist-tmp-JaMIKU
warning: no files found matching 'test_requests.py'
creating /usr/local/lib/python2.7/dist-packages/requests-2.12.4-py2.7.egg
Extracting requests-2.12.4-py2.7.egg to /usr/local/lib/python2.7/dist-packages
Adding requests 2.12.4 to easy-install.pth file

Installed /usr/local/lib/python2.7/dist-packages/requests-2.12.4-py2.7.egg
error: requests 2.12.4 is installed but requests!=2.12.*,>=2.0.0 is required by set(['adal'])
Unable to complete AppScale tools installation.

Users should be able to specify Nginx host

For public port forwarding. Useful for doing a private IP deployment and then having a single public IP for access.

Tools can fail to start AppController

When the tools start AppScale, they begin by starting an AppController on one node, who then starts AppScale on its own node and elsewhere. However, if this starting process fails (e.g., if the message to god to start isn't received successfully), then the tools may hang. Need to properly catch a timeout here and retry accordingly.

Boto key_pair failed to retrieve key from cloud

Need to investigate under what conditions this happens, and does not happen. Effects appscale-tools/lib/agents/ec2__agent.py when using configure_instance_security(). Problem commonly found when using appscale-create-image tool

Tools should validate AppScale version

If users attempt to use version X of the tools with version Y of AppScale, the tools should complain about the mismatch and abort. Currently, they do not, and erroneously report instead about the database versions mismatching.

Tools fails if binary in locations.yaml file

On some Xen deployments, the AppScale tools writes a locations.yaml file that has a lot of binary at the beginning of it. This causes attempts to read the locations.yaml file to fail, and thus causes appscale-run-instances to abort.

Allow for the manual setting of a Public IP for routing based on port forwarding

When a private network is setup, AppScale should know the public IP redirect should be routed to.

Fix Rakefile / default rake task

Currently, the default rake task runs the old unit tests (in test/test_*.rb). Running rake therefore does not run our current unit tests, which are of the form test/tc_*.rb. Need to change accordingly, remove old gem requires that we don't use anymore, and test it out.

Log when app-engine.xml do not exists

If the app-engine.xml does not exists, the following stacktrace happens. The error message at interface is just: "coercing to Unicode: need string or buffer, NoneType found".
Take a look in appengine_helper.py:233 to add a better info message :) It happens if the app-engine.xml is not found.

stacktrace : Traceback (most recent call last):
  File "/usr/local/appscale-tools/bin/appscale", line 87, in <module>
    appscale.deploy(sys.argv[2])
  File "/usr/local/appscale-tools/bin/../lib/appscale.py", line 449, in deploy
    return AppScaleTools.upload_app(options)
  File "/usr/local/appscale-tools/bin/../lib/appscale_tools.py", line 561, in upload_app
    app_id = AppEngineHelper.get_app_id_from_app_config(file_location)
  File "/usr/local/appscale-tools/bin/../lib/appengine_helper.py", line 165, in get_app_id_from_app_config
    app_config_file = cls.get_config_file_from_dir(app_dir)
  File "/usr/local/appscale-tools/bin/../lib/appengine_helper.py", line 233, in get_config_file_from_dir
    elif os.path.exists(cls.get_appengine_web_xml_location(app_dir)):
  File "/usr/lib/python2.7/genericpath.py", line 18, in exists
    os.stat(path)
TypeError: coercing to Unicode: need string or buffer, NoneType found

exception : TypeError

locale : en_US

tools_version : 1.14.0

platform : Linux-3.2.0-32-generic-x86_64-with-Ubuntu-12.04-precise

message : coercing to Unicode: need string or buffer, NoneType found

runtime : CPython

HTTPS certificate validation

Python 2.7.9 enables certificate verification by default for http clients. This causes an issue when the SOAPpy client tries to pass parameters to an AppController. Python throws an SSLError exception (certificate verify failed) since the AppController is using a self-signed certificate.

One (not recommended) fix would be to globally disable verification.

I think a better fix would be to pass a custom SSLContext to SOAPpy, but I don't think it supports that feature at this time. Replacing _create_default_https_context with a function that creates a custom context is a possible temporary workaround.

However, the CN on the self-signed certificate (appscale.com) won't match the hostname that the SOAPpy client requests (the node's ip address). I don't know of a good way to fix this. Telling SOAPpy to set a Host: appscale.com header might work, but I'm not sure if SOAPpy supports this. Generating a certificate for each node (with the CN set to the node's ip address) might also work, but I'm not sure if that would have other undesirable consequences.

I can start working on a pull request once a decision is made on how to address this issue.

Fix SecretStorage dependency version

msrestazure requires keyring which requires SecretStorage. SecretStorage stopped maintaining Python 2.7 starting from its 3.0.0 version. To fix it we have explicitly require SecretStorage<3.

Ubuntu trusty (14.04) not supported?

I didn't know that... oops.
When it will be supported?

Change tools to get username/pass from environment

Currently we use the --test flag to enable users to skip keying in a cloud admin username and password. However, it defaults to one (unsecure) username and password. Would like to extend this to take in the username and password from environment variables.

An IP address-related crash does not give a useful error

When specifying "127.0.0.1" as an IP address for a node in the AppScalefile, the tools do not give a useful explanation about why the "up" failed.

Homebrew friendly version

I created a formula, which successfully installs appscale-tools. However when I ran script, for example appscale init cluster, I faced problems that templates couldn't be found etc.

So I'm here to ask if it would be good to place them under share/appscale or some other directory and then they would be available from /usr/local/share/appscale/.

My current version of formula: https://github.com/samitheberber/homebrew/blob/appscale-tools/Library/Formula/appscale-tools.rb

remove app fails

appscale-remove-app --appname guestbook
on a single or multiple node cluster fails. Just hangs. Either via web interface or this commandline.

appscale.py argument parser error

This error shows up even if you use "--help" or "help".

$ ./appscale-tools/bin/appscale help
Traceback (most recent call last):
File "./appscale-tools/bin/appscale", line 130, in
appscale.help()
File "./appscale-tools/bin/../lib/appscale.py", line 156, in help
raise UsageException(self.USAGE)
custom_exceptions.UsageException:

Usage: appscale command []

Available commands:
init: Writes a new configuration file for starting AppScale.
up: Starts a new AppScale instance.
ssh: Logs into a virtual machine in a currently running AppScale deployment.
status: Reports on the state of a currently running AppScale deployment.
deploy: Deploys a Google App Engine app to AppScale.
undeploy: Removes a Google App Engine app from AppScale.
remove: An alias for 'undeploy'.
tail: Follows the output of log files in a currently running AppScale deployment.
logs: Collects the logs produced by an AppScale deployment.
destroy: Gracefully terminates the currently running AppScale deployment.
down: An alias for 'destroy'.
clean: Forcefully terminates all services in a cluster AppScale deployment.
help: Displays this message.

After I add a node I should be able to use "appscale ssh <new node id>

The node index does not get updated after a new node is added to an existing deployment.

Tools do not fail if no network tags are free

When running over Eucalyptus, if a user creates too many keys or security groups (as can happen by running AppScale over and over again), when Eucalyptus will run out of network tags. When the tools call run-instances in this scenario, the following error message is returned:

RunInstancesType: Failed to allocate network tag for network: arn:aws:euca:eucalyptus:0000000000:security-group/boogroup/: no network tags are free

Currently the tools do not realize this is an error message, and assume that the run-instances command succeeded, and proceed to wait for the instance to start (which never happens). Need to catch this error message and abort execution.

Tools should warn users on unsupported deployments

The advanced deployment support provided by AppScale enables a potentially large number of deployments to be run. However, we have not exhaustively tested these combinations. Therefore, we want to test certain deployments (see below) and warn users if they run something we don't regularly test. For the purposes of the following, these deployments are supported:

1. Simple deployments
2. Advanced deployments with four nodes, where one node is the load balancer, one is the AppServer, one is the database, and one is ZooKeeper.
3. Advanced deployments with eight nodes, where one node is the load balancer, two are AppServers, two are databases, and three are ZooKeepers.

Disable use of --iaas flag in run-instances

Back in the days of Euca2, we introduced a --iaas flag, which was a shorthand form of --infrastructure euca. We would use this flag by default, since the Euca2ools worked on both Eucalyptus and EC2. However, as of Euca3, this is no longer the case, so using --iaas on EC2 will not work. Therefore, disable this feature and make the user tell us what cloud they're on.

README.md for OSX

Should it say sudo bash osx/appscale_install.sh instead of sudo bash osx/appscale_build.sh, as the latter doesn't exist?

Also, should 2.2.0 be marked as the latest release? Current "Latest Release" is still 2.0.0, not even 2.1.0.

Thanks!

Provide more informative error when config is missing

Currently, if the user's app does not have an app.yaml or appengine-web.xml, they will get the following error message during a deploy: "coercing to Unicode: need string or buffer, NoneType found".

Tools fail if same head node seen twice in cloud deployments

Currently the tools assume that the first node in the system can be logged into via ssh without any problems. This isn't true in cloud deployments if the first node's IP / DNS address is the same as something already in the user's ~./ssh/known_hosts file, as SSH will complain about a possible man-in-the-middle attack and not allow the tools to access the machine via SSH or SCP. Need to correctly handle this error condition.

Catch exceptions which should not lead to gathering log

Tools do not copy over JSON metadata file

Right now the tools write both a YAML and JSON file that contain metadata about the cloud deployment, and should be copying these files to the head node in AppScale. We do copy the YAML file, but don't copy the JSON file, so we should copy it as well. Not copying the JSON file means that when the user tries to upload an app via the web UI, they get this error:

Message about your submission:
/usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:92:in `initialize': No such file or directory - /root/.appscale/locations-appscalecgb6.json (Errno::ENOENT) 
from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:92:in `open' 
from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:92:in `read_file' 
from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:800:in `read_nodes_json' 
from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:757:in `get_all_public_ips' 
from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:156:in `update_locations_file' 
from /usr/local/appscale-tools/bin/../lib/appscale_tools.rb:377:in `upload_app' 
from /usr/local/appscale-tools/bin/appscale-upload-app:12

Remove "hypertable" from AppScalefile template

The database that your Google App Engine applications will be backed by.
Defaults to 'cassandra', but 'hypertable' is also supported.
table : 'cassandra'

Optionally, since we only support one datastore right now, do not provide this option at all.

ImportError: No module named version_helper

Running latest version of tools on OS X Mountain Lion with python 2.7

Traceback (most recent call last):
File "/usr/bin/appscale", line 21, in
import version_helper
ImportError: No module named version_helper

create_firewall does not work on GCE

HttpError 400 when requesting https://www.googleapis.com/compute/v1beta15/projects//global/firewalls?alt=json returned "Invalid value for field 'resource.network': 'https://www.googleapis.com/compute/v1beta15/projects//global/networks/appscaleb2dfd327b2424db9ad1aa085cf184680'. Cross-project references for this resource type are not allowed.">

After entering in username and password there should be an acknowledgement of success

The new administrator password must be at least six characters long and can include non-alphanumeric characters.
Enter your new password:
Enter again to verify:
<nothing shows up here until AppScale starts which might be 1-2 minutes>

Wrong Error Message

The error returned is misleading because the ami exists. However, the command failed because the ec2 tools were not setup right on the machine. So we need to check if the local installation of ec2 tools works well.

ubuntu@ip-10-110-94-46:~/keys$ appscale up
The machine image you specified, ami-0936bd60, was not found when querying ec2-describe-images. Please specify a machine image that does exist and try again.
/usr/local/appscale-tools/bin/../lib/../lib/vm_tools.rb:166:in validate_machine_image' from /usr/local/appscale-tools/bin/../lib/../lib/common_functions.rb:1326:invalidate_run_instances_options'
from /usr/local/appscale-tools/bin/../lib/appscale_tools.rb:466:in `run_instances'
from /usr/local/appscale-tools/bin/appscale-run-instances:15

We noticed that your AppScale deployment failed because of a InfrastructureException error. Is it ok if we gather the logs from your AppScale deployment to your local machine, to aid in debugging this problem? (Y/N) N
Not copying over logs - quitting!

Installing via gem doesn't install required software

When the tools are installed via gem, they do not install the euca2ools or ec2 tools (which they rely on for cloud deployments). Need to change this so that installing via gem also checks to see which tools exist, and only installs the ones that are needed.

Include handlers in version create calls

As it is, the tools omit the list of configured handlers when making Admin API calls, requiring the controller to parse the version source. It would be better to keep them together with the rest of the version configuration options.

Tools report EC2_ACCESS_KEY and EC2_SECRET_KEY to standard out

When AppScale deploys, the tools print out the contents of the message it sends to the first AppController in the system. In EC2 and Eucalyptus deployments, these contents include the user's credentials and are thus leaked to the terminal, which causes security problems if the user sends this output (typically as a log) to others for help. Need to obscure these credentials.

Consolidate configuration parsing functionality

Currently, we re-find and re-parse the application configuration files for every element we need. It would be nice if we had a single class to represent a configuration resource.

Advance placement has issues with setting replication

Doing an advance placement of datastore slaves has the tools complain about not having enough db nodes. saying the number of db nodes is less than the requested replication.

Make the tools install a setuptools script

For distributions which are not lucid.

Tools erroneously allow Java apps without <threadsafe> tag

Originally Java apps had an optional flag called <threadsafe>, but at some point this flag became mandatory. Since we don't check for its existence, users can give us an app without it and we will blindly accept it, causing the app to fail when AppScale tries to start up the Java AppServer.

Need to check for this tag, ensure that it's value is either true or false, and abort if it is not.

Root password

When running "appscale up" or "appscale-run-instances" for the first time, the SSH key is copied to the machine. However, if the user has not set a root password before doing so, he/she is unable to complete the process. Users must be instructed to create a root password before deploying AppScale.

Support "service" keyword in app.yaml

At the moment, we just support the "module" keyword.

Error Message Misleading

On line 424 of ec2_agent.py, the script checks to see if the given keyname is being used by any terminated instances. However, the error messages does not say that. It says to check SSH/"Key Pairs" screen an delete the existing key. Actual message:

'SSH keyname {0} is already registered. Please change the "keyname" specified in your AppScalefile to a different value, or erase it to have one generated for you.'

There are three problems with that error message. 1) It's not correct and 2) the script creates that Key Pair earlier in the process which made me keep deleting it over and over again when the real issue is that a terminated instance still had the same key. 3) The startup process has automatically hardcoded that "random" keyname at the end of the AppScalefile without telling the user that it has done so.

Here's what I suggest instead for a message:

SSH keyname {0} is already registered to a terminated instance. Either wait for the terminated instance to be removed or remove the keyname that was automatically added to the end of your AppScalefile.

testing trello connection