pantheon-systems / quicksilver-examples Goto Github PK

Currently the New Relic example is using New Relic's v1 API. Seems that doesn't work anymore. Has anyone had success with v2?

The documentation references a private directory, but it’s unclear where it goes. At the very bottom, the Troubleshooting sections says "pantheon.yml and private go in your site root, ie. your ~/code directory", but the problem is that private actually goes in your web root. So that’s either ~/code/ or ~/code/web/ if you have that set up differently (but pantheon.yml still goes in your ~/code/ root regardless). This needs to be cleaned up.

Issue Description:

The passthru usages will result in a false positive if the called command returns with non-zero exit code.

Suggested Resolution:

Check &$return_var value and raise an error to propagate the error to the calling quicksilver script.

$cmd = 'wp search-replace "://test-example.pantheonsite.io" "://example.com" --all-tables ';
passthru($cmd . ' 2>&1', $exit);
if (0 !== $exit) {
  trigger_error(sprintf('Command "%s" exit status: %s', $cmd, $exit), E_USER_ERROR);
}

Using the Solr index example it times out after 175 seconds. This is fine except no sign of an error is displayed to the user unless watching workflows with Terminus, which we can't reasonably expect folks to do. It would be great if we could surface an error operations in Quicksilver hooks time out. In this case the Solr index operation takes about 20 minutes.

terminus workflow:watch solr-power
 [notice] Watching workflows...
 [notice] Finished workflow c554cc42-0985-11e7-8ce6-bc764e1141f9 Create a Multidev environment () at 2017-03-15 13:51:22
 [notice] ------ Operation: Index Solr Power items after multidev creation finished in 175s (slow-query) ------
Operation exceeded maximum execution time of 120 seconds.

We have this in a lot of scripts:

header('Content-Type: text/plain; charset=UTF-8');

But we don't need it! :)

quicksilver-examples/db_sanitization/db_sanitization_drupal.php is named "Drupal" but it won't work for Drupal 8 because the password and such is in users_field_data, not users.

Please add an example of sanitizing a D8 database and mark the other Drupal script as being for D7.

Using the example slack_notifications.php, the commit ID and message weren't being passed on to Slack. It looks like $fields array wasn't merged properly. To resolve, I changed the following from line 78:

    $fields += array(
      array(
        'title' => 'Commit',
        'value' => rtrim($hash),
        'short' => 'true'
      ),
      array(
        'title' => 'Commit Message',
        'value' => $message,
        'short' => 'false'
      )
    );

To this:

    $fields[] = array(
      'title' => 'Commit',
      'value' => rtrim($hash),
      'short' => 'true'
    );
    $fields[] = array(
      'title' => 'Commit Message',
      'value' => $message,
      'short' => 'false'     
    );

Possibly the numeric keys generated by PHP for the second array start at 0 so are ignored when using +=.

I have used Loader.io in Quicksilver demos and should bring over my code to this repo. It also includes some Slack reporting.

The purpose of the script is to trigger a load test of Test or Live upon deployment.

Hello;

I don't know if it is an issue and I am unsure of here being the right place to ask for help (or pantheon support?).

Is there a way to hook, not on a git commit but on a git tag creation?

my workflow:

workflows:
sync_code:
after:
- type: webphp
description: Notification.
script: private/scripts/my_script.php

is working well when there is a commit. But nothing is happening for tag, sync_code does not seem to capture their creation.

Is there a way to do so?

Gerald

How should we organize this repository?
@newtoid @greg-1-anderson

I'm thinking:

Option 1

• pantheon.yml files/
• webphp files/
• script-type 2 files/

Option 2

Where full site examples make more sense together than apart, I think that the directories here should house sets of quicksilver stuff that all works together, grouped by type. Each example folder would house a set of scripts used for a site. It would result in redundant code, however...

• example type 1/
  • example 1-1/
  • example 1-2/
• example type 2/
  • example 2-1/
  • example 2-2/

I noticed while adding the Diffy Visual Regression Quicksilver script, that username is quite generic. Now, I know for myself, I can change that username secret key in the PHP file to something like diffy_username but I'm not sure if others get that. So, if the key username is used for all script examples, do you think any customers will take pause because they already may have that key used in their secrets file (and not realize it's trivial to update the key accessed in secrets.json)?

Related to #62, we tried to implement the "autoposting" solr schema using the create_cloud_development_environment event with no luck.

On our pantheon.yml file we have this:

api_version: 1

workflows:
  create_cloud_development_environment:
    after:
      - type: webphp
        description: Enables Solr backend
        script: private/quicksilver-scripts/configure_solr.php

Then, we have the configure_solr.php file, that contains the next code:

<?php
// Prevent this script to be executed out of pantheon.
if (!defined('PANTHEON_ENVIRONMENT') || (PANTHEON_ENVIRONMENT === 'live')) {
  return;
}

// Bootstrap Drupal using the same technique as is in index.php.
define('DRUPAL_ROOT', $_SERVER['DOCUMENT_ROOT']);
require_once DRUPAL_ROOT . '/includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_DATABASE);
$default_path = 'sites/all/modules/apachesolr/solr-conf/solr-3.x/schema.xml';
$schema_path = variable_get('pantheon_apachesolr_schema', $default_path);
pantheon_apachesolr_post_schema_exec($schema_path);

But this code is not working and I can not figure out why. Also, I tried to debug it following the steps mentioned in the troubleshooting part of the quicksilver documentation, it says to use terminus workflow:list to get the logs. I just created a new multidev environment and it does not even appear on the list of workflows, so is not possible to get the logs.

I've been doing some profiling of PHP 5.6 vs. PHP 7 and in doing so, I've been creating a lot of Multidev environments.

As I do, I'm reminded that the Apdex T value and alert policy is unique for each environment. If there isn't a way to set default values in New Relic for new Multidevs, it should be fairly trivial to do so with with the REST API called from Quicksilver.

For example:

curl -X PUT 'https://api.newrelic.com/v2/applications/123456789.json' \
     -H 'X-Api-Key:12344567890QWERQASDFASDFZXCV' -i \
     -H 'Content-Type: application/json' \
     -d \
'{
  "application": {
    "name": "string",
    "settings": {
      "app_apdex_threshold": "1",
      "end_user_apdex_threshold": "3",
      "enable_real_user_monitoring": "boolean"
    }
  }
}' 

@ari-gold, @joshkoenig, @ataylorme, Should this be done some other way? If not, I'll make a PR to add this example.

Having the environment url and protocol in the payload would be nice.

Use case: post-deployment url status checker (which could also serve as a cache warmer)

An array of URLs are defined to be checked with Guzzle for returning status 200 after deployment.

It would be nice if the base environment url was available, including protocol if the site is secured (for sites that are setup to restrict requests to https).

If this were the case, the script would only need one set of uris defined, which appended a base url, for status checking across multiple environments.

Now that CloudFlare has opened up their API for dynamic cache purging:

https://api.cloudflare.com/#zone-purge-all-files

We should do an example that shows how to integrate this with the cache_clear workflow for folks who want to be sure they are able to get fresh content out asap.

Slack Notifications one line copy/pasta command on the slack_notification/README.md should be updated to include:

• slack_channel
• slack_username
• always_show_text

Are any environment variables available to our scripts? Specifically I would like to know which environment the script is being run on (dev, test, live, or which multidev).

This might be the uri for the environment, e.g. test-mysite.pantheonsite.io

WordPress has a plugin analogous to Drupal's Features module: https://wordpress.org/plugins/wp-cfm/

Just as we have an example for deploying configuration via Features (https://github.com/pantheon-systems/quicksilver-examples/tree/master/drush_revert_features) we should have an example for WordPress CFM.

In this script:

https://github.com/pantheon-systems/quicksilver-examples/blob/master/db_sanitization/db_sanitization_drupal.php

I have a simplified version which can not seem to bootstrap drupal.

<?php
echo "\n\n";
echo "\n========= START DB POST ===========\n";
if (defined('PANTHEON_ENVIRONMENT') && ($_ENV['PANTHEON_ENVIRONMENT'] !== 'live')) {
  echo "\nON PANTHEON\n";
  define('DRUPAL_ROOT', $_SERVER['DOCUMENT_ROOT']);
  require_once DRUPAL_ROOT . '/includes/bootstrap.inc';
  echo "\nBEFORE BOOTSTRAPPING\n";
  drupal_bootstrap(DRUPAL_BOOTSTRAP_DATABASE);
  echo "\nBOOTSTRAPPED DRUPAL\n";
} else {
  echo "\nNOT ON PANTHEON\n";
}
echo "\n========= END DB POST ===========\n";
echo "\n\n";

And line by line I commented stuff out to see where it fails. Everything runs fine until after the line with the "BEFORE BOOSTRAPPING" message. But when the next line executes it fails with this message:

PANTHEON ERROR:

Expected status 200, got: 301 while running your Quicksilver operation.

Please check your script path is correct and try again.

Has anyone gotten the Quicksilver system to successfully boostrap drupal? If so how?

Thank you!

I've tested the diffy.website integration and though on paper it looks great in reality it's a big hassle and won't work as you would expect (https://github.com/pantheon-systems/quicksilver-examples/tree/main/diffy_visualregression)

A conceptual issue I found with it is that it should trigger after a deploy but also including a db clone. Otherwise the comparison doesn't make sense. test and live need to be copies of each other for this to be useful.

At the moment, the included example triggers on deploy.after - but this doesn't trigger when the deploy+database/file cloning is done. It triggers before that. This way deploys to test become a two step process: clone db and files, then deploy. It would be much more useful if this could be done just in one step and help to automate testing

It would be great to add a timeout to the curl request somewhere around here:
https://github.com/pantheon-systems/quicksilver-examples/blob/master/new_relic_deploy/new_relic_deploy.php#L63L69

As php does not count external processing toward php timeout, it will cause the php worker to hang in case New Relic fails to respond. This will also cause the Deploy workflow to be marked as failed on the site's Dashboard.
http://php.net/manual/en/info.configuration.php#ini.max-execution-time
http://php.net/manual/en/function.set-time-limit.php#refsect1-function.set-time-limit-notes

Internal support ticket reference:
120299

Is it possible to programmatically trigger a backup of the current environment? We want to fire backups when deploying.

It seems like a common task is going to be storing config data for the scripts. (credentials, service endpoints, etc...)

JSON configs are fine for static data, but it would be pretty nice for the scripts to have access to a site-level object store for both static and dynamic use.

Besides an easier way to store and retrieve script data, it would be nice if the data persisted at a global (site) level rather than an environment level. This way when a file system is blown out by a copy, the data would not be lost.

If a site level db service is out of the question, maybe access to write files at a site level directory would be sufficient. The author could maintain an sqlite db if they needed anything fancy. A site level service with a simple getter/setter wrapper functions available to the scripts would provide for a nice way to standardize though.

Is there an attribute that I can use to set the database version ?
ex:

database:
version: 10.4

Thanks

Hello,

I was trying to test out the provided slack notification quicksilver examples and was a bit confused as you speak about the repository and the private files repo at different locations.

Should I be storing the slack notification in:

{repo root}/private/scripts/slack_after_deploy.php?

Or do you mean for me to run the ftp command to deploy the php to the private files folder similar to the JSON with secret?

I"m adding this issue here on the advice of Zvi Epner at Pantheon support.

I have the following simple test script:

<?php
print 'QS test script';
passthru("wp search-replace https://www.marsdd.com http://db-clean-www-marsdd-com.pantheonsite.io —verbose 2>&1");

• It does not appear to run
• No output appears from it in the workflow log (even the first print statement)
• No failure states are mentioned in the workflow log
• When I remove the passthru statement the print output shows up in the workflow log

wp search-replace is quite a long running process. I suspect it is being killed by a timeout, or other resource limitation. A mention of it does happen in php slow log. Zvi was unable to confirm or deny this though, and I don't see any documentation of resource limitations for quicksilver anywhere.

It might be nothing to do with that though. It would be great if someone could look into this and give me a hint as to what might be going wrong.

Maybe we should add an OWNERS file that specifies who is responsible for maintaining this project, with a comment documenting the process for approvals.

An owner is required to merge, and they’ll want a +1 from another owner

When running a script before deployment starts, the correct deployment message should be displayed. Right now it is displaying the $annotation from the previous deployment.

Whether implementing a hook before or after a deployment, references to the $annotation should be consistent, right?

Hi,

After testing the slack notifications, they don't work when actions are executed on multidev with the pantheon.yml specified in the readme.md

This includes code sync, clear cache, creating new cloud environment.

Thanks,

Dominique

Hey,

Do you have any suggestions of how to post pantheon solr schema after cloning db?

Thanks ahead.

In file https://github.com/pantheon-systems/quicksilver-examples/blob/master/slack_notification/README.md, change "Add, and update as needed, the example slack_notification.php script to the private directory in the root of your site's codebase, that is under version control. Note this is a different private directory than where the secrets.json is stored."
For: "Add, and update as needed, the example slack_notification.php script to the private directory in the root of your site's codebase, that is under version control. It should be located under code/private/scripts/slack_notification.php."

Slack accepts full URLs for emoji icons in the slack attachments array, why not enrich the out of box experience with some Pantheon branding? Default can be set to a Pantheon logo URL instead of :lightning_cloud:

Ideal would be 128x128px off a CDN.

#foodforthought

Issue
The issue is that when you run the current version of the URL checker the value is always 1.

Solution
In function url_checker_build_output() it uses count($failed) as opposed to the actual $failed value.

I've created a patch, but it won't let me add it:

diff --git a/url_checker/url_checker.php b/url_checker/url_checker.php
index c58e51d..103d49d 100644
--- a/url_checker/url_checker.php
+++ b/url_checker/url_checker.php
@@ -83,7 +83,7 @@ function url_checker_build_output($results, $failed) {
   foreach ($results as $item) {
     $output .= '  ' . $item['status'] . ' - ' . $item['url'] . "\n";
   }
-  $output .= "--------\n" . count($failed) . " failed\n\n";
+  $output .= "--------\n" . $failed . " failed\n\n";
   return $output;
 }
 

I may be doing it wrong, but the sanitization example does not seem to be correct for resetting the password on Drupal 7.

When I use the statement:

db_query("UPDATE users SET pass = MD5(CONCAT('MILDSECRET', name));");

And then try to log in, it won't accept the password. The sanitation script references this blog post, http://crackingdrupal.com/blog/greggles/creating-sanitized-drupal-database-dump#comment-164, which is dated to 2010, a year before Drupal 7 was released.

Can the example be updated to handle Drupal 7 as well?

EDIT:

The real goal here is to have a way to do basic DB sanitization (email addresses and passwords) in D7. It doesn't matter if it uses Drush or some other way, but I want to be able to log in as [email protected] with password "password". (in other words, more or less the default behavior of drush sql-sanitize)

I expect that many Drupal 8 sites will want to automatically import configuration present in code upon deployments to live and test (and possibly database clones). I see this use case as the Drupal 8 equivalent of the "Features revert all" example that we have for Drupal 7: https://github.com/pantheon-systems/quicksilver-examples/tree/master/drush_revert_features

The code for this example might be as simple as a usage of drush cim.

There are a few good ones out there:

http://backtrac.io/documentation/rest-api

https://eyes.applitools.com/app/tutorial.html

It'd be awesome to kick off a test automatically when there's a new deploy into the test environment!

It appears the timeout for a script with multiple drush commands is 60 seconds. The README.md in one of the examples (config-import) says the timeout is 120 seconds. I would like to see some clarification in the docs about drush timeouts vs quicksilver timeouts.

If I change it to:
shell_exec("drush pm-list --format=json 2>&1");

I get:
Unknown option: --format. See drush help pm-list for available [error]
options. To suppress this error, add the option --strict=0.

If I use terminus to execute the command it works correctly, but using quicksilver it doesn't seem to work.

I'm trying to make sure our dev site always build from the live DB on each push. I was thinking to use terminus to get the latest live DB and import it in to the deb DB before deploying the lastest dev code but I'm not quite sure how to do that from the existing examples. Is it possible for one to be created?

While working on updating slack to use attachments, I saw behavior that is inconsistent with the following statement in the documentation:

"Likewise, if you are adding new operations or changing the script an operation will target, the deploy which contains those adjustments to pantheon.yml will not self-referentially exhibit the new behavior. Only subsequent deploys will be affected."

Just to clarify, updates made to the referenced scripts are executing for me on the first push, but the statement here refers to the registry of which scripts to run within pantheon.yml, and how they don't run self-referentially on the same commit, correct?

Or is it perhaps it's the fact that I've been working on a script that fires after deploy?

Hello,

My goal is to notify AWS SNS (the Notification Service) when something is pushed on our repo.

I deployed a php file, a file with some secrets and a pantheon.yml.

I had the :

Successfully applied pantheon.yml to the 'dev' environment.

Where to read the logs to determine why it is not working? Which file?

Thanks

Gerald

This would make a lot of sense, and we can do it with the deploy workflow. There's an existing PHP project to integrate GitLab that we might be able to borrow some API code from:

https://github.com/lesstif/gitlab-jira-integration

I noticed the example.pantheon.yml code starts with three hyphens. Not a show stopper, but I wonder how many people think those hyphens are needed.

thx

It would be great to performance test each deploy. http://developers.loadimpact.com/api/ will let us do this! Example wanted.

Getting the following notice after deploying code changes to Test env.

Undefined variable: deploy_tag in /srv/bindings/ed851365e8b24fbdb2c5e19bbe784cab/code/private/scripts/new_relic_deploy.php on line 51

It seems to be working in Live.

Any thoughts?

Once I had this properly configured, it looks like the WebHook doesn't think that the private channel is a real channel.

e.g.

Choose workflow: 1
+----------+--------------------------------------+
| Key      | Value                                |
+----------+--------------------------------------+
| Id       | e550e8f4-bfa4-11e5-8b9c-bc764e10d7c2 |
| Env      | dev                                  |
| Workflow | Sync code on "dev"                   |
| User     | Pantheon                             |
| Status   | finished                             |
| Time     | 26s                                  |
+----------+--------------------------------------+
+-------------+--------------------------------------+-----------+----------+
| Type        | Operation Description                | Result    | Duration |
+-------------+--------------------------------------+-----------+----------+
| platform    |                                      | succeeded | 0s       |
| platform    |                                      | succeeded | 16s      |
| quicksilver | post to slack after each code pushed | succeeded | 7s       |
+-------------+--------------------------------------+-----------+----------+

------ Operation: post to slack after each code pushed finished in 7s ------

==== Posting to Slack ====
RESULT: Invalid channel specified
===== Post Complete! =====

We do not have a private directory "in our code repo" on the dev env (at least not according to sftp). In your readme the mkdir command is mentioned in order to get the secrets file on the specific dev environment but that is in the sites/default/files directory and not the root directory.

On Acquia the private dir is above the htdocs dir so that it is outside of the repo on purpose. It seems like you are suggesting we have two private directories, namely:

• root/sites/default/files/private (which is where secrets.json file lives)
• root/private/scripts/ (which is where slack_notifications.php lives)

where root is the git root or (/srv/bindings/!@#!@#!@#!@#@!/)

Is that correct? If so, you should mention the necessary mkdir command in the README.md .

Thanks.

Need to remove Quicksilver script for SpotBot VRT as service appears to be inactive.

reference:
Spotbot.qa
https://twitter.com/spotbot_qa