phase2 / grunt-drupal-tasks

Grunt-based build and testing tasks for Drupal

Grunt Drupal Tasks

A Grunt plugin to automate Drupal 7 and Drupal 8 build and testing tasks.

Features

This project is built on the tools of the Grunt community to provide scripted automation of a number of PHP & Drupal tasks. Here are a few examples of what it provides:

• Configurable code structure that defaults to a clean development practice.
• Composer build workflow (or Drush make-based build workflow for Drupal 7.x).
• Optional use Code Quality & Static Analysis, and Frontend tooling to extend the build process.
• Behat and SimpleTest Testing
• Deployment packaging
• Git Hook management
• Desktop Notifications
• Local Development Friendly
• CI Friendly

We are continuously working to improve this toolchain, adding functionality that we see as common to our continuous integration and everyday development practices.

Requirements

For requirements, installation, use, and customization instructions, see the documentation.

grunt-drupal-tasks's Issues

[META] Add performance testing

We should add performance testing options. http://rupl.github.io/frontend-ops/#/7 points out a few solid options (scroll down).

• yslow: No API key needed, graded score.
• phantomas: Track performance trends.
• grunt-perfbudget: Lots of great data, and useful to specify performance budgets. http://timkadlec.com/2014/05/performance-budgeting-with-grunt/

I'm a little hazy on what I'm suggesting as far as tasks, maybe a grunt test:performance task run as part of the analyze task. The various performance test mechanisms can use a shared configuration of URLs to ping with the tests.

"Jenkins integration" for reporting on this may be tricky, I suspect we'll want to embed results in the job description unless we can dig up an output format Jenkins can interpret.

Automatically report on dependency status

https://david-dm.org/ is a tool to automatically check the status of dependencies. This will give us a means of knowing if our dependencies move on without us.

This is a good opportunity for someone to jump in and contribute with limited time to orient on internals of the project.

Add support to for popup notifications

When --notify is used, lets have messages about errors and completion of tasks.

https://github.com/dylang/grunt-notify

Stop using the grunt-parallel-behat fork

Let's stop using my fork of grunt-parallel-behat by getting fixes/improvements in upstream.

I've already contributed several changes:
linusnorton/grunt-parallel-behat#6
linusnorton/grunt-parallel-behat#7
linusnorton/grunt-parallel-behat#8

However, it looks like an earlier upstream change broke support for config options, especially specifying the Behat feature files through the "src" option. This needs to be fixed before we change to use the main version.

Create a facility to support multiple environments

During the primary build of a project there are often multiple domains basepaths. Development, QA, Demo, Production, Feature-branch instances, and so on. Any of these might need meaningful behat and other sorts of testing processes.

We need a method to make the basepath adapative or swappable via configuration. Making basepath a default configuration with overrides from command-line might be the most natural, for example: grunt --env=grayside.example.com or GRUNT_BASEPATH=grayside.example.com. The latter leverages environment parameters, which may allow for automating how this happens without explicit developer action.

Document a configured CI build in Jenkins

This may be the snippet from the shell script with some light configuration instructions, or an example job export.

Get "watch" to work for theme compilation (Compass)

May be resolved by #22 and #48.

Clarify requirement for Composer

Composer is required to use the example, as it has a composer.json file, which will trigger composer install as part of the default build process.

Also, update the Travis CI process to install composer rather than removing the composer.json.

Improve the list of available grunt tasks

We are using many grunt tasks, in many different ways. Many of these tasks are distracting for developers trying to see their core available options. As far as I can tell there is no way to take over grunt -h, but we can use grunt-available-tasks to create an alternate help listing for our key operations. I have tested this out on a project and it works well.

• This plugin wants to operate top-down. Our system tries to be flexible, unlike a specific project, so we need to preprocess it's configuration to source it bottom up (see proposal below).
• Consider how we get some custom documentation (to emphasize specific flags) in addition to a list of tasks.
• Rename available-tasks to something simpler, such as tasks or help (using grunt.task.renameTask())

Bottom-up Configuration

As mentioned above, one of the weaknesses of this plugin for our purposes is it's top-down configuration. However, we can take care of that by having our own configuration structure, something like:

task: {
  target: {
    display: true
    description: "A substitute description",
    group: "Drupal Utilities"
  }
}

This configuration can be set by individual project Gruntconfig.json files, but more importantly we can use grunt.config.set() in each individual task to setup it's participation in the task listing. By default we can leave the task hidden.

Add option for dependency paths

Add Gruntconfig.json option for the executable path for dependent programs/scripts, like drush, phpcs, phpmd, behat, etc.

By default, GDT should default to the path where the utilities are installed by Composer.

[META] Facilitate operational Drupal scripts

"Operational" scripts such as installation, update, and search indexing of a Drupal site are some of the mostly commonly run tasks after the main build process, and so far we've artfully dodged any acknowledgement of them in this toolchain.

Benefits of a Grunt Connection

• grunt help would be a central list of all standardized practice CLI tools for a project.
• --timer and other instrumentation built into grunt would be applicable.
• Easier integration with other sets of tasks, such as a single command to build & install, or a watch task that re-indexes search.
• Potential standardized interface for common development tasks across projects.
• Detection of a default drush alias (depending on implementation, could also be done in Bash)

How Scripts are Implemented

Since we have the goal of keeping GDT (and node in general) from becoming a production requirement, I don't think we should re-implement these sorts of operations as grunt tasks. Instead, for general portability I think these should be open-ended shell scripts for ease of implementing any given procedure.

By wrapping some simple scripts in a general grunt-shell wrapper, we can produce some default scripts and lean on the Drush alias for some of the basic customization. Longer-term, I think we'll want to use #35 (yeoman generator) to initialize scripts and possibly a drush alias file for the project in a way that can be customized later.

Example Install Script

Command

grunt install

bin/install.sh pseudo-code

drush $ALIAS site-install -yv
drush $ALIAS uli

Next Steps

• Add a bin/ directory to the examples folder with a basic install and update script
• Create a bootstrap script that sets up some common parameters and auto-detects a Drush alias
  • Uses the parent directory name of the code repository as a default alias. So if grayside.example.com/code-repo/Gruntfile.js is the path to the gruntfile, the alias assumed would be @grayside.example.com.
• Add a task to load the contents of a bin/ directory as grunt tasks. This should include default descriptions for all tasks loaded this way, with a general fallback for tasks not explicitly described.

PHPCS should output to terminal on validate, generate report on analyze

The current integration with PHPCS generates a report, and never denies a build. This means it is up to Jenkins reporting to point out problems with coding style. Reports like that are generally not useful to a developer's hourly workflow, but in-line reporting is something else.

Let's refocus the analyze task on generating reports, and the validate task on (time-efficient) terminal output.

To that end I've begun examining SaschaGalley/grunt-phpcs#20, but that PR seems broken in several ways.

Add installation instructions

Be explicit about downloading/installing Node.js/npm and installing the global grunt-cli package.

Relates to #9.

Bump the concurrency for drush make

I.e., include something like --concurrency=16 as an arg for drush make. Is there any reason we can't do that?

This is a drastic speed increase obviously but I wasn't sure if it was left out for a reason. For example, in the past I had heard that concurrency can create race conditions in cases where different make files specify different versions of modules (such as when you override the version of something that Panopoly ships with from a custom make file), but I thought that had been worked out now.

`grunt help` should only document user-facing operations

By using grunt.task.registerTask() to alias any tasks we want to make user facing, we can configure the overriding behavior of the availabletasks plugin to only show user-facing tasks. This well allow us to obfuscate direct plugin configuration somewhat, and mask targets that don't need to be user facing.

This does mean that certain user-facing behaviors we may not want to wrap in a task need some further documentation, such as grunt phpcs:full.

Remove timestamp from `grunt package` tarball name

Since that filename is not in active memory for whatever command does the deployment, we need something more predictable. If older tarballs have value to keep around, we'll circle back and add a logrotate style mechanism based on cdate.

Create Yeoman generator for project set up

Let's explore options to spin off the current example template into a more flexible and interactive starting point using grunt-init or Yeoman.

Use Flow for JS Type Checking

Flow http://flowtype.org/ is an intersting tool that does Javascript type checking. This can help find a lot of errors that are hard to spot initially. It does not require you to do anything different to your code than add a comment at the top. Eventually it can support transitioning to a more explicit typing.

Drupal 8 test should run tests on custom modules

With the adoption of PHPUnit in Drupal 8, hopefully more projects will write unit tests as part of development.

I think the goal would be to take our current grunt test, which is just an alias for grunt behat, and transform it such that

• grunt test:drupal runs the run-tests.sh script (core's test runner)
• grunt test:behat would do what grunt behat does
• grunt test would run all tests.

Drupal 8 support and task/config refactor

Grunt Drupal Tasks should be able to support either Drupal 7 or 8 and include a kickstart example for both.

Ideally the differences for 7 and 8 can be limited to configuration settings in Gruntconfig.json, to changes (if at all necessary) to the build sequence in the example's Gruntfile.js, as well as to updated dependencies in the composer.json.

To get there, we'll need a refactor of some tasks to work more dynamically based on configuration settings.

For example, the symlink task includes individual steps for the symlink for modules, profiles, sites, and themes. There are some variations in how the symlinks are created for each, which may need to be preserved. However, the source and destinations could be configurable in Gruntconfig.json.

Write comprehensive configuration documentation

There's getting to be too much to just dump it into the middle of the README. The current example/Gruntconfig.json is trying to be a template for a minimal implementation, it doesn't explain the configuration it's using, nor does it include a lot of what I would consider "standard" for our projects.

Document the directory structure and other assumptions

The tasks (and especially the included example) make assumptions about how Drupal project files are organized. Explain the approach and its benefits.

Eliminate load - order significance from help task

Current implementation pulls in configuration at time of plugin load. As a result any help configuration set after tasks/zzz_help is loaded does not work.

To support effective extensibility this limitation needs to be removed. The recently released gruntconfig.json option still creates a dependency on project config that hinders gdt-aware plugins from participating in help independently.

I've pushed a branch that creates an api to add config directly but it's only working when accessed from bootstrap.js.

Support .sass files in watch:compile-theme file scanner

watch:compile-theme is configured to check only .scss files. The older syntax style for Sass used .sass.

Follow-up to #108

Document global CLI options in `grunt help`

It's not clear how to add help text for CLI options, like --timer, in the grunt help display powered by grunt-available-tasks.

We need to explore how to document options like this in the help text displayed.

Add support for drush coder-review

If it does not require a module to be enabled simply to run, let's see about using coder review. Between coder review & phpcs, we can automate most of the nit-picky parts of a code review.

http://cgit.drupalcode.org/coder/tree/coder_review

This also seems to have overlap with #23, though the site-audit functionality is not an implementation of Drupal standards but rather a bundling of current best practice checkers.

This should be folded into grunt analyze.

Add examples of overriding task config

Example: https://gist.github.com/arithmetric/bd0af47ca33281b1db3d

Use different JSON parser for improved compass performance.

Even if the performance is not improved, it may be worth it to stop staring at this error every time compass runs.

[WARNING] MultiJson is using the default adapter (ok_json).We recommend loading a different JSON library to improve performance.

Add support for CSS/SASS and JS lint

Help wanted to identify the best tools and sensible defaults for the lint settings.

CSS:
https://github.com/gruntjs/grunt-contrib-csslint
http://csslint.net/

SASS:
https://github.com/ahmednuaman/grunt-scss-lint
https://github.com/causes/scss-lint

JS:
https://github.com/gruntjs/grunt-contrib-jshint
https://github.com/stephenmathieson/grunt-jslint
https://github.com/reid/node-jslint
https://github.com/jshint/jshint

Add support for deployment tooling

When writing the package task, the intent was to support a swappable deployment mechanism. Our two base cases being a fairly straightforward Capistrano rsync method and a pantheon/acquia friendly docroot commit to ops repo.

Nationally, these would be driven from grunt-shell tasks.

Also worth considering ShipIt

default.settings.php unavailable

Since the drush make task runs before the clean task, on a brand new install there is no default.settings.php or default.services.yml (in D8) for drush si to copy.

I wonder if a step after make but before clean could be added to copy those into src/sites/default if they do not yet exist?

To reproduce:

grunt
drush si -y -db-url=mysql://example:example@localhost/example -y

and the error:

Sites directory /.../src/sites/default already exists - proceeding.                                                  [notice]
copy(sites/default/default.settings.php): failed to open stream: No such file or directory drush.inc:695                                                                  [warning]
Failed to copy sites/default/default.settings.php to /.../src/sites/default/settings.php

Update build tools in the example composer.json

We're using older versions of many build tools, notably the Drupal Extension and Behat.

Test of build process broken for phpcs

From the travis console:

Running "phpcs:validate" (phpcs) task

Starting phpcs (target: validate) in src/**/*.css src/**/*.php src/**/*.module src/**/*.inc src/**/*.install src/**/*.profile !src/sites/** !src/**/*.box.inc !src/**/*.features.*inc !src/**/*.pages_default.inc !src/**/*.panelizer.inc !src/**/*.strongarm.inc

ERROR: You must supply at least one file or directory to process.

A fix would require dropping at least one PHP file into the working copy for the test.

Use "minimal Gruntconfig" as implicit defaults

Rather than universally mandate basic configuration, let's handle that implicitly and come as close to zero-configuration as possible. All we need to do is pull srcPaths and buildPaths into the codebase as fallback settings.

This will require adjustments to the documentation.

Trying to load other Grunt tasks fails because grunt-drupal-tasks monkeypatches grunt.loadNpmTasks()

I ran into a situation today where I tried to load another task from an npm module that put into my package.json file. Since grunt-drupal-tasks monkeypatches grunt.loadNpmTasks(), I could not load the tasks in question.

To work around this, it is necessary to either run grunt.loadNpmTasks() before the require() statement that loads grunt-drupal-tasks, or to run grunt._loadNpmTasks() to load your task if it comes after that require statement.

@mikecrittenden and I had discussed two possible solutions:

• Fall back and try to load tasks from the top-level node_modules directory after trying the initial load from the grunt-drupal-tasks node_modules directory.
• Detect errors in our monkeypatched loadNpmTasks() so that someone who hits this error will know what's going on.

PHPCS is loud about a lot of relatively minor issues

Looking at https://github.com/SaschaGalley/grunt-phpcs, there are a few out-of-box options we may be able to use to tune down the volume slightly.

Fix issues with shared plugin dependencies

@timcosgrove reports an issue where if a project extending grunt-drupal-tasks includes a plugin that is also included by grunt-drupal-tasks (i.e. grunt-shell), then "npm install" only installs a copy of the plugin in the project's node_modules and not in grunt-drupal-tasks/node_modules. This causes errors for grunt-drupal-tasks tasks that use the shared plugin.

Fix grunt-drush support

The drush make process is now broken, as shown in recent Travis CI builds (and verified locally):
https://travis-ci.org/phase2/grunt-drupal-tasks/builds/46159472

I think this is due to upstream changes in grunt-drush. See also #106.

Add drush site-audit to analyze task

Investigate addition of Site Audit to the analyze task.

I expect we'll want plain text at minimum, html for generated report files, possibly leverage JSON output if we want fail conditions.

This may look like integration with a separated grunt-drupal-site-audit plugin.

Behat and other tests should fail the Grunt jobs

Ensure that a Behat test fail is registered as a Grunt task fail. This should use Grunt's default behavior, which will prevent subsequent tasks unless the --force option was given.

`grunt package` produces a tarball without the Drupal .htaccess file

Jobs that fail on watch block other watch tasks from executing.

Using the grunt-parallel plugin will allow us to treat each of the watch tasks as a distinct job. Randall has some prior art on this that we can port.

Resolve Behat test issues on Travis CI

Behat tests fail on Travis CI with the following error:

Fatal error: Cannot inherit previously-inherited constant BEFORE in /home/travis/build/phase2/grunt-drupal-tasks/test/working_copy/vendor/drupal/drupal-extension/src/Drupal/DrupalExtension/Hook/Scope/BeforeUserCreateScope.php on line 13

For example, see: https://travis-ci.org/phase2/grunt-drupal-tasks/builds/45110439

The tests pass on a local env with similar configuration (PHP 5.4.30).

Add test for compass functionality

Let's add some test coverage that compass is working, which incidentally should cover bundleInstall.

Doing so will require:

• A Gemfile to pull in compass dependencies.
• A Drupal theme-like structure that will allow testing config.rb & sass compilation, but does not otherwise need to be functional.
• Code to set up the above two things as part of the working copy for testing.
• A mocha to look for a generated CSS file.

We do not want these changes built directly into the examples folder, as examples should remain available as a stripped down template for starting projects, and SASS functionality should not be part of that.

Use drush runserver to provide an env for testing, validation

Behat testing and some system validation tasks require an installed (database-backed) and hosted/accessible Drupal site.

Explore using drush runserver (and optionally a sqlite database) as an alternative to requiring manual set up of a local env.

Pass grunt verbose, debug, and no-write options into drush

This may be something to push into grunt-drush instead, but grunt has --debug, --verbose, and --no-write options that all have a parallel with drush, and developers using them would benefit from drush being similarly responsive.

Add dynamic concurrency level

Follow up from #78 and #94: We should add a universal concurrency setting that is applied to a number of tasks that can be run concurrently (drush make, Behat, etc.) and also identify a default value dynamically per @grayside's suggestion (based on grunt-concurent, see: https://github.com/sindresorhus/grunt-concurrent/blob/master/tasks/concurrent.js#L12).

Make the system testing-agnostic by wrapping behat

• Create a grunt test command that executes behat.
• Move features/ directory as a sub-directory of tests/ for a more generally intuitive name and ability to support multiple test systems with minimal clutter.

Add accessibility testing with Quail

@bblake suggests we check out Quail (https://github.com/quailjs/quail) to add accessibility testing.

Switch analyze task to use concurrent

The heavy static analysis operations of the analyze task can be done in parallel. Let's switch the analyze task alias to a wrapper on a concurrent:analyze task. Improves on top of #90