Coder Social home page Coder Social logo

wttech / gradle-environment-plugin Goto Github PK

View Code? Open in Web Editor NEW
4.0 10.0 0.0 714 KB

Provides seamless Gradle integration with Docker & Compose/Swarm for creating local development environments in a fully automated manner.

Home Page: https://tech.cognifide.com

License: Apache License 2.0

Kotlin 95.42% Shell 4.58%
docker docker-compose docker-image docker-container swarm health-checks docker-gradle gradle gradle-plugin gradle-plugin-kotlin

gradle-environment-plugin's Introduction

WTT logo

Apache License, Version 2.0, January 2004

Gradle Environment Plugin

About

Provides seamless Gradle integration with Docker & Compose/Swarm. Hides many OS-specific glitches from unexperienced Docker users.

Controls virtualized environment consisting of any Docker-compatible software (e.g Apache Web Server (HTTPD), SQL/noSQL databases). Such software is deployed as Docker Containers orchestrated by Compose or Swarm. This solution is cross-platform just like Docker.

Provides environment related tasks: environmentUp, environmentDev, environmentHosts etc.

Setup

As of plugin is published at official Gradle Plugin Portal, see instructions from there.

Prerequisites

Most of the configuration steps are automated. However, there are three manual steps to make this setup fully operating:

Notice for Docker on Windows

Ensure having shared folders enabled via GUI:

Docker Desktop Shared Drive

Please note that after each Windows credentials change (e.g forced by company policy), it is required to click button "Reset credentials" to ensure having shared drives working correctly. When running task environmentUp first time, Windows User Account Control (UAC) may require user confirmation to allow virtualized container to access local configuration files stored on shared drives.

Notice for Docker Toolbox

While using Docker Toolbox, beware that, by default, there is only one shared folder in VirtualBox configured which mounts C:/Users to /c/users on Docker Machine / Ubuntu. As a consequence, if project is located outside of C:/Users directory, there is a need to manually add corresponding shared folder and mount it on Docker Machine using VirtualBox GUI.

Configuration

Docker compose file

To configure containers to be deployed using Docker Compose (by default) or Swarm, simply create standard Docker Compose file but with templating (file at path src/environment/docker-compose.yml.peb)

version: "3"
services:
  httpd:
    image: centos/httpd:latest
    command: ["tail", "-f", "--retry", "/usr/local/apache2/logs/error.log"]
    ports:
      - "80:80"
    volumes:
      - "{{ rootPath }}/env/src/environment/httpd/conf/httpd.conf:/etc/httpd/conf/httpd.conf"
      - "{{ rootPath }}/app/aem/dispatcher/src/conf.d:/etc/httpd/conf.d"
      - "{{ rootPath }}/app/aem/dispatcher/src/conf.dispatcher.d:/etc/httpd/conf.dispatcher.d"
      - "{{ workPath }}/httpd/modules/mod_dispatcher.so:/etc/httpd/modules/mod_dispatcher.so"
      - "{{ workPath }}/httpd/logs:/etc/httpd/logs"
      {% if docker.runtime.safeVolumes %}
      - "{{ workPath }}/httpd/cache:/var/www/localhost/cache"
      - "{{ workPath }}/httpd/htdocs:/var/www/localhost/htdocs"
      {% endif %}
    {% if docker.runtime.hostInternalIpMissing %}
    extra_hosts:
      - "host.docker.internal:{{ docker.runtime.hostInternalIp }}"
    {% endif %}

Docker has differences in case of how it works on different operating systems. While using templating and variables in that file, Gradle Environment Plugin tries to mitigate these differences, to make your build working smoothly on all OSes and hide glitches from unexperienced Docker users. For example, when using Docker Toolbox, paths should be absolute and having Unix separators despite that host OS is Windows.

Example above comes from that file which is setting up Apache HTTPD for Adobe Experience Manager.

DSL options

Moreover, plugin comes with rich DSL which allows to:

  • run callbacks on each container lifecycle event (resolve, up, reload, dev),
  • configure where container resources are stored,
  • configure health checks to be able to detect moment in build lifecycle when are container services are stable and be ready to perform next build tasks using these services,
  • configure hosts which should be appended to OS-specific hosts file.
environment { // config for AEM environment running on Docker
    rootDir.set(prop.string("environment.rootDir")) // project.file(".environment")
    hosts { // domains to be appended to hosts file automatically
        "http://example.com" { tag("publish") }
        "http://locahost:4502" { tag("author") }
    }
    docker { // Docker specific configuration
        containers {
            "httpd" { // control container described in 'docker-compose.yml.peb'
                resolve {
                    resolveFiles {
                        download("http://download.macromedia.com/dispatcher/download/dispatcher-apache2.4-linux-x86_64-4.3.2.tar.gz").use {
                            copyArchiveFile(it, "**/dispatcher-apache*.so", file("modules/mod_dispatcher.so"))
                        }
                    }
                    ensureDir("cache", "logs")
                }
                up {
                    ensureDir("/usr/local/apache2/logs", "/opt/aem/dispatcher/cache/content/example/demo", "/opt/aem/dispatcher/cache/content/example/live")
                    execShell("Starting HTTPD server", "/usr/local/apache2/bin/httpd -k start")
                }
                reload {
                    cleanDir("/opt/aem/dispatcher/cache/content/example/demo", "/opt/aem/dispatcher/cache/content/example/live")
                    execShell("Restarting HTTPD server", "/usr/local/apache2/bin/httpd -k restart")
                }
                dev {
                    watchSourceDir("conf")
                }
            }
        }                        
    }
    healthChecks { // checks (e.g GET requests) verifying running Docker containers like HTTPD
          http("http://example.com/en-us.html", "Live site", "English US")
          http("http://author.example.com/sites.html", "Author module 'Sites'") {
              containsText("Sites")
              options { basicCredentials = authorInstance.credentials }
          }
    }
}

Health checks

In case of the dispatcher it takes few seconds to start. Service health checks could be described by following configuration. For example, to wait for all domains to be available and having particular text in response, write:

environment {
    healthChecks {
        http("http://example.com/en-us.html", "Live site", "English US")
    }
}

These checks are performed automatically after each file change applied when running task environmentDev or on demand when running task environmentAwait.

Task environmentUp

Turns on local environment.

Environment up task

Task environmentDown

Turns off local environment.

Environment down task

Task environmentDev

Allows to listen for Apache Web Server / Dispatcher configuration files changed and then automatically reload HTTP service.

Workflow:

  1. Run command gradlew environmentDev,
  2. Edit files located in src/environment/httpd/conf ,
  3. Notice that HTTPD service should be restarted automatically after file changes,
  4. Check results of health checks,
  5. Optionally, check:
    • HTTPD logs located at path .environment/httpd/logs

Environment dev task

Task environmentAwait

Performs environment service health checks on demand.

Task environmentReload

Reloads virtualized environment by reloading all Docker containers (e.g. removing Dispatcher cache files then restarting HTTPD server).

Task environmentHosts

Amends hosts entries in OS-specific file.

Example configuration:

environment {
    hosts { // domains to be appended to hosts file automatically
        "example.com" { tag("live site") }
        "demo.example.com" { tag("demo site") }
        "cms" { tag("cms authoring") }
    }
}

The task will run on Windows & Mac OS interactive process asking for administrative/sudo permissions to be able to edit the hosts file. On other OSes / Unix-based systems, the task will print a path to the generated script to be executed with sudo rights but in a non-interactive way (to be done separately right after this task execution.

Options:

  • hostUpdater.enabled=false - allows to disable temporarily host amendmend (e.g on CI/CD),
  • hostUpdater.interactive=false - potentially useful when interactive mode does not work for unknown/OS-specific reason,
  • hostUpdater.force=true - by default, updater is lazy - it will not ask for administrative/sudo mode if the hosts file does not need to be updated.

License

Gradle Environment Plugin is licensed under the Apache License, Version 2.0 (the "License")

gradle-environment-plugin's People

Contributors

github-actions[bot] avatar jaroslaw-pietraszek-wttech avatar krystian-panek-wttech avatar mierzwid avatar pun-ky avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gradle-environment-plugin's Issues

Init callback option

I want to execute some script inside the container (something like service CLI). It is the common pattern for sample data, installing some additional extensions etc. I understand that resolve executes before the container starts.

Right now, we have:

resolve, up, reload, dev

I would imagine something like

resolve, init, up, reload, dev

Example:

"magento" {
  init {
    execShell("Install sampledata", "/usr/sbin/install-sampledata")
  }
}

Hosts jar execution error

java -jar /Users/xxx.yyy/Documents/Projects/foo/gapenv/.gradle/environment/hosts/hosts.jar

works but environmentHosts not:  

/Users/xxx.yyy/Documents/Projects/foo/gapenv/.gradle/environment/hosts/hosts.sh
Successfully started process 'command 'sh''
0:297: execution error: Error: Unable to access jarfile /Users/xxx.yyy/Documents/Projects/foo/gapenv/.gradle/environment/hosts/hosts.jar (1)
> Task :environmentHosts FAILED

Await all container services

ContainerManager stores as a list of defined services that should match these defined in docker-compose.yml.peb.

However, if some of the services are not defined in ContainerManager, then the awaiting container step is not being performed.

It would be nice if ContainerManager will discover services deployed to the stack and complete the list by defining not defined by hand (services which are having defined lifecycle callbacks). The order of predefined services should be preserved.

Easy file linking

how about Container.linkFile() which will do execShell("ln -s source $target")

Support Kubernetes

Right now when running envUp Docker Swarm (docker stack deploy) is being used and deploying containers defined in docker-compose.yml.peb.

The extra option could be to have the capability to create files like pod.yml.peb and run them using Kubernetes (kubectl apply)

Await all container services

ContainerManager stores as a list of defined services that should match these defined in docker-compose.yml.peb.

However, if some of the services are not defined in ContainerManager, then the awaiting container step is not being performed.

It would be nice if ContainerManager will discover services deployed to the stack and complete the list by defining not defined by hand (services which are having defined lifecycle callbacks). The order of predefined services should be preserved.

Embed hosts scripts in plugin

e.g unpack scripts then run them to be able to avoid running them externally by our own

imagine 

cli: sh gradlewenvironmentHost

kotlin:

  • generate host entries near file hosts.[bat|sh] e.g hosts.txt
  • call hosts.[bat|sh], script should ask sudo/super-user interactively

Unable to check docker network availability

Tech stack:

  • Ubuntu 20.04.5 LTS ARM
  • Docker version 23.0.0
  • Gradle environment plugin - any version

Behaviour:
NetworkAvailability check in Stack.kt uses hardcoded "Error: No such network" docker error string as a sign of non existing network. In latest Docker the the message returned in such case is "Error response from daemon: network <network_name> not found" - result of command: docker network inspect <network_name>
Because of this, network availability check throws an exception instead of false in case network is not there.

environmentUp & Docker Desktop 4.8.1 (78998) issue

After updating the docker to the latest version, there was a problem with turning on the dispatcher.

The reason for this error is that the docker compose -p gat-aem down --remove-orphans function now returns error code 1, not 0

Remove orphans flag

time="2021-09-13T09:58:59+02:00" level=warning msg="Found orphan containers ([gat-aem_httpd_1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up."

Use Gradle properties in docker-compose.yml.peb

Hello,

In docker-compose.yml.peb I can use values from docker, system and env. I would like to also be able to use Gradle properties, e.g. from ~/.gradle/gradle.properties.

It turns out that it is possible to use {{docker.environment.project.properties['yourprop']}} (thanks, @pun-ky !). A nice improvement would be to have them accessible with shorter path.

Consume Docker image extracted by GAP

  • use docker load to load AEM dispatcher Docker image provided by GAP - wttech/gradle-aem-plugin#775 (comment) ; do it once if running it all the time on envUp will be slow
  • test if the loaded image could be used in docker-compoer.yml.peb
  • note that GEP is generic purpose plugin, not only AEM related, so Docker image loading mechanism need to be AEM agnostic / universal

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.