automotivemastermind / condo Goto Github PK

By default, the dotnet-cli will create a local package cache for the dotnet core runtime and core libraries on first run after it is installed. On a hosted build agent where this is immediately thrown away, this makes the build take longer than necessary.

Condo should opt out of this experience by setting the following environment variable from both condo.sh and condo.ps1 before downloading and installing itself:

DOTNET_SKIP_FIRST_TIME_EXPERIENCE = true

Waffle.io is currently running under @keith9820 and it probably should be @fritz-build -- maybe we rename @fritz-build to something more generic?

Condo should support execution of E2E tests via detection of docker-compose.test.yml within the .docker folder. It should include the following docker-compose files in the execution, if present:

docker-compose.yaml
docker-compose.test.yaml
docker-compose.override.yaml

If any of the above are present and docker is already available on the build agent, then condo should build and run the containers without intervention.

Potential issues:

• Timeout if the container(s) fail to exit?
• Always set the --abort-on-exit flag for docker-compose

• - detect .net core rest apis
• - support IsOpenApi property in .csproj
• - generate [api-name].swagger.json file and place in docs root
• - update docfx template to include swagger generated docs
• - generate docfx toc.yml for services
• - append services to root level docfx toc.yml
• - ensure docfx creates docs from swagger files

Currently, any nukpg's created by condo are copied to a local cache which can be used as a nuget package source. This path is: ~/.nuget/localcache

This path can grow very large rapidly for projects that depend largely on nuget packages. Condo should introduce a target used to clean this path.

In addition; the fully-qualified local cache path should be reported in a summary at the end of a condo build for easy reference.

add support for executing condo as a docker container:

• derive a docker container from microsoft/dotnet/latest (Linux)
• derive a docker container from microsoft/dotnet/nanoserver (Windows)
• map /src within the container to $pwd
• copy precompiled copy of condo tasks to /condo within the containers
• create simplified condo init script for use within the container
  • use well-known paths /condo and /src
  • do NOT update condo within the container
  • remove support for sourcing condo from different paths, uris, and branches
  • do NOT install dotnet or the sdk within the container (will already exist)
  • exec msbuild against /src/condo.build with the condo targets path set appropriately
• update default condo.sh and condo.ps1 to execute condo within a container when available
  • detect docker-cli
  • pull and run latest condo container with mapped /src path
  • pass cli arguments through to the condo init script within the container

In the event that a release tag contains a pre-release version tag with the release branch build quality pointing to a commit that is on a different branch, the semver analysis prematurely incremented the build based on its own history rather than commits that are new as of the creation of the tag.

This effects builds that are created within the develop/feature branches while a release branch is still active and prior to the final release commit being merged back from the production (master) branch.

The resolution is to temporarily disable build increments until such tie as the final release commit makes its way back through the development branches.

When I changed the license from Apache 2.0 to MIT, I inadvertently wiped out the LICENSE file. This should be corrected using the exact some text as ./docs/about/license.md.

By default, when a nuget restore occurs, XML docs are extracted from those nuget packages that are referenced and pulled, which slows down the build. To increase build-time performance, condo should set the following environment to disable this behavior before installing itself from both condo.sh and condo.ps1:

NUGET_XMLDOC_MODE = skip

A release can be created with the vnext version and tag in the event that a build is manually queued outside of a PR.

Condo should detect gruntfile.* files within the project and automatically execute grunt tasks with the following names:

• - ci
• - build
• - condo

In order to do this, the following must be true:

• - detect gruntfile.*
• - detect npm
• - use npm to install grunt (if defined -- we will not install grunt manually)
• - use the grunt install to list available gulp tasks
• - execute one of the following grunt tasks, in ascending order of priority:
  • - condo
  • - ci
  • - build

If multiple matching supported grunt tasks exist, execute only one of them based on the the first match from the list above in order.

@keith9820 I saw that you added waffle.io to this project. I am wondering whether or not we need it given that GitHub added project support:

https://github.com/automotiveMastermind/condo/projects

What are the advantages of using waffle.io over this? I'm fine either way, I was just wondering. I've never used it. :)

... because windows.

Commits that precede the first tag are sometimes excluded from the git log used for parsing semver and changelog generation.

Analysis lead to the realization that git log --tags does not work as documented.

Build.GetPathsContaining() relies on Directory.EnumerateFiles(), whose implementation is case sensitive in mono and not case sensitive on Windows.

Since not all *nix systems are always case sensitive (OS X is case-insensitive by default), this can cause issues.

An alternative approach to deeply scanning a directory tree for a file pattern should be investigated to replace this implementation until such time as mono is file-system aware with HFS+ file systems to disable the case-sensitive matching.

Review and finalize condo documentation:

• add lifecycle docs
  • define lifecycle
  • define build target
  • define publish target
• add extensibility docs
  • condo.build
  • default targets
  • initial targets
  • extending existing targets
  • adding new targets
• add default goals docs
• add examples
  • add example for dotnet projects
  • add example for npm projects
  • add example for task runner projects (gulp, grunt)
  • add example for polymer-cli projects
  • add example for docfx generation

Now that the dotnet-cli is stabilized and dnvm / dnx is no longer supported post RC2, we need to update to the final bits and remove support for dnx/dnvm.

By default, the dotnet-cli will collect and send telemetry, which can slow down the build. This can be overridden with an environment variable DOTNET_CLI_TELEMETRY_OPTOUT=true, which condo should set in both condo.sh and condo.ps1 before installing itself.

The RecommendVersion task that was refactored currently requires the build quality to be specified, which is null/empty on the main production branch. This is causing issues building when a build quality is not available.

• update the vitals to reflect the new builds (travis, appveyor, circle-ci)
• update the gitter.im link
• update the CREDITS
  • add @sjk07 (contributor)
  • add @brandon-a-hay (contributor)
  • add @tiffanywang3 (contributor)
  • add @falquan (reviewer)
  • add @keith9820 (reviewer)
  • add @juri-miyamoto (reviewer)
• update "What is Condo" to reflect the current state
  • remove npm, bower, gulp, and grunt support
  • remove project.json support for condo >=2.0.0
  • only dotnet tests are discovered and executed (remove protractor, mocha, etc)
  • add docfx generation
    • reference prerequisites for docfx generation
      • windows
      • 2015 build tools (https://www.microsoft.com/en-us/download/details.aspx?id=48159) dotnet/docfx#1628
  • add github-pages publishing
• reference the project-json branch for backward support
  • support for project.json will be dropped entirely with the launch of .net standard 2.0

The CreateRelease target is not called on the master branch, which means that the changelog is not generated and the tag is not set.

Properties/Condo.AssemblyInfo.cs(15,32): warning CS7035: The specified version string does not conform to the recommended format - major.minor.build.revision [/app/src/AM.Example.Repository/AM.Example.Repository.csproj]

Currently, condo will execute "gulp" against any path that is discovered to contain a gulpfile.js, gulpfile.ts, or gulpfile.babel.js, which causes gulp to attempt to execute the default task.

Since a default task may or may not exist, we should call gulp --tasks-simple first to verify that the requested task exists and is available, emitting a warning if it is not (for anything other than the default task).

To make this easier, we should introduce a gulp_task argument with a default value of default.

So, if the gulp shade is called without a gulp_task argument:

• Only execute if the 'default' task is available.
• Do not emit any warning if no 'default' task is specified.

If the gulp shade is called with a gulp_task argument:

• Only execute if the task is available.
• Emit a warning indicating that the specified task does not exist.

Condo should detect gulpfile.* files within the project and automatically execute gulp tasks with the following names:

• - ci
• - build
• - condo

In order to do this, the following must be true:

• - detect gulpfile.*
• - detect npm
• - use npm to install gulp (if defined in package.json -- we will not install manually)
• - use the local gulp install to list available gulp tasks
• - execute one of the following gulp tasks, in ascending order of priority:
  • - condo
  • - ci
  • - build

If multiple matching supported gulp tasks exist, execute only one of them based on the the first match from the list above in order.

For now, I would like to get all of the shades supported by condo documented.

For each of the macros, I would like the following details:

• name
• description
• supported operating systems
• arguments
• global arguments (those not prefixed with the name of the shade)
• related shades (see also)

The documentation markdown file should be named after the shade (excluding the _), and placed in the docs/shades path within the gh-pages branch.

---
layout: docs
title: Update self
group: shades

---

Updates the build scripts located at the root of the project structure.

## Contents
* Will be replaced with the ToC, excluding the "Contents" header
{:toc}

## Supported Operating Systems

{% icon fa-apple fa-3x %} {% icon fa-windows fa-3x %} {% icon fa-linux fa-3x %}

## Arguments

This uses no arguments.

## Global Arguments

The following global arguments are used by bower:

<div class="table-responsive">
    <table class="table table-bordered table-striped">
    <thead>
        <tr>
            <th style="width:100px;">Name</th>
            <th style="width:50px;">Type</th>
            <th style="width:50px;">Default</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>base_path</td>
            <td>string</td>
            <td><code>$PWD</code></td>
            <td>The base path in which condo was executed.</td>
        </tr>
    </tbody>
    </table>
</div>

## Examples

Condo can update itself by adding the following call within any target:

{% highlight sh %}
    update-self
{% endhighlight %}

The default lifecycle includes a target aptly named `#update-self` that calls this shade.

## See Also

There are no related shades.

The following macros exist and need to be documented in the gh-pages branch:

• bower
• bower-download
• bower-install
• brew
• brew-download
• brew-install
• bundle
• bundle-download
• bundle-install
• dotnet-clean
• copy
• dotnet-cover
• dotnet-cover-report
• dnu
• dotnet-build
• dotnet-pack
• dotnet-restore
• dnu-locate
• dnx
• dotnet-run
• dotnet-test
• dotnet-watch
• dnx-watch-download
• exec
• gem
• gem-install
• gem-locate
• git
• git-config
• git-tag
• gm-download
• grunt
• grunt-download
• gulp
• gulp-download
• info
• info-collect
• log
• mkdir
• mocha
• mocha-download
• node
• node-download
• npm
• npm-install
• nuget
• nuget-pack
• nuget-push
• nuget-restore
• rimraf
• rimfraf-download
• ruby
• ruby-locate
• run
• tsd
• tsd-download
• update-self
• verify-hash

When building a pull request using a VSTS agent, it is currently not possible to process the automatic semver correctly since VSTS does not expose either the source or target branch of a pull request.

We would like to be able to process semver correctly within condo for the VSTS agent.

This is being tracked/blocked by:

microsoft/azure-pipelines-tasks#2147

Currently, some source control servers that enforce branch policies require a --no-ff merge to master, which impacts the ability to perform gitflow correctly.

The recommended solution is the following:

• when current branch is master
• when build is a server-side build (CI)
• when HEAD commit is a merge commit
• when merge commit has no diff to HEAD~1 commit
• when HEAD~1 contains NO_CI (versioned by condo)
• revert merge commit
• force push HEAD~1 commit as new HEAD

When trying to run unit tests with dotnet test ... --filter "Category!=Platform-Not-Linux&Category!=Agent-Not-CI" we get the following error in the VSTS build: Exception filtering tests: No tests matched the filter because it contains one or more properties that are not valid (Category, Category). Specify filter expression containing valid properties (DisplayName, FullyQualifiedName) and try again.

To solve this you need to specify a Purpose trait attribute on each Test class in your project, for example: [Purpose(PurposeType.Unit)]

1. I have installed yo and condo-generator on ubuntu 14.04
2. when i run the condo script file i get
   fail: dnvm-locate: could not find an instance of dnvm. The build process cannot continue, though there are instances of dnvm.

Running condo.build fails now if no property <GitHubPages> is set. Error is error MSB4100: Expected "$(GitHubPages)" to evaluate to a boolean instead of "", in condition " $(CI) AND $(CreateRelease) AND $(GitHubPages) AND $(DocsGenerated) ". [/opt/vsts/work/1/s/condo.build]. Solution is to set a default value for GitHubPages

Once #83 is complete, we should light up circle-ci as an option for executing builds. This should use the new circle-ci implementation that relies on docker containers whereby the condo docker container from the previously successful release is used to build condo itself as part of CI.

My current project is targeting both net461 and netstandard1.5 when trying to publish with condo i get the following error:

 "C:\Users\scott.AMM\projects\Business.DealCalculator\condo.build" (Publish target) (1) ->
       (DotNetPublish target) ->       

C:\Users\scott.AMM\AppData\Local\Microsoft\dotnet\sdk\1.0.1\Sdks\Microsoft.NET.Sdk\buildCrossTargeting\Microsoft.NET.Sdk.targets(31,5): error : The 'P
ublish' target is not supported without specifying a target framework. The current project targets multiple frameworks, please specify the framework for the pu
blished application. [C:\Users\scott.AMM\projects\Business.DealCalculator\src\Business.Domain\Business.Domain.csproj] [C:\Users\scott.AMM\projects\Business.Dea
lCalculator\condo.build]
         C:\Users\scott.AMM\.am\condo\.src\src\AM.Condo\Targets\Publish.targets(27,5): error MSB3073: The command "dotnet publish "Business.Domain.csproj" --ou
tput "C:\Users\scott.AMM\projects\Business.DealCalculator\artifacts\publish\\Business.Domain" --configuration "Debug" /p:Version=0.0.1-alpha-00107-1535 /p:Gene
rateAssemblyInfo=false" exited with code 1. [C:\Users\scott.AMM\projects\Business.DealCalculator\condo.build]

I don't see any elegant way except for looping over the target frameworks and publishing each with the -f flag. Im probably missing something magical... i hope

This is currently blocked by the following:

DotNetAnalyzers/StyleCopAnalyzers#2330
DotNetAnalyzers/StyleCopAnalyzers#2331

Once these are resolved, we will upgrade to either 1.0.1 or 1.1.0, depending on which one supports the headerDecoration setting for file headers.

Condo should be able to detect an npm run-script called "ci" or "condo" and, if present, execute the associated script by calling npm run.

In the event that a script is found and executed, condo should NOT check for a default gulp or grunt task in the associated project. NPM will take precedence over executing gulp or grunt directly.

Condo currently reads the SKIP_DOTNET_INSTALL environment variable to determine whether or not to skip this installation. In addition to this, a command line flag of --skip-dotnet-install (sh) and -SkipDotnetInstall (posh) should be added to set this at the CLI.

Condo should detect angular-cli configuration files and automatically execute the build and test targets.

Condo should automatically install the NodeJS and NPM dependencies in the event that a package.json is detected.

• - install npm on windows
• - install npm on macos
• - install npm on linux

A recent move to use dotnet-nuget-push directly rather than the NuGet xplat library in a custom task cannot work despite the documentation made available. The current CLI command does not support the --config-file argument even though it is in the examples.

We should move back to the custom task until this is resolved.

Related issue: dotnet/cli#6123

I found that when I try to use condo to generate a project on OS X, I followed the instructions and used Yeoman, i get the following error:

./build.sh: line 84: mono: command not found

It appears to be using "mono" to install Sake but mono isn't installed on the system. Doesn't the new DNX not rely on mono anymore?

Note the output below:

myuser@mac : ~/src/test/dnx/web-api
 ==> yo condo
? ==========================================================================
We're constantly looking for ways to make yo better! 
May we anonymously report usage statistics to improve the tool over time? 
More info: https://github.com/yeoman/insight & http://yeoman.io
========================================================================== Yes

     _-----_
    |       |    .--------------------------.
    |--(o)--|    |      Welcome to the      |
   `---------´   |     remarkable Condo     |
    ( _´U`_ )    |        generator!        |
    /___A___\    '--------------------------'
     |  ~  |     
   __'.___.'__   
 ´   `  |° ´ Y ` 

? Source folder name: ~/src/test/web-api/test001
? Test folder name: test
? Project name: test001
? Simple version: 0.0.1
? Company name: MethodCity Corporation
? Runtime: Core Common Language Runtime (CoreCLR)
   create build.cmd
   create build.ps1
   create build.sh
   create .gitignore
   create .gitattributes
   create .vscode/settings.json
   create global.json
   create make.shade
   create .editorconfig
   create .jshintrc


Your solution structure is now created. Use the following commands to get started.
     ./build.sh verify



myuser@mac : ~/src/test/dnx/web-api
 ==> ll
total 80
-rw-r--r--  1 myuser  staff   2.2K Sep 30 20:17 build.cmd
-rw-r--r--  1 myuser  staff    50B Sep 30 20:17 build.ps1
-rwxr-xr-x  1 myuser  staff   2.5K Sep 30 20:17 build.sh
-rw-r--r--  1 myuser  staff   135B Sep 30 20:17 global.json
-rw-r--r--  1 myuser  staff   390B Sep 30 20:17 make.shade
drwxr-xr-x  2 myuser  staff    68B Sep 30 20:17 test
drwxr-xr-x  3 myuser  staff   102B Sep 30 20:17 ~

myuser@mac : ~/src/test/dnx/web-api
 ==> la
total 160
drwxr-xr-x  15 myuser  staff   510 Sep 30 20:17 .
drwxr-xr-x   3 myuser  staff   102 Sep 30 20:15 ..
-rw-r--r--   1 myuser  staff   196 Sep 30 20:17 .editorconfig
-rw-r--r--   1 myuser  staff  2794 Sep 30 20:17 .gitattributes
-rw-r--r--   1 myuser  staff  1834 Sep 30 20:17 .gitignore
-rw-r--r--   1 myuser  staff   304 Sep 30 20:17 .jshintrc
drwxr-xr-x   3 myuser  staff   102 Sep 30 20:17 .vscode
-rw-r--r--   1 myuser  staff    27 Sep 30 20:17 .yo-rc.json
-rw-r--r--   1 myuser  staff  2299 Sep 30 20:17 build.cmd
-rw-r--r--   1 myuser  staff    50 Sep 30 20:17 build.ps1
-rwxr-xr-x   1 myuser  staff  2566 Sep 30 20:17 build.sh
-rw-r--r--   1 myuser  staff   135 Sep 30 20:17 global.json
-rw-r--r--   1 myuser  staff   390 Sep 30 20:17 make.shade
drwxr-xr-x   2 myuser  staff    68 Sep 30 20:17 test
drwxr-xr-x   3 myuser  staff   102 Sep 30 20:17 ~

myuser@mac : ~/src/test/dnx/web-api
 ==> ./build.sh verify
Downloading dnvm.sh from https://raw.githubusercontent.com/aspnet/Home/dev/dnvm.sh 
######################################################################## 100.0%
Determining latest version
Latest version is 1.0.0-beta7 
dnx-coreclr-darwin-x64.1.0.0-beta7 already installed in /Users/myuser/.dnx
Adding /Users/myuser/.dnx/runtimes/dnx-coreclr-darwin-x64.1.0.0-beta7/bin to process PATH
Updating alias 'default' to 'dnx-coreclr-darwin-x64.1.0.0-beta7'

./build.sh: line 84: mono: command not found
./build.sh: line 89: mono: command not found
./build.sh: line 95: mono: command not found

myuser@mac : ~/src/test/dnx/web-api
 ==> 

Condo should detect polymer.json files and automatically execute the test and build targets.

Hello,

As part of Project Orleans move to CoreCLR, we are first porting the current *.csproj to the new .xproj, so we can target both dnx451 and dnxcore50 on the same project build.

The porting progress is tracked by this issue: dotnet/orleans#368

Right now, we have on one of the projects, some MSBuild tasks that we need to port to the .xproj as follow:

<Target Name="OrleansDllBootstrapUsingCodeGen" AfterTargets="BeforeCompile" BeforeTargets="CoreCompile" Inputs="@(Compile);@(ReferencePath)" Outputs="$(ProjectDir)$(IntermediateOutputPath)Generated\$(TargetName)$(TargetExt)" Condition="'$(Bootstrap)' != 'true'">
    <Message Text="[OrleansDllBootstrapUsingCodeGen] - Bootstrapping the Orleans.dll" Importance="high" />
    <!-- Clean Generated directory -->
    <Message Text="[OrleansDllBootstrapUsingCodeGen] - Deleting/cleaning generated files for Project=$(ProjectName)" />
    <MakeDir Directories="$(IntermediateOutputPath)Generated" />
    <Touch Files="$(ProjectDir)Properties\orleans.codegen.cs" ForceTouch="true" AlwaysCreate="true" ContinueOnError="true" Condition="!Exists('$(ProjectDir)Properties\orleans.codegen.cs')" />
    <PropertyGroup>
      <BootstrapOutputPath>$(OutDir)Bootstrap\</BootstrapOutputPath>
      <BootstrapOutputPath Condition="'$(BuildingInsideVisualStudio)' == 'true'">$(ProjectDir)$(OutDir)Bootstrap\</BootstrapOutputPath>
      <ExcludeCodeGen>$(DefineConstants);EXCLUDE_CODEGEN</ExcludeCodeGen>
    </PropertyGroup>
    <Message Text="[OrleansDllBootstrapUsingCodeGen] - OutputPath: $(BootstrapOutputPath)" Importance="high" />
    <!-- Compile code generator -->
    <Message Text="[OrleansDllBootstrapUsingCodeGen] - Compiling Orleans.dll for bootstrap" Importance="high" />
    <MSBuild Projects="$(MSBuildProjectFullPath)" Targets="Build" Properties="Bootstrap=true;BootstrapOutputPath=$(BootstrapOutputPath);DefineConstants=$(ExcludeCodeGen)" UnloadProjectsOnCompletion="true" UseResultsCache="false" />
    <Message Text="[OrleansDllBootstrapUsingCodeGen] - Compiling code generators for bootstrap" Importance="high" />
    <MSBuild Projects="$(SolutionDir)\OrleansCodeGenerator\OrleansCodeGenerator.csproj" Targets="Build" Properties="Bootstrap=true;BootstrapOutputPath=$(BootstrapOutputPath);OutputPath=$(BootstrapOutputPath);OutDir=$(BootstrapOutputPath);DefineConstants=$(ExcludeCodeGen)" UnloadProjectsOnCompletion="true" UseResultsCache="false" />
    <MSBuild Projects="$(SolutionDir)\ClientGenerator\ClientGenerator.csproj" Targets="Build" Properties="Bootstrap=true;BootstrapOutputPath=$(BootstrapOutputPath);OutputPath=$(BootstrapOutputPath);OutDir=$(BootstrapOutputPath);DefineConstants=$(ExcludeCodeGen)" UnloadProjectsOnCompletion="true" UseResultsCache="false" />
    <!-- Finally invoke code generator on the recently built Orleans.dll -->
    <Message Text="[OrleansDllBootstrapUsingCodeGen] - Preprocessing $(TargetName)$(TargetExt) with ClientGenerator.exe" Importance="high" />
    <PropertyGroup>
      <ArgsFile>$(IntermediateOutputPath)$(TargetName).codegen.args.txt</ArgsFile>
    </PropertyGroup>
    <ItemGroup>
      <CodeGenArgs Include="/bootstrap" />
      <CodeGenArgs Include="/in:$(BootstrapOutputPath)$(TargetName)$(TargetExt)" />
      <CodeGenArgs Include="@(ReferencePath->'/r:%(Identity)')" />
      <CodeGenArgs Include="@(Compile->'/src:%(Identity)')" />
    </ItemGroup>
    <WriteLinesToFile Overwrite="true" File="$(ArgsFile)" Lines="@(CodeGenArgs)" />
    <Message Text="[OrleansDllBootstrapUsingCodeGen] - Invoking Code Generator" />
    <Exec Command="&quot;$(BootstrapOutputPath)ClientGenerator.exe&quot; &quot;@$(ArgsFile)&quot;" />
  </Target>

The idea there is to build one of the projects, a console app, and than use it to code-generation of the Orleans.csproj.

The big challenge here is that as we are going to be coreCLR compliant, the build must be supported on Linux/OSX (including using VS Code for that) so we can't rely on powershell scripts which would be a good replacement for that. We could have a build.ps1 and a build.sh for each environment if the only usage of the code generator was at the CI server however, we need it to run when building from Visual Studio/Code as well and that is why the MSBuild tasks were chosen at first place.

The other problem is that we can't install anything on the CI/build machine widely since we are using the .Net Foundation CI systems. So if something must be downloaded and run in order to perform the build, it must be done locally.

So, does Condo help us with that cases?

Thanks!