mochify / renderly-legacy

Archived: A non-proprietary version of my Rendering automation library/toolkit.

This Version of Renderly is Deprecated

After building the original Renderly, I realized more things that I wanted to do with the core library, and I'm in the process of splitting this repository up. Only maintenance and bug fixes will go into this version.

I split out RenderlyApp from Renderly and have moved it to Renderly-New. The new project is backward compatible, but if you want to use this project as-is, note that there will be no new active development for this project.

If you have any questions or issues, please file an issue.

Renderly

Renderly is a .NET library and application for automated comparison of images. It is intended for use in environments where you want to periodically check generated images against a reference image to see if any changes have caused the images to render differently or incorrectly.

Source Code Layout

The code is broken up into a few projects, described below.

Renderly

This is the core library which provides the basic interfaces for building an application.

RenderlyApp

This is a reference application built on top of Renderly that takes a CSV file with test cases, runs image comparisons against the test cases, and generates a report of results.

RenderlyTests

This project has all the unit tests for the Renderly library.

Documentation

Read the wiki!

Build

You will need Visual Studio 2012 or Visual Studio 2013 to build. I assume the code should build fine in Visual Studio 2010, but it has not been tested with that version.

I have not provided a build script yet, so right now you should open Renderly and RenderlyApp in a solution and rebuild.

Running RenderlyApp

This applies to you if you're using the included RenderlyApp. Running RenderlyApp.exe without any arguments should produce the list of supported commands:

> .\RenderlyApp.exe -h

Available commands are:

    batchadd    - Add test cases as a batch
    deletetest  - Delete test cases (and their reference images) from a model
    run         - Run a rendering job
    updateref   - Update reference images for provides test cases.

    help <name> - For help with one of the above commands

batchadd

The batchadd command allows you to create a new test case file, or append test cases to an existing one. Note that if you append, it does not perform an in-place modification of the input file.

The command takes the following arguments:

 > .\RenderlyApp.exe help batchadd

'batchadd' - Add test cases as a batch

Expected usage: RenderlyApp.exe batchadd <options>
<options> available:
-f, --file=VALUE           CSV file with the test cases to generate.
-o, --out=VALUE            CSV file to output with test cases.
-a, --append=VALUE         Test Case file to append to

The file you pass to the -f argument is a CSV file that should start with the following header:

Type,Reference Location,Reference Save Directory,Release,Description

• The Type column is the type of test this is.
• The Reference Location column is a URI to the image where the image should be copied from. This image becomes the test case's reference image, and the location is where the test will obtain the image from in the future.
• Reference Save Directory is the directory where the reference image will be saved.
• Release is what software/version release this test was originally created for, or what version it should apply to. This is mostly a reference field in case you are curious about what version/release you were testing an image in.
• Description is a short comment about the test case. Only used for your reference.

deletetest

The deletetest command allows you to delete a test case from a file.

The command takes the following arguments:

> .\RenderlyApp.exe help deletetest

'deletetest' - Delete test cases (and their reference images) from a model

Expected usage: RenderlyApp.exe deletetest <options>
<options> available:
  -f, --file=VALUE           Model file to delete test cases from.
  -d, --dates=VALUE          Comma-separated list of dates to delete test
                               cases for. MM-DD-YYYY format.
  -r, --releases=VALUE       Comma-separated list of releases to delete test
                               cases for
  -t, --testids=VALUE        Comma-separated list of test IDs to delete
  -o, --outfile=VALUE        File to save test cases to. If no file is
                               specified, the input file is overwritten.

Options should be rather self-explanatory. You can specify multiple different parameters to erase from a test case file. Any reference images will also be deleted from their saved location, if possible (may not be possible due to permissions or other accessibility issues).

updateref

The updateref command lets you update test cases, making the application redownload reference images. This may be useful in case your images or rendering changes, and you want to update the reference image a test case uses.

The command takes the following arguments:

> .\RenderlyApp.exe help updateref

'updateref' - Update reference images for provides test cases.

Expected usage: RenderlyApp.exe updateref <options>
<options> available:
  -f, --file=VALUE           Model file to update test cases in.
  -t, --testids=VALUE        Comma-separated list of test IDs to redownload
                               reference images for.

run

The run command lets you run a rendering and comparison test, using an input CSV file. A report will be generated detailing the results of the run.

The command takes the following arguments:

> .\RenderlyApp.exe help run

'run' - Run a rendering job

Expected usage: RenderlyApp.exe run <options>
<options> available:
  -f, --file=VALUE           The model to get test cases from.
  -n, --name=VALUE           The name of the report to generate
  -o, --outdir=VALUE         The directory to generate the report in
  -m, --templatedir=VALUE    The directory to get templates for report
                               generation
      --threshold=VALUE      Threshold value to configure how aggressive
                               image comparison is (0-100). 100 is exact match.
                               Defaults to 100.
      --showall              Show all results in report (including successes-
                               ). By default, only failures are shown.
      --copyref              Copy reference images locally to report
                               directory. Default false.
  -t, --testids=VALUE        Comma-separated list of test IDs to run
  -r, --releases=VALUE       Comma-separated list of releases to run
  -d, --dates=VALUE          Comma-separated list of dates to run. MM-DD-YYYY
                               format. Runs for Date Added.
  -y, --types=VALUE          Comma-separated list of types to run.

The CSV file you pass to the -f option should start with the header:

Test Id,Type,Source Location,Reference Location,Date Added,Date Modified,Release,Description

• Test Id is the number of this test. It should be a unique identifying number.
• Type is the type of test this is.
• Source Location is the URI to obtain the image for comparison.
• Reference Location is the URI to the base image that is used for comparisons.
• Date Added is the date this test was added to the model.
• Date Modified is the date this test was last changed/updated.
• Release is the software release/version this test was originally created against.
• Description is a short comment about this test case.

A valid CSV file can be obtained by using the batchadd command.

License

Renderly is released under the Apache License, Version 2.0 license, because information wants to be free, but we want to be protected from patent retaliation.

renderly-legacy's Issues

Renderly: `RenderlyNativeAssetManager.FetchToRandomFilename` throws exception if `outputDirectory` doesn't exist

The method RenderlyNativeAssetManager.FetchToRandomFilename attempts to download or copy a file from one location to another. If the supplied path outputDirectory doesn't exist, it throws an exception that crashes the app.

Make sure the directory exists before the copying is attempted.

Make UNC Paths Work

Description of Required Functionality

UNC paths should work in Renderly. They don't!

App crashes with no logging when batchadd encounters a bad URL

Description of Bug

If the batchadd process encounters a bad URL (one of mine is apparently throwing a 500), it crashes. This wouldn't be so bad, except that since the test case file it's generating is being held in memory, there's no way to tell what line/URL it failed on.

Reproduction

1. Create an input file with a bad URL.
2. Run a batchadd on it.
3. Notice the crash:

Unhandled Exception: System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at System.Net.HttpWebRequest.GetResponse()
   at Renderly.Utils.RenderlyNativeAssetManager.FetchToRandomFilename(String fetchUri, String outputDirectory) in f:\git
projects\renderly\Renderly\Utils\RenderlyNativeAssetManager.cs:line 89
   at Renderly.Utils.TestCaseGenerator.Generate(ShellTestCase shell) in f:\gitprojects\renderly\Renderly\Utils\TestCaseG
enerator.cs:line 36
   at Renderly.Utils.TestCaseGenerator.<GenerateTestCases>d__0.MoveNext() in f:\gitprojects\renderly\Renderly\Utils\Test
CaseGenerator.cs:line 26
   at Renderly.Models.Csv.CsvModel.AddTestCases(IEnumerable`1 it) in f:\gitprojects\renderly\Renderly\Models\Csv\CsvMode
l.cs:line 83
   at RenderlyApp.Commands.BatchAddTestCaseCommand.Run(String[] remainingArguments) in f:\gitprojects\renderly\RenderlyA
pp\Commands\BatchAddTestcaseCommand.cs:line 56
   at ManyConsole.ConsoleCommandDispatcher.DispatchCommand(IEnumerable`1 commands, String[] arguments, TextWriter consol
eOut, Boolean skipExeInExpectedUsage) in c:\src\ManyConsole\ManyConsole\ConsoleCommandDispatcher.cs:line 81
   at RenderlyApp.Program.Main(String[] args) in f:\gitprojects\renderly\RenderlyApp\Program.cs:line 35

Ideal Behaviour

The app will either:
(A) Write to the generated file continuously so that the user can see where it "left off".
(B) Crash, but output some text indicating the line of the input CSV file that is problematic.
(C) Ignore errors and not crash, but spit out a log of any skipped URLs afterward.

Utilities: Reference Image Updater

Description of Required Functionality

In the event that a test case fails for a benign/innocuous/allowable reason (eg. the font in a combo is updated by creative that changes the way something looks when rendered), we need the ability to update the reference image to match. Ideally, a (comma-separated?) list of Test Case IDs could be supplied and all reference images would be "re-pulled" and replaced with the new (good) versions.

Run Test Cases Based on Filtered List

Description of Required Functionality

Right now, Renderly will run all tests in a given CSV file. It would be nice if Renderly could run a filtered selection of those tests. To start, being able to filter by Test Type would be nice. As well as the ability to run a specified test (by Test Case ID) for testing purposes.

Allow Renderly to Accept File Paths Instead of URLs

Description of Required Functionality

Renderly currently accepts URLs for test cases. It should also be able to accept images from file paths (mainly for ease of testing).

RenderlyApp: Make report name optional and supply default report name

himekat wants to automate usage of RenderlyApp and would rather not have to specify a report name. Make the -n option optional in the app and have it default to using something like

ReportName = string.Format("renderly-{0:yyyy-MM-dd_hh-mm-ss-tt}", DateTime.Now);

Report: Comma-separated list of Failed Test Case IDs in Report

Description of Desired Functionality

In addition to the image comparator results that the report spits out, it would be nice if it also spit out a comma-separated list of all the failed Test Case IDs. This would be helpful for when additional utilities are implemented, like deleting and updating.

Provide image link in report for failed test cases

Description of Desired Functionality

In the case of a failed test case, please include a column in the report which has a link to the URL for that test case (not the reference image, the 'new' image). Realistically, if it's easier, you can just include the link for all test cases.

Refactor: split out test generation from the CSV model

Test case generation is currently built into the CSV model, but this needs to be split out for other models that may be implemented (like a SQLite3-backed model or whatever).

RenderlyApp crashes if files are not found

The app crashes if an exception is thrown from the RenderlyAssetManager if it can't retrieve a file for any reason (whether it be from disk or reading from the web).

Make Pass/Fail Reporting Configurable

Description of Required Functionality

Renderly does not need to spit out "passed" test cases in the report. They are fairly useless to the person triaging the automation. By default, Renderly should only report "failed" cases, although it can be a configuration option if desired.

App crashes when reference and target images are different dimensions

Description of Bug

If the target image is a different size (dimension-/pixel-wise) than the reference image, the app crashes with the one of the following exceptions (depending on the specific change):

Unhandled Exception: AForge.Imaging.InvalidImagePropertiesException: Template's size should be smaller or equal to source image's size.
   at AForge.Imaging.ExhaustiveTemplateMatching.ProcessImage(Bitmap image, Bitmap template, Rectangle searchZone)
   at Renderly.Imaging.ExhaustiveTemplateComparer.Matches(Bitmap reference, Bitmap compare) in f:\gitprojects\renderly\R
enderly\Imaging\ExhaustiveTemplateComparer.cs:line 32
   at Renderly.Controllers.RenderingController.RunTest(TestCase tc) in f:\gitprojects\renderly\Renderly\Controllers\Rend
eringController.cs:line 49
   at Renderly.Controllers.RenderingController.<RunTests>d__0.MoveNext() in f:\gitprojects\renderly\Renderly\Controllers
\RenderingController.cs:line 31
   at RenderlyApp.Commands.RunRenderingCommand.Run(String[] remainingArguments) in f:\gitprojects\renderly\RenderlyApp\C
ommands\RunRenderingCommand.cs:line 119
   at ManyConsole.ConsoleCommandDispatcher.DispatchCommand(IEnumerable`1 commands, String[] arguments, TextWriter consol
eOut, Boolean skipExeInExpectedUsage) in c:\src\ManyConsole\ManyConsole\ConsoleCommandDispatcher.cs:line 81
   at RenderlyApp.Program.Main(String[] args) in f:\gitprojects\renderly\RenderlyApp\Program.cs:line 35

Unhandled Exception: AForge.Imaging.InvalidImagePropertiesException: Overlay image size must be equal to source image si
ze.
   at AForge.Imaging.Filters.BaseInPlaceFilter2.ProcessFilter(UnmanagedImage image)
   at AForge.Imaging.Filters.BaseInPlaceFilter.Apply(BitmapData imageData)
   at AForge.Imaging.Filters.BaseInPlaceFilter.Apply(Bitmap image)
   at Renderly.Imaging.ExhaustiveTemplateComparer.GenerateDifferenceMap(Bitmap reference, Bitmap compare) in f:\gitproje
cts\renderly\Renderly\Imaging\ExhaustiveTemplateComparer.cs:line 60
   at Renderly.Controllers.RenderingController.RunTest(TestCase tc) in f:\gitprojects\renderly\Renderly\Controllers\Rend
eringController.cs:line 60
   at Renderly.Controllers.RenderingController.<RunTests>d__0.MoveNext() in f:\gitprojects\renderly\Renderly\Controllers
\RenderingController.cs:line 31
   at RenderlyApp.Commands.RunRenderingCommand.Run(String[] remainingArguments) in f:\gitprojects\renderly\RenderlyApp\C
ommands\RunRenderingCommand.cs:line 119
   at ManyConsole.ConsoleCommandDispatcher.DispatchCommand(IEnumerable`1 commands, String[] arguments, TextWriter consol
eOut, Boolean skipExeInExpectedUsage) in c:\src\ManyConsole\ManyConsole\ConsoleCommandDispatcher.cs:line 81
   at RenderlyApp.Program.Main(String[] args) in f:\gitprojects\renderly\RenderlyApp\Program.cs:line 35

Reproduction

1. Begin with a test case that passes comparison.
2. Modify the reference image to be slightly smaller than the target image.
3. Run a comparison
   
   --It should fail with the above exception

Ideal Behavior

Ideally, the app won't crash, it will actually indicate the size difference in the report (by failing the test case).

Utilities: Test Case Deletion

Description of Required Functionality

Should a test case or test cases become unnecessary/unusable, we should be able to delete it/them. Ideally, the base requirement of this is for the deletion process to take a (CSV?) list of Test Case IDs to delete.

Description of Additional Desired Functionality

A bad-ass rockstar process, though, would be a bit more flexible. I can see cases for wanting to delete cases based on date added, date modified, release, or test type...

Utilities: Test Case Generation

Description of Required Functionality

A Test Case Generation utility which can do the following:

1. Take in a CSV file with pertinent test case information (eg. Test Type, URL to image, Release) and insert the info into the Test Case Repository CSV file, with auto-population of other fields (eg. Test Case ID, Date Added, etc).
2. Do an initial pull of the image(s) from the URL(s) and store in the file system.

Note: should (obviously) be able to take more than one new case at a time.

RenderlyApp: Report generation fails if `--showall` option isn't enabled and all test cases pass

The ReportService class only calls CreateReportLayout() when it has to write any assets to disk. However, this doesn't happen when all test cases pass, and causes report generation to fail because the report.html file can't be written to a non-existent directory.

Report: Constrain Image Size

Description of Required Functionality

The report should constrain the image sizes so that they are uniformly displayed.

Image Comparator Thresholding Option

Description of Required Functionality

The Image Comparator should be able to accept a different "difference" threshold at run-time to make the comparison fuzzier or more precise.

Allow RenderlyApp to filter by test type

Related to issue #4 - I added filtering based on Date Added, Release, and Test Ids, but forgot to add filtering by test type.

This is simple - add a new option to the RunRenderingCommand, take a CSV-list, and add predicates.

Bitmaps cannot be saved because of GDI+ internal exception

This is happening because the preview/reference files are being loaded from streams in the RenderingController. Depending on the backing source (this happened when loading a bitmap using a stream backed by a local file), closing the stream resulted in the Bitmap objects being unusable.

RenderlyApp: Set User-Agent and allow custom User-Agents for web downloads

This came up on a site that gated visitors if they weren't using a supported browser, even when attempting to directly download an image.

Allow setting of the UserAgent field on HttpWebRequest and WebClient objects in the RenderlyNativeAssetManager.