Fix flakey snapshots based tests. about eidolon

[orta]

I'm seeing this on UIView based tests. The issue can lie in four places.

• FBSnapshots (unlikely)
• Quick (unlikely)
• Our Nimble Extension (highest chance imo)
• Swift ( sigh )

[ashfurrow]

Yeah could be. We also don't have the same robust supporting framework that we do in eigen to ensure that view controllers are presented with a frame, for example.

[ashfurrow]

OK, seeing some more consistent behaviour. If you have a test that's failing (for me it's usually ConfirmYourBidArtsyLoginViewControllerTests), you can force the view to load:

expect(sut.view) != nil

That alone won't help the test pass. Instead, pause the tests in the debugger just before the sut is tested. Quick look its view (it's empty). Close the quick look window and quick look it again (it's no longer empty). And the test will pass from now on.

Not sure what to make of these findings but will continue looking.

[ashfurrow]

Interestingly, using an asynchronous matcher solves the problem:

expect(sut).toEventually(haveValidSnapshot(named:"default"), timeout: 1.0, pollInterval: 0.05)

The timeouts are interesting. Anything less than 0.3 and they fail frequently. Anything over that and it never fails. So I'm thinking there's some part of this that thinks it's being animated maybe? Not sure.

[ashfurrow]

@orta Good hunch to test the FBSnapshotTest case individually. I used this code on a failing test:

let result = FBSnapshotTest.compareSnapshot(sut, snapshot: "ConfirmYourBidEnterYourEmailViewControllerTests_swift_default", testCase: sut, record: false, referenceDirectory: "//Users/ash/Dropbox (Personal)/bin/eidolon/KioskTests/ReferenceImages")
expect(result) == true

And it never failed. So the problem is almost certainly in the Nimble-Snapshots matchers (note that the method I'm calling above belongs to FBSnapshotTest, a class in Nimble-Snapshots).

[ashfurrow]

Actually I may have spoken too soon...

[ashfurrow]

Yeah so this is happening without using anything from Nimble-Snapshots.

var snapshotController: FBSnapshotTestController = FBSnapshotTestController(testClass: sut.dynamicType)
            snapshotController.recordMode = false
            snapshotController.referenceImagesDirectory = "//Users/ash/Dropbox (Personal)/bin/eidolon/KioskTests/ReferenceImages"
            let result = snapshotController.compareSnapshotOfView(sut.snapshotObject, selector: Selector("ConfirmYourBidEnterYourEmailViewControllerTests_swift_default"), identifier: nil, error: nil)
expect(result) == true // failed

So my gut says that it's either in iOS 8 or storyboards. Going to try and reproduce in a fresh project.

[ashfurrow]

Unless it's snapshotObject ...

@objc protocol Snapshotable {
    var snapshotObject: UIView? { get }
}

extension UIViewController : Snapshotable {
    var snapshotObject: UIView? {
        self.beginAppearanceTransition(true, animated: false)
        self.endAppearanceTransition()
        return self.view
    }
}

Maybe Swift doesn't like side-effects in computed properties in protocols?

[orta]

[ashfurrow]

Ha.

Yeah sorry for all the chitchat here. Kind of rubber-ducking I guess.

[ashfurrow]

Nope, this is something with Facebook's lib and iOS 8 / Swift.

let sut = ConfirmYourBidEnterYourEmailViewController.instantiateFromStoryboard()

sut.beginAppearanceTransition(true, animated: false)
sut.endAppearanceTransition()

var snapshotController: FBSnapshotTestController = FBSnapshotTestController(testClass: sut.dynamicType)
snapshotController.recordMode = false
snapshotController.referenceImagesDirectory = "//Users/ash/Dropbox (Personal)/bin/eidolon/KioskTests/ReferenceImages"
let result = snapshotController.compareSnapshotOfView(sut.view, selector: Selector("ConfirmYourBidEnterYourEmailViewControllerTests_swift_default"), identifier: nil, error: nil)

expect(result) == true // fails

[ashfurrow]

OK, so I can't reproduce the error with Storyboards and FBSnapshotTestController. So I think it might be Quick.

[ashfurrow]

I can't get it to reproduce with a plain XCTestCase, so I think it's something with Quick. Going to look at it again in the morning.

[ashfurrow]

This is a problem with either iOS 8, Xcode 6.x, or Storyboards, as the issue occurs with vanilla Xcode projects. I've opened a Stack Overflow question with more details.

[segiddins]

@ashfurrow given that 0.3 seconds is the default CoreAnimation implicit animation timing, my guess is that there's a so sort of animation going on.

[ashfurrow]

@segiddins Nah, this is a problem with the iOS API. This issue was actually closed by #80 – @orta, feel free to reopen if you're still experiencing intermittent failures.