buildasaurs / buildasaur Goto Github PK

I'd be really nice to extract whole BuildaCIServer to separate framework. Also CocoaPods and Carthage support are welcome.

I think that this implementation bring a lot of possibilities as you've captured a big part of Xcode Bots API in a nice container called XcodeServer.

A lot of people can take benefit from this and create other tools allowing them to use Xcode Bots from the outside of Xcode.

Right now when I go to the Xcode Server web site (where you can download/install integration builds), it shows me a long list of bots that all look like:

BuildaBot [mozilla/firefox-ios] PR #498

I would like to customize this name so that it shows something like:

PR #409 Do not stop system audio on start

(I don't care about the repo name or the BuildaBot static string).

Ideally this can be configured with some really simple template string like PR #{{PR_NR}} {{PR_TITLE}} so that people can customize in different ways?

Find a way to gradually slow down the sync interval when we're approaching the GitHub rate limit. So the sync interval set by the user is the minimum time between two syncs, but not the maximum - we might slow down to not hit the rate limit before the hour is over. Will become more important with multi syncer support and branch tracking, but would be good to know Builda is never blocked by just plain waiting for the rate limit to catch up.

Plus show it in the UI together with a button to download it.

It would be great to report failed tests, errors and warnings in detail in the GitHub comments already.

Test failures: we receive in the integration object.
Warnings/errors: we need to make an extra call to the integration's /issues endpoint.

• required refactor + tests: #184
• add object that constructs the new comments with input data
• pipe through the logic for asking for details of issues if build summary tells us that there was at least one error or warning)
• only report the most serious problem (e.g. don't report warnings when a test failure or error occured)
• split test (failure) information per device
• add which devices the tests ran on

Currently blocked by

• rdar://23142264 (Integrations endpoint doesn't return test result when asked for >1 integration)

Right now, the bots are automatically named BuildaBot [organisation/repo] PR #pr-no. In our office, we have all integrations showing up on a big screen, and in this case these bot names make no sense.

It would be nice to be able to specify bot names with a small DSL in a field. Especially, more interesting than the PR number is to have the actual PR title.

Suggested solution

Add a text field in the UI where we can input names. Also, supply a few variables that can be used in the string that will be replaced if they exist. For example, the following input:

BurgerBot :: "{pr_name}" (#{pr_no})

where {pr_name} and {pr_no} are variables, would generate this bot name:

BurgerBot :: "Flip burgers twice instead of once" (#3)

Additional variables could for example be organisation, repo etc.

Allow buildasaur to be setup as a daemon in order to run headless.

Possible ideas:
https://github.com/modcloth-labs/github-xcode-bot-builder
https://github.com/modcloth-labs/XBotBuilder

In this kind of setup possibly there would be no need to check out the Xcode project and workspace, but the configuration can be described into a file.

//Additional improvement:
When the server running buildasaur is restarted, there is no way to have buildasaur running without manual intervention.

Even when automatically login with an user and having buildasaur in the launch items we have to click on three buttons: Xcode server config, github config and start to have buildasaur working.

This is not the best situation in an automated environment.

To allow #64 and #29, we need to uniquely identify bots by not their name. Instead, see if we can add a custom key to the bot's json when we're creating it, which would also get returned when we're fetching bots. If so, we could just add the identifier there and the name would be free to be whatever users want.

I'd like a checkbox to prevent Buildasaur from posting stuff in the PR conversation. The build status of the PR is enough for us!

Currently Builda fails because it can't find the PR branch, because it lives in a fork instead of the original repository. I need to find a way to detect that case and create the bot with the correct origin URL and such.

Dependency: #18. Builds extra features on top of it.

Not just make Builda mark the commit appropriately on GitHub, also (optionally) create an issue if a watched branch suddenly starts failing tests (something like [Buildasaur]: Build failing on branch master) and comment what tests are failing exactly/what build error was emitted.
Then, when a new commit comes in that fails again, add a new comment with the new details (like we do with PRs).
Then, when a successful commit is pushed, comment that the issue was resolved and automatically close the issue.

• Add functionality to:
• Get open Issues
• Create a new issue
• Close an issue
• Add logic for creating the Issue when a tracked branch fails, mark the issue with a special label so that we can then only fetch these ones instead of all open Issues (reference 1, reference 2)
• Add logic for finding an existing matching issue (pull all open Issues with the special label first), create one if none exists
• Make sure the PR logic for only posting a comment if it hasn't done so etc is working
• Add logic for closing the Issue when the branch succeeds again

Currently we only validate the GitHub token, we also need to make sure the SSH keys are ok. Discussion began in #27.

Mostly for debugging purposes, it'd be nice to have Builda logging all of its logs into a file instead of just the standard output.

There is a websocket published by Xcode Server which posts very detailed updates on the build. You can see it in Xcode when watching a running build. This might be nice to pipe into Builda and maybe, in limited quantities, use for better updating of the state of the build in PRs.

Is there any merit in adding support for BitBucket pull requests into Buildasaur?

You asked for it. ;)

It would be nice to be able to watch multiple repositories at the same time from the same machine.
Maybe a simple approach would be to allow multiple instances of Buildasaur running at the same time.

The only problem would be the 5000 requests limit which are used up by all instances but then one could simply configure each instance to poll less frequently.

Pull code coverage information out of the build results and add it to the comment in the PR, so that users can see if a particular integration/PR is making code coverage better or worse.

When manually triggering an integration on the pull requests bot (no code changes between the first integration and the manual one being triggered), buildasaur post on GitHub a comment with the info about the first integration instead than about the last one.

The same happens when triggering a third integration manually.

I am almost sure this is caused by having no code changes in between those integration.

For large open source projects - only allow repo's owners to "lttm" pull requests, so that random people can't lttm their own PRs if they are not owners.

Implementation
Look at the comment poster and figure out, whether they belong to the set of owners of the repo.
Actually, let's base it on Collaborators.
Ha! There's a really easy way to check whether a user is a collaborator.

Steps

• Add a toggle for using Collaborator-based "lttm" (Not sure if necessary, should probs always be on)
• When checking the enabled state of a bot, first filter only the comments made by collaborators and then look for "lttm" (so that anyone else's potentail lttm would be ignored)
• Catch when a non-collaborator tries to "lttm" and post a comment informing them they can't do that as non-collaborators. (Maybe)

The default NSURLSession's configuration respects GitHub's response TTL, which is up to 60 seconds. However, if we want to sync more often (GitHub's authenticated rate limit is 5000 requests per hour, which is plenty to make Buildasaur very responsive to the team's workflow, when the sync interval is lower, cca 15sec), currently we get the same, cached response for ~4 more syncs, leading to duplicate comments etc. We need to disable the cache to always get the uncached state of the PR (status and comments, most importantly)

I have some ideas for how the UI should look in Buildasaur. Right now it looks very crammed, lots of info that doesn't really need to be shown. I started working on migrating it to how I would like it to look, but I won't put too much work until I know what direction @czechboy0 wants to take.

Anyway, I've used the OS X user settings pane as a basic guide:

In Builda, this roughly translates to having a list of repos on the left side, and +/- buttons to add/remove them as needed.

What's more important is the right side. Right now, the Builda UI shows 3 different things on the main screen - project setup, server setup and console/running status. These 3 things should be 3 different top tabs in the new UI. No need to show them all at once, and it will make a much more pleasant and light default view.

Builda is also showing a bunch of settings buttons today, which really could go in an advanced settings page for the repo. Maybe something like the gear in the list to the left in the example above.

@czechboy0 What do you think? Is this a direction you'd like the Buildasaur UI to take?

Xcode Server that shipped with Xcode 7 breaks Buildasaur, because the API version has been upped.

Back to reverse engineering.

Buildasaur immediately crashes after being started, Nothing in the logs. Is there a debug mode so that I can provide some more feedback?

Now that I've ripped out BuildaCIServer and turned it into a separate project and a pod XcodeServerSDK, let's integrate it back again.

Pull from scheme -> id -> project -> buildConfigurations -> take first -> find XCBuildConfiguration. Look for IPHONEOS_DEPLOYMENT_TARGET and MACOSX_DEPLOYMENT_TARGET (probably also something like WATCHOS_DEPLOYMENT_TARGET. This should indicate the type of a target.

See $ man /Applications/Xcode.app/Contents/Developer/usr/bin/xcsbridge for documentation on xcsbridge. It might be useful for blueprint validation and preflighting, making sure that the blueprint we're creating a bot with will indeed succeed. Will require a way to trigger a shell script from Swift and wait for its result. This might be a good way to catch invalid SSH keys etc before we create an invalid bot!

It would be nice to have a way to also archive certain PR's code, so that it can be installed OTA from Xcode Server and tested on a device quickly (for UI changes).

Control over this would preferably be in the UI, so now users comment "lttm" to start testing a bot, there would be another keyword, like "archive", which would switch the Build Template to one that uses an Ad Hoc provisioning profile for the Archive action.

This would basically require a Build Template to be tagged as "adhoc" or each Build Template would have an optional property of "adhoc-scheme", which it could use in this case. This needs more thought, but should be pretty useful.

I'm trying to edit a trigger script but when i type a return, the text field 'submits' instead of inserting a newline.

Blocks #69. Migrate over so that we can pull in the new version of XcodeServerSDK.

Keep the feedback coming, but please note that I might not be able to reply in the next two weeks, because I'll be in SF. Ping me on Twitter if you wanna meet there!

Body Test

When generating SSH keys, you can optionally enter a passphrase. When setting up bots in Xcode, and setting credecentials -> Existing SSH keys, there is a field to enter this passphrase so Xcode can decrypt the keys.

There is no such field in Buildasaur, and when trying to use encrypted keys, buildasaur will say "Verified access, all is well" but all bots will fail to check out.

Suggested solution: Add a passphrase field and forward it to the Xcode bots if not left empty.

Hi, Honza.
You probably already anticipated it, so here's the request for the possibility to remove the "lttm barrier". :)

We're using your awesome tool for our pull requests which go to private repositories with only trusted users to access them.
Forgetting to type lttm leads to no hints at all of any failed tests.

Right now we simply return isEnabled: true from isBotEnabled() in our own build of Buildasaur but it would be nice if you could incorporate that as an option so we won't have to modify your code after each update.

Body Test

To save rate limit hits, GitHub recommends using conditional requests, so that when nothing changes, no data is returned (only a special code). Look into using either a built-in solution (NSURLCache) or writing something from scratch. This will help to save a significant portion of now-consumed hits, allowing for a bright future with multiple project support and more features, while staying safely under the rate limit.

As part of #78 I had to disable SSH key validation due to the new server fingerprint validation kicking in. Let's bring it back, as it's a nice feature (to validate keys before confirming that all credentials are ok).

Think about how we could also add a syncer for not just PRs, but only for a "monitored" branch, e.g. master. A way to monitor a branch with a bot and to update the commit's status based on the bot's result. Would require a slight refactor where we'd allow the syncer to instead of resolving (PR + Bot) pairs to (branch + Bot) pairs, where PRs would just be one source of the monitored branch and we might have the user manually specify another, non-PR'ed branch to also be monitored and updated on GitHub.

• Split the main syncer into multiple classes
• Migrate to a SyncPair process, where each pair (Bot + PR, No Bot + PR, Bot + No PR) has its own class that handles its actions. Will scale better for new syncable pairs (Bot + Branch, ...)
• Add tests for the right SyncPair creation based on inputs from GitHub and XCS
• Introduce a Bot + Branch Sync Pair, which will share a lot with Bot + PR, share as much as possible
• Add a way in the UI to mark particular branches to be synced permanently (no need for a PR)
• Answer the question: will we allow picking a different Build Template for synced branches?
• Enable this on master here in Buildasaur and test that it works well for a couple of days

This is building with an xcworkspace, none of the schemes show up in template settings

We ran into a problem with our private repository and the GitHub user account used for Buildasaur had read only access to the repository.
Credential validation from Buildasaur was successful and Bots were created, integrations ran.
But posting the integration result to GitHub and adding the comment failed and it took us a while to figure out why.

It would be cool if Buildasaur could check for write access as well and at least warn that integration results cannot be published on GitHub without write access.

This is to make setting up Builda even easier. Instead of having to have user's project to be tested checked out locally (so that Builda can parse the Xcode project and other files), make it check out the project in a temp location just from the URL and credentials.

Possible issue: how to force Xcode to create the .xccheckout file if it's not there?

Look into xcscmblueprint, #89.
Is there a way to force xcode to create the xcscmblueprint or xccheckout files?

• Configure the script to checkout the repo in a temp location from the blueprint
• Add this to the onboarding step when choosing projects (offer to pull from GitHub directly)

We need to be smarter about GitHub's rate limiting, right now we're doing basic math and hoping for the best, but we need a bailout strategy for when we go over the limit and a way to show that in the UI.

Add a badge specifying the status of the latest master commit, something like and