freebsd / redports-1 Goto Github PK

If someone maintains changes of Mk in his own portstree we need to be much more 
carefully with the packages that are generated because they are all tainted.

- Do not cache any packages in the cache if a User has Mk/ in his portstree
- Forbid Mk/ for regular users

IPv6 support for the frontend:
- Webserver
- Mail (awaiting IPv6 reverse records)
- Proxy for fetching distfiles

prbot should work but it needs some testing and probably a few small fixes 
before it can be activated.

Default setting should be to build all new PRs but only send a message to the 
PR in case the build failed.

Right now it's only possible to delete queued jobs but not running jobs.
Deleting a running job means we need to tell tinderbox to cancel the job and 
wait until it has finished with the cleanup.

It is great when looking at source commit of build to see what was changed, 
however if you do a web rebuild, it shows just the most recent committed 
revision. This revision isn't necessarily tied to the code you are working on. 
Please update the link to be latest revision of path to port you are working 
on, and not the latest revision in the entire redports repository, if possible.

Thanks!

-jgh

I am unable to build devel/quilt because perl is not added. It is specified in 
the makefile using USES=perl5. It works find on my local machine, but not on 
the buildbot. 
see https://redports.org/~samm/20140217212743-20669-177408/quilt-0.51.log

Currently QAT checkouts can create a lot of I/O because we do a regular svn 
update. Performance could be improved by using svn update --changelist and 
applying only the required changes.

https://redports.org/~meta/20130319021900-35536-108803/perl-5.14.2_3.log

Poudriere has daemon support now so we can start to implement a poudriere based 
backend.

1) Start daemon
2) Add job: echo '{command:bulk;arguments:"-j environment www/nginx"}' | nc -w 
1 -U /tmp/poudriered.sock
3) Job status: ??

Allow to limit RAM for tmpfs and jail.

mount -t tmpfs -o size=4G tmpfs /mountpoint
rctl -a jail:<jailname>:memoryuse:deny=2G/jail

Clean call leaves tinderbox package cache in an corrupted state. Symlinks in 
All aren't properly deleted.

It could be that subversion on the backends was compiled without 
ENHANCED_KEYWORD so $FreeBSD$ is not expanded properly.

The value of that gets into the .finished file, and when it is sourced, it 
fails with:

[Tue Mar 25 23:56:21 UTC 2014] status - build=8.4-FreeBSD-i386-QAT&line=836
/tmp/rptinderbox/8.4-FreeBSD-i386-QAT/.finished: 4: Syntax error: EOF in 
backquote substitution
/tmp/rptinderbox/8.4-FreeBSD-i386-QAT/.finished: 3: Syntax error: Error in 
command substitution

Admin interface needs some improvements:

* Backends change
* Buildgroups change
* Portrepositories change (also for teams)

Sometimes you need to build additional slave ports after
a commit to be sure that it wasn't broken by a commit.

A simple way to do that would be to add an additional tag
to the commit message like this:

Redports-Build: www/nginx devel/subversion

External subversion repositories for some
teams would benefit from automatic testing
on redports.org per commit.

So we need to provide a bot that polls their
repository (like qatbot) and then creates new
jobs (like rpaddcommit). It should be okay to
treat that repository as overlay like we do
for svn.redports.org.

If a dependency does not exist it should provide a nice error message like we 
already do for "port does not exist".

Distfiles are usually taken from the distfile cache but this makes it hard to 
detect missing distfiles if they disappear at some point in time. Regularly 
cleaning distfile cache would be good.

Some text fields are a bit too short in
rare cases. It would be good to increase
them.

- builds.buildlog to varchar(100)
- builds.wrkdir to varchar(100)
- builds.portname to varchar(100)
- buildqueue.description to text
- buildqueue.notice add as text

To properly support Trac 1.0 a few changes need to be made:
- CSS adjustments
- Migrate DB API usage to Trac 1.0
http://trac.edgewall.org/wiki/TracDev/DatabaseApi

It would be a very nice feature to be able to generate and submit a PR directly 
from redports.org via the webinterface.

Create a wiki page that documents the Frontend setup and configuration like we 
have already done for backends.

https://redports.org/~meta/20130322060203-45823-109333/gnutls-2.12.23_1.log

Experimental environments support for "non supported"
configurations.

Features:
- Needs to be optional (opt-out)
- Private mails only

Environments:
- 10-CURRENT/i386, 10-CURRENT/amd64
- CLANG/i386, CLANG/amd64
- GCC 4.7
- Custom LOCALBASE/PREFIX
- Server Options Profile
- Desktop Options Profile
- pngng DEVELOPER_MODE=on
- REGRESSION_TESTS=on
- bmake for ports

GitHub supports sending pull_requests via the Events API that we already use to 
an remote application. Redports could build that pull request and signal 
success/failure state in a "special comment" to the pull request.

Example: https://github.com/Homebrew/homebrew/pull/32346
API: https://developer.github.com/v3/pulls/

Admin build delete does not send a mail if it was the last job of a build.

Builds sometimes end up in finished state. It looks like they finish properly 
but there could be a race condition between status reporting the build as 
finished and the time the ".finished" file is created with the proper content.

https://code.google.com/p/redports/source/browse/trunk/rptinderbox/rptinderbox#3
52

Rpdd does lock backends very often but it's usually fine to just lock 
backendbuilds and reduce lock contention.

Provide additional statistics for QAT.

Statistics:
- per user
- % per user
- failed per month
- % failed per month

Backends currently do not load linux.ko so ports that need Linux emulation 
support will fail.

Possible candidate:
- pybot http://labix.org/pybot

For monitoring purposes an internal page should provide a list of backends and 
environments including status. That allows to have an external monitoring 
solution independent from redports.

QAT needs filter for failed builds so we need some basic filtering 
functionality to find them in the Trac GUI.

Filters:
- build status (failed, duds, depends, leftovers, all !success)
- owner
- environment

Right now the pkg cache is too simple and does not work properly for "exp-run 
like" builds.

- Job specific shared cache (all packages are saved in a job specific place 
that can survive multiple builds)
- Globally shared cache (job specific shared caches can be made available 
globally to all backends (shared storage?))

Locking in step 80 is too heavy. Needs rewrite to reduce lock contention.

Circular dependencies in a port cause a file $TB/builds/$buildname/make.0 to 
grow until the filesystem is full.

Best solution to reduce the impact would be to make $TB/builds/$buildname an 
own filesystem with maximum 50MB size.

Ports without staging support tend to generate leftover messages only on 10.0.

10.0-QAT builds are disabled because of that right now.

- github webhooks get notified after git push so we could use 
https://github.com/danheberden/gith to build a job for it that we can send to 
the redports web api

- redports web api as part of trac which can create buildjobs

- user configuration/assignment to github accounts

It would be nice to add a note to a buildqueue entry. Needs to be changeable at 
any time.

tmpfs mountpoints can remain mounted in case of errors in tinderbox. This is 
probably a bug in the tinderbox tmpfs patch and should be either fixed in 
tinderbox or a workaround could unmount existing tmpfs filesystems when 
mounting new ones.

fetch ftp://ftp.stack.nl/pub/users/johans/texinfo/20140209/texinfo.tex

behind the squid proxy results in a file with windows line endings. Fetching 
without the proxy gives the proper file with unix line endings.

To be able to have an idea why an environment failed it would be good to log 
all error messages from rpdd in a separate database table. The webinterface can 
use that and display the last error per build and also show the list of error 
messages in a separate mask.

columns:
id         int
backendid  int
buildgroup varchar(25)
loglevel   int
logmsg     varchar(1024)