Coder Social home page Coder Social logo

hive-scheduler's Introduction

Build Status

Hive Scheduler

The scheduler component behind the hive, and the main web view. For a full list of release notes, see the change log

Installing

Pre-requisites

  • git - Some of the gems are only currently available via Github
  • MySQL client libraries - sudo apt-get install libmysqlclient-dev on Ubuntu
  • A javascript runtime - For example, NodeJS - sudo apt-get install nodejs on Ubuntu

Configuration

Hive-Scheduler is configured using the Chamber gem. You can add configuration directly into the config/settings.yml file, or by setting the following environment variables (in the following lines to ~/.bashrc (or equivalent):

export RAILS_ENV=production
export DATABASE_ADAPTER=mysql2
export DATABASE_ENCODING=utf8
export DATABASE_HOST=your_database_host
export DATABASE_PORT=3306
export DATABASE_POOL=5
export DATABASE_USERNAME=database_username
export DATABASE_PASSWORD=database_password
export DATABASE_DATABASE=database_name
export ATTACHMENT_STORAGE=filesystem
export ATTACHMENT_STORAGE_PATH_BASE=public
export HIVE_QUEUES=true

Set up database

./bin/rake db:migrate
./bin/rake db:seed
./bin/rake assets:precompile

Example data

Some example execution scripts and projects can be created by editing the db/seeds.rb file before executing the rake tasks above.

Start the server

./bin/rails s

Examples

Hello world

Go into the 'Scripts' section and create a new script. Select 'Shell Script' as the target platform. Set the name to 'Hello <name>' and in the template box enter:

# This will use the execution variable 'word'
echo Hello $HIVE_WORD

Add a new execution variable called word and set the field type to 'String'. Save the script.

Go into the 'Projects' section and create a new project. Set the name to 'Hello world' and select the 'Hello <name>' script. Select the 'Manual' population mechanism and enter 'world' in the Word field and 'bash' in the Queues field. Leave all other fields as the defaults. Save the project.

Go into the 'Batches' section and create a new batch. Set the name to 'First test batch'. Select 'Hello world' as the project and set the version to '1.0'. Leave all other fields as the defaults. Save the batch.

In the 'Batches' section you will now see the batch you have just created and by clicking on the name you can see a single job belonging to this batch for the queue 'bash'. If you have a hive set up to run shell tests on a queue named 'bash' then this test will be executed and by clicking on the job number you can view the output.

Android test

The Android Calabash script is set up by default. In the 'Projects' section create a new project and select 'Android Calabash' as the execution type. Set the name to 'X Platform Example' and enter the following in the repository field:

[email protected]:calabash/x-platform-example.git

Select 'Manual' as the population mechanism and enter 'android-5.1.1' in the queue field. Click on the 'Add queue' link and enter 'android-4.4.4' in the new queue field.

Note; the hive runner is configured to run tests listed in queues specified by name. The values entered in these fields should be changed as appropriate for the configuration of your hives.

Save the project.

In the 'Batches' section create a new batch and select the project 'X Platform Example'. Set the name to 'Testing X platform example' and set the version. Upload the apk, which can be found at

Leave all other fields unchanged (the list of queues may be amended as required) and save the batch. As with the 'Hello world' example, the jobs will run if hives are set up with devices for the given queues and output can be viewed.

License

Hive Scheduler is available to everyone under the terms of the MIT open source licence. Take a look at the LICENSE file in the code.

Copyright (c) 2015 BBC

hive-scheduler's People

Contributors

asimk21 avatar davidbuckhurst avatar jitgo avatar jrmhaig avatar maikeru avatar wordswords avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hive-scheduler's Issues

"Jobs group is invalid" when not entering all mandatory fields

When creating a new batch, if you don't enter all of the mandatory fields (i.e. Version) then you receive the following error message:

screen shot 2015-09-03 at 09 46 12

But 'Job groups' isn't a valid field nor is it mandatory - in this example entering a version is enough to pass the validation.

Project page duplicates execution script

Submitting an incorrectly filled in project results in two copies of the execution script being shown.

Steps to reproduce:

  1. Go to the 'Projects' page and press the 'New' button
  2. Select an execution type - you should now see the execution script at the bottom of the page
  3. Hit 'Create' and you should see an error saying you need to fill in name and Population Mechanism
  4. Change 'Population Mechanism' to 'Manual'
  5. Scroll to bottom of page and two copies of the execution script will be there

screen shot 2015-07-21 at 12 00 32

For attached devices: display 'user friendly' name

At the moment it's difficult to determine which devices are connected as the 'device names' displayed in the UI aren't user friendly and it requires us to go and search in Google to see what the actual device is.

E.g. instead at the moment one of our device names is 'mobile-motorola-xt1039 3' whereas it would be much easier if it was displayed as 'Moto G'
Same for all other devices.

Is this something which could be implemented?

Retry individual tests

Ensure test URIs are reported into the database, and can be appended to a test script using $HIVE_RETRIES

Improve page names

Currently all pages are called 'Hive'. This makes navigating through the back/forward buttons and history difficult - the page name should be more descriptive

Error when submitting empty batch

Steps:

  1. Click on new (batch)
  2. Without filling any form details click on Finish button

I think Finish button should be disabled if nothing is filled in form.

Use scope for latest_batch

The latest_batch method in the Project method needs to be changed to a scope so that Active Record can optimise queries.

CSS Cleanup

Go through css files, work out what's in use, and consolidate result badges and label css. Ideally we would have simple result classes that we can apply consistently.

Target platforms needs relook

Mobile Browser works same as Shell script

I think there should be some suggestion like populating default variables when particular target is selected

Artifact view

Need a new view of a batch that groups artifacts produced by the tests and shows them in a side-by-side view for comparison.

Issue while creating new batch as it didn't take users to next screen immidiately

Issue :
User confuse while creating new batch as when clicked on "Finish" the filed "Test" gets highlighted but batch started in the background.

It takes some time to take user to next page or publish message that batch has been started.

Steps :

Created New batch : https://hive.cloud.bbc.co.uk/batches/new

Fill name, project and version : Select bbc-blogs project

Click next till you see "Finish" Click on finish

Observe :

Job Information

  • Tests
    Fileds gets highlighted.

2] Batch started in the background but user doesn't know about it

This will confuse user and user will click on "Finish" again now another batch started in the back ground

3] After some time user get back to batches page see he has started many batches

Expected : It should take user to batches page as soon as user clicks on "Finish"

It happen in the chrome and Safari 6 on Mac

Seperate result upload

Ensure that upload test results separate from the running test is supported by the system.

Should not be able to select empty project

Step:

  1. On 'Create a new batch' select project dropdown. Next button is enabled.
  2. Now select the empty option from dropdown. Next button is still enabled.

On submission of form it errors out.

Investigate public folder accessibility

Just re-ran the test to get some fresh logs
I get this: E 2016-03-23 14:36:27: Error uploading artifact executed_script.sh: 795: unexpected token at
followed by a bunch of html
This looks like the most important part of it
This message was deleted

Errno::EACCES in Api::ArtifactsController#create

Permission denied @ dir_s_mkdir - public

maikeru 13:43
For some reason it gitter didn't like it when I tried to format it nicely

jrmhaig 13:54
Does the user running the scheduler have write access to the public directory?

maikeru 13:55
If you mean ./public under the hive-scheduler working directory then yes.
I've done everything with the same (non-root) user (git clone, bundle install, rails etc)
Would it be anything to do with this? http://stackoverflow.com/questions/11864175/paperclip-errnoeacces-permission-denied-system
I.e. are we trying to right to /public instead of ./public for some reason?
Looking in hive-scheduler/app/models/artifact.rb
I see this line
has_attached_file :asset, path: "#{Chamber.env.attachment.path_base}/artifacts/:id/:filename", url: "/artifacts/:id/:filename"
I wonder if we need to add something like :rails_root in the front of the path

jrmhaig 14:00
What do you have in your configuration file for attachement/path_base?

davidbuckhurst 14:00
What is set for the dev environment?

maikeru 14:01
attachment:
storage: filesystem
s3_bucket:
path_base: public
Ah, gitter lost the whitespace
That's the same for both development and production (which just uses default)

maikeru 14:34
I changed path_base to /home/hive/hive-scheduler/public and that worked

davidbuckhurst 15:09
That's a strange one -- I don't get that behaviour at all
I'll copy this conversation into a ticket so we can properly investigate it

Include detailed results in json

When a res file is submitted back on completion of a test the results are displayed as a table on the job page. It would be beneficial if this information could also be included in the json version of the page.

New left-hand menu

Need to rework the menu.

Main view page:

{ Hive Logo }
[ Hive Status ]

  • Build version
  • Queued
  • Running
    [ Recent Projects ]
    List of last n projects run + result (and link)

Miscounted retries

On a second retry the list of tests to retry is duplicated. Ie, if 1 test failed then the retry indicates that there are 2 test to retry.

Link scheduler up to hivemind

Time to abandon the old devicedb

We need to switch the links over to pull stuff back from hivemind.

On the batch page we need: <device_name> ( )

On the model page we need the two bits separately:
Device Type:
Device: <device_name>

Clicking anywhere on <device_name> should send you to the appropriate device in hivemind

Rework top-right bar

Keep the two external links (learn more and raise an issue) but allow customisable links lists

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.