Coder Social home page Coder Social logo

planterbox's People

Contributors

ltamshen avatar npilon avatar wholewheattoast avatar

Stargazers

avatar avatar

Watchers

avatar avatar avatar

Forkers

gcarothers wholewheattoast ltamshen

planterbox's Issues

Planterbox does not support 'But' in step

In addition to words like, 'And', 'Given', 'Then', 'But' is typically supported.

https://github.com/cucumber/cucumber/wiki/Given-When-Then#and-but

Example use in a Scenario

Scenario: A Landing Page without Client Matter renders
        Given I sign in
        Then I should see "Welcome Enterprise Test" within 5 seconds
        But there should not be an element matching $(".fa-clock-o")

Fitst test uses, 'And', second, 'But';

(deus_lex) shawn@debian8 ~/source/deus_lex/winslow> nose2 -c unittest-sauce-law-stage.cfg winslow.tests.features:landing_page.feature:0 -v
Landing Page Permutations (winslow.tests.features:landing_page.feature:0)
Test each permutation of the Landing Page; ... ok

----------------------------------------------------------------------
Ran 1 test in 21.207s

OK
(deus_lex) shawn@debian8 ~/source/deus_lex/winslow> nose2 -c unittest-sauce-law-stage.cfg winslow.tests.features:landing_page.feature:0 -v
Landing Page Permutations (winslow.tests.features:landing_page.feature:0)
Test each permutation of the Landing Page; ... ERROR

======================================================================
ERROR: Landing Page Permutations (winslow.tests.features:landing_page.feature:0)
Test each permutation of the Landing Page;
----------------------------------------------------------------------
Scenario: A Landing Page without Client Matter renders
    Given I sign in
    Then I should see "Welcome Enterprise Test" within 5 seconds
    But there should not be an element matching $(".fa-clock-o")
Traceback (most recent call last):
  File "/home/shawn/source/planterbox/planterbox/__init__.py", line 196, in run_scenario
    step_fn, step_arguments = self.match_step(step)
  File "/home/shawn/source/planterbox/planterbox/__init__.py", line 136, in match_step
    raise UnmatchedStepException(step)
UnmatchedStepException:         But there should not be an element matching $(".fa-clock-o")

----------------------------------------------------------------------

Possible mangeling of scenario outline scenario name

https://qa.tool.lexmachina.com/report/364270

The scenario outline looks malformed.

Feature
Walk And Shoot
Scenario
Scenario Outline: Winslow Help Center pages - {u'': u'', u'url': u'help/quick-tips', u'wait_condition': uspan:contains('Tip 1: Analyze Judge Behavior'), u'page': u'Help Quick Tips page'}
Step
Scenario Outline: Winslow Help Center pages <- {u'': u'', u'url': u'help/quick-tips', u'wait_condition': u"span:contains('Tip 1: Analyze Judge Behavior')", u'page': u'Help Quick Tips page'}

The uspan:contains('Tip 1: Analyze Judge Behavior') should look like u'span:contains('Tip 1: Analyze Judge Behavior')

Full traceback

Scenario Outline: Winslow Help Center pages <- {u'': u'', u'url': u'help/quick-tips', u'wait_condition': u"span:contains('Tip 1: Analyze Judge Behavior')", u'page': u'Help Quick Tips page'}
    Given I point browser to "help/quick-tips"
    And I dismiss notifications
    Then I take a screenshot
Traceback (most recent call last):
  File "/home/lex/local/lib/python2.7/site-packages/planterbox/feature.py", line 215, in run_scenario
    step_fn(self, *step_arguments)
  File "/home/lex/src/deus_lex/qatool/qatool/tests/features/__init__.py", line 175, in record_visdiff_details
    common_planterbox.record_visdiff_screenshot(test, _scenario_id)
  File "/home/lex/src/deus_lex/deus_lib/deus_lib/common_planterbox.py", line 149, in record_visdiff_screenshot
    screenshot = take_screenshot(test, time_of_shot)
  File "/home/lex/src/deus_lex/deus_lib/deus_lib/common_planterbox.py", line 214, in take_screenshot
    cloud_storage_screenshot,)
  File "/home/lex/src/deus_lex/bias/bias/webscraping.py", line 75, in upload_to_gs_bucket
    gs_key.set_contents_from_file(upload_file, rewind=True)
  File "/home/lex/local/lib/python2.7/site-packages/boto/gs/key.py", line 580, in set_contents_from_file
    self.send_file(fp, headers, cb, num_cb, size=size)
  File "/home/lex/local/lib/python2.7/site-packages/boto/gs/key.py", line 313, in send_file
    hash_algs=hash_algs)
  File "/home/lex/local/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
    query_args=query_args
  File "/home/lex/local/lib/python2.7/site-packages/boto/s3/connection.py", line 665, in make_request
    retry_handler=retry_handler
  File "/home/lex/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/home/lex/local/lib/python2.7/site-packages/boto/connection.py", line 940, in _mexe
    request.body, request.headers)
  File "/home/lex/local/lib/python2.7/site-packages/boto/s3/key.py", line 884, in sender
    response.status, response.reason, body)
  File "/home/lex/local/lib/python2.7/site-packages/boto/exception.py", line 296, in __init__
    super(StorageResponseError, self).__init__(status, reason, body)
  File "/home/lex/local/lib/python2.7/site-packages/boto/exception.py", line 115, in __init__
    h.parseString(self.body)
  File "/home/lex/local/lib/python2.7/site-packages/boto/handler.py", line 60, in parseString
    return self.parser.parse(StringIO(content))
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 107, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 210, in feed
    self._parser.Parse(data, isFinal)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 1441: ordinal not in range(128)

And the original Scenario
https://github.com/LexMachinaInc/deus_lex/blob/production/qatool/qatool/tests/features/winslow_walk_and_shoot.feature#L411

Scenario Outline: Winslow Help Center pages
        Given I point browser to "<url>"
        And I dismiss notifications
        Then I take a screenshot

        Examples:
            | page                        | url                    | wait_condition                                 |
            | Help Documentation page     | help/documentation     | span:contains('Data Coverage')                 |
            | Help Quick Tips page        | help/quick-tips        | span:contains('Tip 1: Analyze Judge Behavior') |
            | Help Video Tutorials page   | help/video-tutorials   | .caption:contains('Size Up Counsel')           |
            | Help Published Reports page | help/published-reports | .text-region:contains('PTAB Report 2015')      |
            | Help Quick Tip Analyze Judge Behavior page | help/quick-tips/analyze-judge-behavior | span:contains('Case Timing by Judge') |
            | Help Video Tutorial Size Up Counsel page | help/video-tutorials/size-up-counsel | .scrollable-content-layout:contains('Accelerate with Legal Analytics: Size Up Counsel') |

Configuration values defined below [planterbox-summary] block do not load.

In refactoring some cfg's I happened to move the position of

[planterbox-summary]
always-on = True

From the bottom of a config file closer to top.

All config values below it are not loaded.

Able to verify by running nose2 with -D and self["some_expected_value"] not being defined.

Is this expected behavior?

Allow a Planterbox Scenario to be run by Scenario name

Sometimes feature files get big.
When one scenario in a feature gets updated, it's a pain to manually count each test in the file to find it's ordinal number so you can run just that test.

@npilon suggests
"The scenario name must be unique in the file and allow it to be specified on the command line"

So I'm guessing this makes running a test like;
nose2 winslow.tests.features:walk_and_shoot.feature:0
look somthing like;
nose2 winslow.tests.features:walk_and_shoot.feature:"Winslow pages requiring user to be signed out"
or;
nose2 winslow.tests.features:walk_and_shoot.feature --scenario "Winslow pages requiring user to be signed out"

This would make it a bit easier to run a specific scenario in a feature.

Bad scenario outline variables hang run

Running (deus_lex) shawn@debian8 ~/source/deus_lex/winslow> nose2 -c ../qa_tool/webtests/shawn_sauce_windows_7_c44_law-stage.cfg winslow.tests.features --plugin=nose2.plugins.mp -N=2 --log-level debug -v

Nose2 / planterbox would hang.

running with --log-level debug

DEBUG:nose2.main:Internal Error
Traceback (most recent call last):
  File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/nose2/main.py", line 265, in runTests
    self.result = runner.run(self.test)
  File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/nose2/runner.py", line 53, in run
    executor(test, result)
  File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/nose2/runner.py", line 41, in <lambda>
    executor = lambda suite, result: suite(result)
  File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/unittest2/suite.py", line 87, in __call__
    return self.run(*args, **kwds)
  File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/unittest2/suite.py", line 126, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/suite.py", line 108, in run
    test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
    return self.run(*args, **kwds)
  File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/planterbox/__init__.py", line 168, in run
    result=result,
  File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/planterbox/__init__.py", line 248, in run_outline
    example_scenario = substitute_steps(scenario, example)
  File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/planterbox/__init__.py", line 432, in substitute_steps
    for step in scenario
KeyError: u'see_all_link'
Internal Error: runTests aborted: u'see_all_link'

Only seems to hang when running in parallel.

Resolving error (typo in variable) in scenario outline solved issue.

Possibly need to better catch exception in scenario outlines.

Consider support for comments

Running nose2 with planterbox-webdriver tests that include comments produces the following output:

(planterbox-webdriver)shawns-MacBook-Air:planterbox-webdriver shawn$ nose2
..............FE....F..F.
======================================================================
ERROR: I should see, I should not see (planterbox_webdriver.tests.test_webdriver:basics.feature)
# Forms stuff
# Requires known failure support
# Requires Multi-line step support
# alerts
#possible bug here?
# Requires known failure support
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/shawn/source/planterbox-webdriver/lib/python2.7/site-packages/planterbox/__init__.py", line 242, in run_scenario
    step_fn, step_arguments = self.match_step(step)
  File "/Users/shawn/source/planterbox-webdriver/lib/python2.7/site-packages/planterbox/__init__.py", line 192, in match_step
    raise UnmatchedStepException()
UnmatchedStepException

======================================================================
FAIL: I should see, I should not see (planterbox_webdriver.tests.test_webdriver:basics.feature)
# Forms stuff
# Requires known failure support
# Requires Multi-line step support
# alerts
#possible bug here?
# Requires known failure support
----------------------------------------------------------------------
Scenario: Combo boxes fail
    Given I go to "basic_page"
    Then I should not see option "Mercedes" in selector "car_choice"
Traceback (most recent call last):
  File "/Users/shawn/source/planterbox-webdriver/lib/python2.7/site-packages/planterbox/__init__.py", line 246, in run_scenario
    step_fn(self, *step_arguments)
  File "/Users/shawn/source/planterbox-webdriver/planterbox_webdriver/webdriver.py", line 414, in select_does_not_contain
    test.assertIsNone(option_in_select(test.world.browser, id_, option))
AssertionError: <selenium.webdriver.remote.webelement.WebElement object at 0x1098d8850> is not None

======================================================================
FAIL: I should see, I should not see (planterbox_webdriver.tests.test_webdriver:basics.feature)
# Forms stuff
# Requires known failure support
# Requires Multi-line step support
# alerts
#possible bug here?
# Requires known failure support
----------------------------------------------------------------------
Scenario: test labels
    When I go to "basic_page"
    And I click on label "Favorite Colors:"
    Then element with id "fav_colors" should be focused
    And element with id "bio_field" should not be focused
Traceback (most recent call last):
  File "/Users/shawn/source/planterbox-webdriver/lib/python2.7/site-packages/planterbox/__init__.py", line 246, in run_scenario
    step_fn(self, *step_arguments)
  File "/Users/shawn/source/planterbox-webdriver/planterbox_webdriver/webdriver.py", line 286, in element_not_focused
    test.assertEqual(elem, focused)
AssertionError: <selenium.webdriver.remote.webelement.WebElement object at 0x1098d86d0> != <selenium.webdriver.remote.webelement.WebElement object at 0x1098f7ed0>

======================================================================
FAIL: I should see, I should not see (planterbox_webdriver.tests.test_webdriver:basics.feature)
# Forms stuff
# Requires known failure support
# Requires Multi-line step support
# alerts
#possible bug here?
# Requires known failure support
----------------------------------------------------------------------
Scenario: test date input
    When I go to "basic_page"
    And I fill in "dob" with "1900/01/01"
Traceback (most recent call last):
  File "/Users/shawn/source/planterbox-webdriver/lib/python2.7/site-packages/planterbox/__init__.py", line 246, in run_scenario
    step_fn(self, *step_arguments)
  File "/Users/shawn/source/planterbox-webdriver/planterbox_webdriver/webdriver.py", line 235, in fill_in_textfield
    test.assertTrue(field, 'Can not find a field named "%s"' % field_name)
AssertionError: Can not find a field named "dob"

----------------------------------------------------------------------
Ran 25 tests in 29.038s

FAILED (failures=3, errors=1)
  1. Comments are echoed out which is a bit odd
  2. They are printed for each failing test 'report'

Comments can be useful and I've used them sparingly in lettuce-webdriver tests. One useful example;

Scenario: 65. A Shared Page
        # This will fail every two weeks and need to be regenerated in steps.py
        ...

A comment like this can be useful if someone besides myself takes over test maintenance and is looking for information to why that particular test has started failing.

Proposal to treat each run through a scenario outline as a distinct scenario.

Currently when you iterate through a scenario outline like:

Scenario Outline: Winslow pages requiring user to be signed out
        Given I go to "<foo>"
       Then there should be "<bar>"

        Examples:
            | foo                  |  bar | 
            | example.com | Hi |
            | google.com   | how are you |

The hooks ie @hook('before', 'scenario') only fire at the beginning/end of the entire Scenario Outline.

In my mind it seems that each 'run' through a scenario outline should be treated as a distinct scenario. Behavior that should be triggered before or after a scenario (as defined by the hooks) should be triggered for each outline example row.

The gherkin syntax isn't 100% clear on this https://github.com/cucumber/cucumber/wiki/Scenario-Outlines but the example at the top of the page that starts with "Copying and pasting scenarios to use different values quickly becomes tedious and repetitive:" leads some credibility to my assumptions.

Let me know if you need more details.

Test report format repeats first line of Feature summery description over and over

Example output

(deus_lex) shawn@debian8 ~/source/deus_lex/winslow> nose2 -c unittest-sauce-law-stage.cfg winslow.tests.features:auth.feature -v
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... FAIL
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok

Example Feature summery;

Feature: Winslow Authentication Testing
    Sign in as a user with an incorrect email
    Sign in as a user with an incorrect password
    Sign in as a registered user
    Sign in as a user with an expired token
    Sign in as a user who's account has been disabled
    Sign in as a user who's trail has expired
    Register as a user who is already singed in
    Attempt to reset a password when non is set

Instead of showing the same first line of the Feature summery show the relevant Scenario name?

First line of Scenario Outline reporting doesn't include details.

Given the following test;

Feature: Test for Testing
    Tests for testing

    Scenario Outline: Test if scenario outlines work properly
        And I point browser to "<url>"
        Then there should be an element matching $("<wait_condition>") within 5 seconds
        And I capture a screenshot

        Examples: 
            | page            | url                     | wait_condition  |
            | example         | https://www.example.com | .foo            |
            | another example | https://www.example.com | #bar            |

You'll get the following output.

(deus_lex) shawn@debian8 ~/source/deus_lex/winslow> nose2 -c ../qa_tool/webtests/planterbox.cfg -c ../qa_tool/webtests/sauce-shawn.cfg -c ../qa_tool/webtests/law-prod.cfg winslow.tests.features:test.feature:0 -v
Test for Testing (winslow.tests.features:test.feature:0)
    Scenario Outline: Test if scenario outlines work properly ... FAIL
Test for Testing (winslow.tests.features:test.feature:0)
    Scenario Outline: Test if scenario outlines work properly <- {u'': u'', u'url': u'https://www.example.com', u'wait_condition': u'#bar', u'page': u'another example'} ... FAIL

======================================================================
FAIL: Test for Testing (winslow.tests.features:test.feature:0)
Tests for testing
----------------------------------------------------------------------
Scenario Outline: Test if scenario outlines work properly
    And I point browser to "https://www.example.com"
    Then there should be an element matching $(".foo") within 5 seconds
Traceback (most recent call last):
  File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/planterbox/__init__.py", line 202, in run_scenario
    step_fn(self, *step_arguments)
  File "/home/shawn/forks/planterbox-webdriver/planterbox_webdriver/css_selector_steps.py", line 89, in wait_for_element_by_selector
    test.assertTrue(elems)
AssertionError: [] is not true

======================================================================
FAIL: Test for Testing (winslow.tests.features:test.feature:0)
Tests for testing
----------------------------------------------------------------------
Scenario Outline: Test if scenario outlines work properly <- {u'': u'', u'url': u'https://www.example.com', u'wait_condition': u'#bar', u'page': u'another example'}
    And I point browser to "https://www.example.com"
    Then there should be an element matching $("#bar") within 5 seconds
Traceback (most recent call last):
  File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/planterbox/__init__.py", line 202, in run_scenario
    step_fn(self, *step_arguments)
  File "/home/shawn/forks/planterbox-webdriver/planterbox_webdriver/css_selector_steps.py", line 89, in wait_for_element_by_selector
    test.assertTrue(elems)
AssertionError: [] is not true

----------------------------------------------------------------------
Ran 2 tests in 50.869s

FAILED (failures=2)

The second test includes the deatils of that particular scenario outline step <- {u'': u'', u'url': u'https://www.example.com', u'wait_condition': u'#bar', u'page': u'another example'} but the first does not.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.