seattletestbed / utf Goto Github PK

View Code? Open in Web Editor NEW

1.0 1.0 9.0 35 KB

Unit Test Framework for SeattleTestbed

License: MIT License

Python 92.42% Shell 7.58%

utf's People

Contributors

Stargazers

Watchers

Forkers

choksi81 priyam3nidhi karthikgvss kellender awwad rayfok lukpueh sandro8620 kcg295

utf's Issues

Unit test timeout fail after r6484

All other unit test passed except timeout after r6484:

Running: ut_nm_validate_connection_timeout.py               [ PASS ]
Running: ut_nm_listfilesinvessel_clean_state_add.py         [ PASS ]
Running: ut_nm_changeadvertise_invalidsequenceidnegative.py [ PASS ]
Running: ut_nm_changeadvertise_invalidsequenceid.py         [ PASS ]
Running: ut_nm_changeownerinfo_longstring.py                [ PASS ]
Running: ut_nm_changeowner_expired.py                       [ PASS ]
Running: ut_nm_private_retrievefile.py                      [ PASS ]
Running: ut_nm_listfilesinvessel_add_and_remove.py          [ PASS ]
Running: ut_nm_addfiletovessel_emptyfilename.py             [ PASS ]
Running: ut_nm_changeowner_validsequenceid.py               [ PASS ]
Running: ut_nm_startstopvessel.py                           [ PASS ]
Running: ut_nm_setrestrictions.py                           [ PASS ]
Running: ut_nm_changeownerinfo.py                           [ PASS ]
Running: ut_nm_timeout2.py                                  [ PASS ]
Running: ut_nm_addfiletovessel_invalidfilename.py           [ PASS ]
Running: ut_nm_changeadvertise_invalidsequenceidstring.py   [ PASS ]
Running: ut_nm_addfiletovessel_parentdir.py                 [ PASS ]
Running: ut_nm_addfiletovessel_emptyfile.py                 [ PASS ]
Running: ut_nm_deletefileinvessel.py                        [ PASS ]
Running: ut_nm_rawsaynonexistentmethod.py                   [ PASS ]
Running: ut_nm_private_deletefile.py                        [ PASS ]
Running: ut_nm_changeusers_invalidActions.py                [ PASS ]
Running: ut_nm_private_listfiles.py                         [ PASS ]
Running: ut_nm_fastclient.py                                [ PASS ]
Running: ut_nm_addfiletovessel_duplicate_filename.py        [ PASS ]
Running: ut_nm_changeadvertise.py                           [ PASS ]
Running: ut_nm_changeownerinfo_emptystring.py               [ PASS ]
Running: ut_nm_changeusers.py                               [ PASS ]
Running: ut_nm_resetvessel_multireset.py                    [ PASS ]
Running: ut_nm_private_resetvessel.py                       [ PASS ]
Running: ut_nm_signedsayemptymethodname.py                  [ PASS ]
Running: ut_nm_timeout.py                                   [ FAIL ]

Standard out :
..............................Produced..............................
Did not timeout!

..............................Expected..............................

None

Running: ut_nm_getresources.py                              [ PASS ]
Running: ut_nm_changeowner_invalidkey.py                    [ PASS ]
Running: ut_nm_validate_connection_limit.py                 [ PASS ]
Running: ut_nm_changeownerinfo_unicode.py                   [ PASS ]
Running: ut_nm_changeownerinfo_escapecharacters.py          [ PASS ]
Running: ut_nm_resetvessel.py                               [ PASS ]
Running: ut_nm_signedsaynonexistentmethod.py                [ PASS ]
Running: ut_nm_simple.py                                    [ PASS ]
Running: ut_nm_joinsplitvessels.py                          [ PASS ]
Running: ut_nm_changeowner.py                               [ PASS ]
Running: ut_nm_rawsayemptymethodname.py                     [ PASS ]
Running: ut_nm_readvessellog.py                             [ PASS ]
Running: ut_nm_getoffcut.py                                 [ PASS ]
Running: ut_nm_addfiletovessel.py                           [ PASS ]
Running: ut_nm_listfilesinvessel.py                         [ PASS ]
Running: ut_nm_private_addfile.py                           [ PASS ]
Running: ut_nm_listfilesinvessel_no_files.py                [ PASS ]

Now stopping subprocess: ut_nm_subprocess.py

Make sure all integration tests inform current project members

Not all core project members know when components go down, so we need to make sure that everyone does so that we can react to problems quicker.

Modify test_time_tcp.py to provide more information when the test fails.

Currently the test_time_tcp.py seems to fail almost every hour due to some time descrepancy. We need to modify the test to see why its failing and/or provide more information when the test fails, so we can better understand the problem

UTF error traceback prints escapes

When UTF prints errors, it prints out carriage returns and line breaks instead of escaping them.

This should be changed.

Some repy tests seems to fail on windows xp.

Some repy tests produce a 'fail' notification on windows machine because the test result expects a return value of '' when None is produced instead. The noted tests are:

ut_repytests_randomratetest.py
ut_repytests_testmemoryallocwithexceptions2.py
ut_repytests_testmemoryquota.py

continuous build nodemanager tests fail more since UTF integration

The nodemanager tests of the continuous build fail more since the UTF was added.

http://blackbox.cs.washington.edu/~continuousbuild/

My impression from the logs there is that it's probably just the tests running before the nodemanager has fully started. A simple sleep or a loop-with-sleep while checking on the nodemanager's progress starting up is probably needed in the startup script for those tests. I think this used to be done, but maybe it got lost in the migration to utf.

Some network semantics tests rely on ports they are not given

Some of the tests seem to use ports not given by restrictions.default which causes failures. I noticed this from test_openconn_nolistener2.py

Threaderr test will fail if run_tests was not previously run

The restrictions file needs to be pre-processed so that the test will work.

Create start / stop integration tests...

We should have some integration tests that start / stop seattle a bunch of times. I did this today and found two separate bugs (#599 and #600)!

Unwanted unit test exception for python 2.7

The following error is often raised when the unit tests are run on Python 2.7. It has been around for a while and should be fixed as it makes it hard to run unit tests on the latest Python.

monzum@TestUbuntu:~/exdisk/work/repy_v1_client$ python2 utf.py -Tm nm
Testing module: nm
    Running: ut_nm_addfiletovessel.py                           [ FAIL ] [ 28.45s ]
--------------------------------------------------------------------------------
Standard error :
..............................Produced..............................
Exception safety_exceptions.RunBuiltinException: RunBuiltinException() in <bound method Popen.__del__ of <subprocess.Popen object at 0x9b9754c>> ignored
Exception safety_exceptions.RunBuiltinException: RunBuiltinException() in <bound method Popen.__del__ of <subprocess.Popen object at 0x9b9754c>> ignored

testprocess.py is a python file?

Why is testprocess a python file? It has nmclient.repy and session.repy embedded in the file that is committed in SVN. What the heck is going on?

Seash unit test gets false negatives

The seash unit tests fail when they should not. We get the following kind of error message:

Standard output:
.........................Produced.....................

.........................Expected.....................
None

This issue seems to occur only on Mac. This could be an issue with the utf module.

Need support added to choose layers when running utf tests...

In repyV2, one can easily interpose security layers, shims, models from CheckAPI, etc. into any program. We need to add support to utf to allow unit tests to be run with different modules.

For example, here is how I run the allpairspingv2 program with the check_api portability verification.

python repy.py restrictions.full encasementlib.repy check_api.repy dylink.repy librepy.repy allpairspingv2.repy 12345

Without check_api verification, it looks like this:

python repy.py restrictions.full encasementlib.repy dylink.repy librepy.repy allpairspingv2.repy 12345

Note: more than one security layer can be used at a time. I believe that all security layers will need to be loaded between the encasementlib and dylink. I'm not sure if this is true for shims. (Monzur: please chime in)

verifyprocess.py script needs to be upgraded

The script needs to be able to log information regarding how many software updater / node manager instances are running (and PIDs for additional information).

Non-existant child process causes run_tests to produce an error...

If the repy tests are run and there is an error that is causing repy to abort before forking a child process, this messes up run_tests. Namely, the odd-ball test that kills the monitor process errors out inside of run_tests. This prevents the person who is running the tests from seeing the error that is causing the repy errors.

[Newcomer] Unit tests are not run in alphabetical order

Unit tests run out of order. This makes it hard to read!

Architect an improved unit testing infrastructure for the Seattle project

The current unit testing system was adequate at the time of its creation, but is now an inefficiency in developing the Seattle software. An improved unit testing system would define standard protocols and conventions on which to run automated tests. As the project is seeing increasing exposure, it is crucial that we be able to easily run all unit tests to make sure that bugs are detected early and fixed before a general release.

run_tests.py doesn't run python tests correctly.

run_tests.py doesn't run python tests correctly unless they are z_ tests. For example, a test py_n_foo.py will be run as: python repy.py py_n_foo.py.

This wasn't detected because this printed the usage message and this produces output (which passes!).

Integration tests for ntp servers...

We should create a test that checks the time needed to talk to the (publicly run) NTP servers we use in our NTP library.

This is related to / motivated by SeattleTestbed/attic#1363.

Check the number of openDHT servers and email if it's less than 100.

Arvind emailed and requested:

BTW, can you make one tweak to your check-up script that sends out an email if there are less than 100 servers listed on http://opendht.org/servers.txt

I can rig up something to do it, but I figured it might be easier for you to include that in your script. My hope is to have 250+ nodes on OpenDHT and I have a few scripts for keeping them up. I just wanted an independent checking script that might not get nixed by say the controlling machine going down.

We should make this change to the integration test.

server.cfg for remote testing format extensions

The server.cfg file needs to ignore blank lines, and commented lines that start with '#' so that server.cfg can be commented and formatted well.

UTF -- Add functionality for setup and shutdown scripts for individual tests

Setup and shutdown scripts functionality still needs to be created for individual tests.

UTF doesn't indicate which test is currently running

Right now when UTF is running a test there is no information printed. Once the test finishes, it prints the name and the result.

It would be nice for debugging tests that are timing out, and other weirdness if it would print the test name before it starts running.

remove obsolete tests from repyV2...

There are a lot of tests in the V2 tests directory that are unused. We should remove / port these.

utf should print an error for an empty module name...

The following output is confusing. It should simply say there is no such module if no tests exist.

justinc@nara:~/test2> python utf.py -m jkasdfjkasjkdf
Testing module: jkasdfjkasjkdf
justinc@nara:~/test2>

Need a good way to run multiple test instances on a machine...

It should be possible to run multiple test instances on the same system. This allows us to use systems for automated and manual testing.

ut_registerhttpcallback_dictionary_check.repy Error

following line makes error.

recvd_content = httpretrieve_get_string('http://127.0.0.1:12345/fake/path.html', http_query, http_post)

Error:
Uncaught exception! Following is a full traceback, and a user traceback.
The user traceback excludes non-user modules. The most recent call is displayed last.

Full debugging traceback:
"repy.py", line 189, in main
"/share/trunk/test_server/virtual_namespace.py", line 116, in evaluate
"/share/trunk/test_server/safe.py", line 304, in safe_run
"ut_registerhttpcallback_dictionary_check.repy", line 2021, in

User traceback:
"ut_registerhttpcallback_dictionary_check.repy", line 2021, in

Exception (with type 'exceptions.KeyError'): 'Content-Length'

Terminated

*note
I used the newest httpretrieve.repy and httpserver.py from the turnk.

many test_nat_servers_running.py using all memory on blackbox

blackbox.cs ran out of memory (including swap) and was refusing to allocate memory to new processes. My guess is that the culprit is the couple hundred instances of test_nat_servers_running.py that are running:

geni@blackbox:~$ ps -ef | grep "/usr/bin/python /home/integrationtester/cron_tests/nat_tests/test_nat_servers_running.py" | wc -l
257
geni@blackbox:~$ free
             total       used       free     shared    buffers     cached
Mem:        254016     248412       5604          0       1344      15688
-/+ buffers/cache:     231380      22636
Swap:       746980     746908         72

apache was the next biggest culprit, so maybe my seattlegeni testing on blackbox was the problem and the many nat test processes were intended. After restarting apache, here's the free memory:

jsamuel@blackbox:~$ free
             total       used       free     shared    buffers     cached
Mem:        254016     171804      82212          0       1520      24196
-/+ buffers/cache:     146088     107928
Swap:       746980     700772      46208

There's not a lot of memory to start with here, though, so if those nat tests aren't supposed to be hanging around, they should be cleared up.

Some unit tests fail, even after waiting

The following unit tests fail even after waiting. Tested when building a new installer for Mac.

ut_repytests_testconcurrentopenconns.py
ut_repytests_testnetbothdup.py
ut_repytests_testnetconndup.py
ut_repytests_testnetmessdup.py
ut_repytests_testrecvmessfunctions.py
ut_repytests_testwaitforconnfunctions.py

random rate test often fails on Windows XP...

On Windows XP, it is common for the ut_repytests_randomratetest.py to fail. The root cause is unclear.

Integration tests aren't working...

I haven't had any emails and opendht is down. We need to know if our integration tests fail. They don't do us any good if they don't work.

Integration test to periodically pull nodemanager logs from nodes

I propose having an integration test that will pull nodemanager logs from our nodes (softwareupdater.old and nodemanager.old, *.new files if they exist as well) periodically, to potentially uncover issues that we may not be aware of. Monzur mentioned he had a script that was used for this purpose before, so if possible we should get that integrated into our monitoring test suite and into our repository.

If utf hangs, it does not print test name...

The utf module does not print the test that it is running until after it finishes. This means that if the test hangs, you cannot determine the culprit easily.

UTF to support checking strings that span multiple lines

The strings that you tell UTF to check for can only be one line long. They cannot span multiple lines. For example, for the next code block, you cannot test for the entire string. You can only test for specific segments of the code, e.g. 'hello world' or 'this is a test message'. The newline character cannot be represented in the command used to indicate the test message.

# These work:
#pragma out hello world
# -- OR --
#pragma out this is a test message

# This will fail
#pragma out hello world\nthis is a test message

# Test case:
print "hello world"
print "this is a test message"

UTF test has error on windows xp when the test ends.

After the UTF has run the tests of a module (for example uft.py -m nm). The test module tries to kill a subprocess and fails. The error is noted below:

Now killing subprocess: ut_nm_subprocess.py
Internal error. Trace:
--------------------------------------------------------------------------------
Traceback (most recent call last):
  File "utf.py", line 596, in <module>
    main()
  File "utf.py", line 173, in main
    test_module(module_name, module_file_list)
  File "utf.py", line 284, in test_module
    os.kill(sub.pid, signal.SIGTERM)
AttributeError: 'module' object has no attribute 'kill'

run_tests stderr stripping issue

exec_command(command) parses for "Terminated" token which is specific only to some shells. ZSH for instance acts differently, producing:

zsh: terminated python ./repy.py ./restrictions.fullcpu re_semaphore_simple.py

Code responsible for this:

236 # I want to look at the last line. it ends in \n, so I use index -2
237 if len(theout.split('\n')) > 1 and theout.split('\n')[-2].strip() == 'Terminated':

P. S. run_tests.py is full of issues. I might group all of them in a single ticket.

preparetest -t should include the software updater tests.

When preparetest -t is run, it should include the necessary files to run the software updater tests. This will make it easier for us to test that the software updater is working.

This will also include the tests in the installers built by make_base_installers when they are told to include the unit tests.

[Newcomer] Need an integration test for Zenodotus service

We have a Zenodotus service running on blackbox.poly.edu. We need an integration test to ensure that the zenodotus service is up and working correctly.

More information on the Zenodotus service can be found here:

UTF accepts bogus and zero arguments

UTF should reject invalid arguments (including no arguments).

It current accepts:

"python utf.py" --runs all tests
"python utf.py asd" --runs all tests

utf ignores second and further arguments to #pragma repy

r6986 removed blanket dylink support for unit tests, claiming some tests would break otherwise. Unfortunately, it also removed a perfectly valid statement that allowed for the inclusion of more parameters in the #pragma repy directive than only a restrictions file.

Additional parameters were possible before, and the comment to r6986 only extends to the dylink line added in r6926, so I conclude this removal was done in error. I'm adding the line back in.

Error in verifyfiles.py

There is a simple typographical error in verifyfiles.py (a script that checks for consistency of a seattle install).

Produces the following error msg:

File "verifyfiles.py", line 635, in <module>
     main()
   File "verifyfiles.py", line 620, in main
     look_dir = sys.argv[  File "verifyfiles.py", line 576, in verify_files
     filestatus_dict[reqfile](3])
)[0] + " not found"
 KeyError: 'xmlrpc_common.repy

Ability to run individual unit tests

Something similar to run_tests.py but with the ability to run a selective test, or subset of tests. Now, if you only fail a few unit tests, it is necessary to re-run all of them. This is especially annoying when working with slow systems. (E.g. Windows under virtualization or mobile devices).

[Newcomer] Not all UTF options are printed out

UTF has many useful options (like -T to show how much time each test took) that are hidden in the source file, and not show in the usage string.

Interrupting node manager tests leaves node manager running...

If you interrupt utf.py -m nm it will leave the node manager running. While not a major problem, this is annoying.

Nodes with stopping processes

I ran testing on problematic nodes (that I brought up a few days ago using start_seattle.sh). Here are some of the nodes with a long uptime but, failed tests.
(longest *** separates hosts)
shorter *** sequence separates separate problematic runs for that host: first: run to check uptime, second - run regular test (never mind that 1st and 2nd are separate runs), third - after starting them up (just indicates that they were brought up)

Testing on node: pl1.6test.edu.cn

REACHED!
file_checker_finished
[Node Manager is not running.
ProcessCheckerFinished
08:00:29 up 6 days, 17:14, 0 users, load average: 2.20, 1.69, 1.60
PROBLEM OCCURRED!
PO! pl1.6test.edu.cn

Testing on node: pl1.6test.edu.cn

REACHED!
file_checker_finished
NodeManager Node Manager is not running.
ProcessCheckerFinished
PROBLEM OCCURRED!
PO! pl1.6test.edu.cn