jeroendoggen / arduino-testsuite

Arduino TestSuite to automate Arduino unit tests

Arduino TestSuite: Automated Arduino Unit Tests

This Python scripts allows automated running of several Arduino unit tests. The testing process is started on the PC but the tests run on the actual Arduino hardware. One set of unit tests is typically used to test one Arduino library.

The following steps are performed for each set of unit tests:

1. The script compiles and uploads an Arduino sketch that contains the unit testing.
2. The unit tests are run on the Arduino board.
3. The results of the test are printed over the serial port and analyzed by the Python script.
4. The script starts the next test, repeating steps 1,2 and 3 for all test that are requested in the configuration file. The script prints a summary showing an overview of all the failed/passed tests in the complete testsuite.

Installation:

• Download the source and run python setup.py install
• Python Package available in the Python Package Index at: http://pypi.python.org/pypi/arduino_testsuite
• Install using pip: pip install arduino_testsuite

Usage:

• Start the program with: python -m arduino_testsuite
• Select the tests you want to run by editing: planned-tests.conf
• Getting help: python -m arduino_testsuite --help
• Post issues to GitHub http://github.com/jeroendoggen/Arduino-TestSuite/issues.

Typical test output:

user@computer:~/tests$ python -m arduino_testsuite
==============================
Planned tests:
 1. examples/floatingPin
 2. examples/inInterval
 3. examples/simpleTest
 4. examples/buildErrorTest
 5. examples/nonExistingFolder
 6. examples/slowTest
 7. examples/verySlowTest
==============================
.... output normally continues for a while ... link to full log below.

https://raw.github.com/jeroendoggen/Arduino-TestSuite/master/resources/example.log

Requirements:

• The unit tests are written with the "Arduino Unit Testing Library": http://code.google.com/p/arduinounit
• The tests also use "Arduino Unit Testing Helper Library": http://code.google.com/p/arduino-unit-test-helper-library
• The code is uploaded to the Arduino board with "Arscons: scons script for Arduino": http://github.com/suapapa/arscons
• Python 2.6+ packages: pyserial

Limitations:

• Currently only tested on Linux and Windows 7
• The program was created with other OS users in mind, so it will eventually get full cross-platform support.

License:

If not stated otherwise arduino_testsuite is distributed in terms of the GPLv2 software license. See COPYING in the distribution for details.

Bug reports:

• Jeroen Doggen [email protected]

Changelog:

0.2.2: Error handling:

• Compile & upload errors
• Timing: timeout for uploading & running of the tests
• Missing files: wrong test folder, no config file
• Hardware: Disconnect Arduino during test

0.2.1: Settings & configuration

• Passing cli arguments & cli help messages
• Settings separated from other code
• Start tests from config file

0.2: First working OOP version

• Subprocess for arscons
• Classes: Test, TestSuite, InfoPrinter
• Divided the code in several modules

0.1: Initial release:

• Call scons
• Run unit test
• Print summary

arduino-testsuite's Issues

Measure flash & RAM overhead

Maybe this has been done in upstream?

Create a basic GUI

Show test status
Enable/disable test
Read a lists, (write a test list?)

Maintain a single copy of arscons SConstruct

Currently every example test contains a copy of arscons SConstruct. Maintaining several copies (e.g. when upgrading to a new version) is time consuming and error prone.

Perhaps the SConstruct file should be moved out of the examples directory?

Web page is broken

The page http://jeroendoggen.github.com/Arduino-TestSuite/ does not access the docs.

Compatibility with "arduinounit 2.0"

Not yet tested, 2.0 probably breaks my code.

https://github.com/mmurdoch/arduinounit

Compatibility with "latest arscons"

New version is broken on Debian Wheezy.
Automated arscons update? (outside of this project?) (Jenkins?)

https://github.com/suapapa/arscons

See:
suapapa/arscons#22
suapapa/arscons#16

Missing is_windows function in Settings causes error in TestSuite

File "C:\Python27\lib\site-packages\arduino_testsuite\testsuite.py",
    line 75, in upload_sketch
if self.config.is_windows():
AttributeError: Settings instance has no attribute 'is_windows'

Run multiple tests in parallel using multiple Arduino's

Connect multiple Arduino boards at the same time to run multiple tests at the same time.

Compile tests while running other tests

Program flow:

• compile test 1
• start compiling all the other tests at the same time
• upload & run test 1
• upload & run test 2
• upload & run test 2

Pylint warning 'ser' attribute defined outside __init__

W: 91,8:TestSuite.analyze_output: Attribute 'ser' defined outside __init__

Integration into Python unit testing:

• http://docs.python.org/2/library/unittest.html
• define standard unit tests that call (parts of?) the Arduino testsuite (this would allow other testing tools to do auto test discovery)

Include support for Windows

• The default port (/dev/ttyUSB0) is Unix-specific, on Windows it should probably be COMx (the first Arduino comes up as COM5 on my Windows 7 box). The port can be overridden using the -p command line parameter but it would be good if there is a reasonable default
• scons.bat needs to be on the path (it is installed into <python_installation_dir>\Scripts)
• To execute scons on Windows subprocess.Popen needs to be passed scons.bat (and not simply scons as currently). It also needs to run scons.bat from the directory in which it resides, by passing it: cwd=<python_installation_dir>\Scripts
• To kill a process portably use Popen.terminate() rather than os.kill() in testhelper.py
• When a parent process (python -m arduino_testsuite here) is using a serial port then a child process (scons in this case) gets an 'Access Denied' error when it tries to use it on Windows. The fix for this may be to open the serial port after the test upload to retrieve the test results and close it again before the next upload.

Logging Interface

• Use logging interface http://docs.python.org/2/library/logging.html
• Send log info to console
• Send log to a file
• Send out email with summary of tests http://docs.python.org/2/howto/logging.html#useful-handlers

Build-system flexibility/independence

Currently we build the the Arduino source code with arscons.

It would be nice to make the testsuite program more flexible towards build-systems (independent of) by detecting the existence of a makefile, SConstruct, CMake or other similar files and just use them to build & upload the code.

If none of these files exist, we could fall back to building the Arduino sketch with pyavrutils (which is based on arscons).

Links:
https://github.com/mjoldfield/Arduino-Makefile
https://github.com/queezythegreat/arduino-cmake
https://github.com/ponty/pyavrutils
https://github.com/amperka/ino