Running Tests

This chapter tells you how to playback tests that you've written/recorded and played back.

* Important Note: Testing Safari when it's running your test application on localhost will give you a Python traceback, this is currently not supported.

Running and Loading

Windmill distinguishes between running and loading tests. This distinction helps when you're debugging and editing tests.

When running a test file or set of test files you are telling the service to run those tests in the IDE as soon as it sends them. Loading tests means that you want all the test to load in to the server and in the IDE as quickly as possible and not run them. This is very useful for when you want to load up a test for editing.

All the test authoring libraries written and maintained by the windmill project support loading and running of tests but windmill's open architecture allows and encourages third parties to develop authoring libraries and it is up to them to implement loading and running.

Running JSON or Python Tests from the Command Line

When windmill starts you can give it the test option. This option allows you to define a single test entry point to be run. This can be a file or directory ( provided the directory is a valid python module, more on this later ).

windmill firefox test=mytest.json

Or load your test with

windmill firefox loadtest=mytest.json

Again, these options can also take test directories if they are valid python modules.

Running or Loading JSON or Python Tests from the Shell

The following methods can take either a single file or a directory and will run each as expected. If you are passing a directory, it can either be absolute or relative.

Running

run_test('mytest.json')

or

run_test('/path/to/test/dir')

Loading

load_test('mytest.json')

or

load_test('/path/to/test/dir')

Command Line

Run Tests: ex. windmill firefox test=/path/to/tests/ http://www.example.com

Load Tests: ex. windmill firefox loadtest=/path/to/tests/ http://www.example.com

Organizing Tests

When you're using multiple directories/modules to run tests those directories/modules need to being with "test". Those directories also need to include a file named __init__.py. This file can be empty.

touch __init__.py

JSON and Python

JSON files are collected and run inside of a python framework. Our JSON based tests don't have their own ESL and framework, they simply leverage the Python framework. This means that if you're going to run more than one test you need to make your test directory a valid python module. Python makes this pretty easy, all you need to do is make sure there is a file named __init__.py in your test directory, as shown above. Now if you pass that directory to windmill as the test point to run it will collect any json or python tests and run them.

If you wish to call JSON tests with functest, you will need to put the following code into __init__.py:

from windmill.authoring import setup_module, teardown_module, enable_collector
enable_collector()

Creating setup/teardown and Managing Test Dependencies

The setup/teardown process is handled in python by functest. You can read in depth about the way tests are collected and advanced setup/teardown in the functest documentation. In this section we'll try to provide you with a crash course in setting up simple setup/teardown and dependencies.

Imagine the following scenario. You have about 30 tests across 3 pages (page1.html, page2.html, page3.html) on your site. Pages 2 and 3 require you to login, page 1 requires you are not logged in. The directory structure of your tests should look something like this;

test_mysite/
           /__init__.py ( Defines any overall setup/teardown for the entire test run)
           /test_page1/
           /test_page1/__init__.py
           /test_page1/test1.json .... test10.json
           /test_logged_in/
           /test_logged_in/__init__.py (Defines a setup_module() which logs in and a teardown_module() which logs out.
           /test_logged_in/test_page2/
           /test_logged_in/test_page2/__init__.py
           /test_logged_in/test_page2/test11.json ... test20.json
           /test_logged_in/test_page3/
           /test_logged_in/test_page3/__init__.py
           /test_logged_in/test_page3/test21.json ... test30.json

This illustrates fairly simple the way you can use the module hierarchy as your dependency structure.

If you need to work on test11.json and tell windmill to run/load it, windmill (through the functest framework) will work it's way up the module chain and find the root module, it will then run any setup_module's that exist in parent directories before it runs test11.json.

Detailing the __init.py __ ::

This file is the core of the setup and tear down functionality, and it is important for you to have an understanding of how it works.

Example:

#Define the JSON actions required to get your app into the state needed (setup)
login = """{"method": "type", "params": {"id" : "loginDialogUsernameInput", "text": "root"}}
{"params": {"milliseconds": 500}, "method": "waits.sleep"}
{"method": "type", "params": {"id" : "loginDialogPasswordInput", "text": "cosmo"}}
{"params": {"milliseconds": 500}, "method": "waits.sleep"}
{"method": "click", "params": {"id" : "loginSubmitButton"}}
{"method":"reWriteAlert", "params":{}}
{"method": "waits.forElement", "params": {"id": "contentWrapper", "timeout": 40000}}"""


#Define the JSON actions to execute after the rest run is finished (teardown)
logout = """{"method": "click", "params": {"link" : "Log out"}}
{"params": {"url": "\/"}, "method": "open"}
{"method": "waits.forElement", "params": {"link" : "Log in to Chandler Server"}}
{"params": {"milliseconds": 1000}, "method": "waits.sleep"}
"""

#Import the windmill functionality for running json files from string variables
from windmill.authoring import RunJsonFile
#Import the windmill functionality needed to setup your extensions
from windmill.bin import shell_objects
#import the windmill object
import windmill

#Define the setup method to be called
def setup_module(module):
    #Load the extensions from the directory 'extensions'
    shell_objects.load_extensions_dir(os.path.join(os.path.dirname(__file__), 'extensions'))
    
    #Run the login JSON actions from the string login defined above
    RunJsonFile('login.json', lines=login.splitlines())()

#Define the teardown method to be called    
def teardown_module(module):
    #Run the logout JSON acions from the string logout defined above
    RunJsonFile('log_out.json', lines=logout.splitlines())()

Running JavaScript Tests

To run your JavaScript tests you can use one of the following methods.

From the shell: run_js_dir('dir/to/js/tests')

From the command line: windmill firefox jstests=./js/test/dir