Making applications “behave”

After the previous post wherein we setup gnome-continuous, the next phase of my project involves writing tests for applications installed in it. I started with gnome-weather.
To start with one can upgrade the previously installed image to latest version from build.gnome.org with “ostree admin upgrade” (restart required).
There are majorly 2 ways to approach writing tests.

  • By playing around directly with the installed tests located at “/usr/libexec/installed-tests/<application-name>
  • By writing “features” and “steps” for testing the application via “behave

In my case the former option doesn’t quite work as much as the later. A major could be that I find it faster to work on my local machine and then test the tests on the gnome-continuous platform. And also because the VM tends to undergo major connectivity issues every now and then. So, this post will majorly talk about “behave”.

We segregated a list of components which need work, along with the priority in which we will be working on them.
The compiled list can be found HERE.

Since, the resources which talk about behave are scarce, my mentors linked me to some good snippets for beginners. They helped me crawl through the first bits of my code for gnome-weather ( HERE ). And so I decided to document a short summary of how to make our applications “behave”. (TLDR)

What is Behave?


Behave is Behavior Driven Development in Python (BDD). This testing methodology doesn’t use ‘tests’ in its usual sense, but replaces them with scenarios, which are basically an equivalent of ‘use cases’. These scenarios are organized in features, which are bdd’s equivalent of test suites. These features describe how some particular feature is supposed to work. Note, that features can use BBD’s equivalent of ‘setup’ – Background, which describes the state, in which the application under test should be put in order to execute the following scenarios.

 

BDD’s core concepts are Gherkin and DRY

Gherkin

Gherkin is Business Readable Domain Specific Language. This is a subset of plain text language, which is designed to be both readable by non-technical folks and be parsable by test automation tools. This language can also be used be designer to describe desired behavior before the implementation starts. Another feature is that engineer can use failing Gherkin scenario as ‘Steps to reproduce’ The core Gherkin concept is a step – an instruction, which is both human-readable and can be parsed by test automation tool. Step examples are:

   Given main Evolution window is displayed
    When I click ‘New Message’ button on the toolbar
    Then new composer window is opened
Note the gherkin keywords – Given, When, Then. These core keywords are used to highlight the purpose of the step.

Given is used to describe prerequesites to be executed to put the system into a known state.
When is generally used to execute actions
Then is used to verify result or check some important clause
Checkout a tutorial on basic concepts of Given When Then

Though Behave uses and implements all Gherkin features, I suggest to use a simplified version of this syntax with these simple rules:

Use asterisk (*) instead of When (and Given). ‘When’ is the most usable keyword in scenarios, so we should keep it short
‘Then’ keyword is used in the last step of the scenario only. The last step should verify the result of the whole scenario and should contain assertEquals (etc.)
Gherkin steps are organized into scenarios:

Scenario: Create a contact with categories
   * Evolution is opened
   * Create a new contact
   * Set “Full Name…” in contact editor to “Adam Jones”
   * Set “Categories…” in contact editor to “Business,Key Customer”
   * Save the contact
Asterisk here means “any keyword”, as step definitions don’t care which keyword is used.

DRY Don’t repeat yourself

DRY principle in Gherkin is important in automation using BDD, as in order to execute Gherkin scenarios, automation tool has to refer steps to parts of code, called step definitions. As Gherkin steps can be used in various scenarios and not organized in packages, class etc. (in other words, have no hierarchy), each step must match one and only one step definition. As a result, steps should be written in the way to be re-used in various Gherkin scenarios, as step definition code doesn’t know about previous nor following steps.

Using scenario from the previous section, we can create the following step definitions:

Toggle line numbers
   1 from behave import step
   2
   3 @step(u’Create a new contact’)
   4 def create_new_contact(context):
   5    ….
Note, that step definition is basically a Python function with @step decorator, which contain step matching string. Note context variable, which contains existing execution. Context should be used to store variables between steps, scenarios and features during test execution.

The step matching string can contain named properties, using parse format (think inverted format). This allows to pass parameters to steps:

Toggle line numbers
   1 from behave import step
   2
   3 @step(u’Set “{field}” in contact editor to “{value}”‘)
   4 def set_field_to_value(context, field, value):
   5    ….
Using previous scenario, set_field_to_value here will be called twice:

Toggle line numbers
   1  set_field_to_value(context, field=’Full Name…’, value=’Adam Jones’)
   2  set_field_to_value(context, field=’Categories…’, value=’Business,Key Customer’)
Behave also allows to call steps withing step definitions:

Toggle line numbers
   1 @step(u’Set contact categories to “{value}”‘)
   2 def set_contact_categories_to_value(context, value):
   3    context.execute_steps(u”””
   4      * Set “Categories…” in contact editor to “%s”
   5      * Another step if request
   6    “”” % (value))
   7    ….
These simple features of Behave allow us to write generic, easy-to-maintain code, which doesn’t depend on actual scenarios.

Note, that some code can be shared across projects, for instance, handling file open/save dialogs, start/stop test for applications etc.

Scenarios = Test case, Feature = Test suite

Gherkin scenarios are organized into features, which contain separate scenarios. If some actions should be performed before each scenario, use Background keyword:

Feature: Contact categories

Background:
  * Open Evolution via command and setup fake account
  * Open “Contacts” section
  * Select “Personal” addressbook

Scenario: Create a contact with categories
  …

Scenario: Set categories using Categories dialog
  …

Scenario: Contact is not listed in unchecked categories
  …
Note, that Behave doesn’t have ‘setup’ or ‘teardown’ concepts, but you still can control a group of scenarios/features using tags:

@needs_google_goa_account
Scenario: Setup Google account in evolution via GOA
  …
In step defintion code you can specify actions to be performed before and after execution of scenario/feature with a tag:

Toggle line numbers
   1 def before_tag(context, tag):
   2     if tag == ‘@needs_google_goa_account’:
   3         # setup google goa account here
   4
   5 def after_tag(context, tag):
   6     if tag == ‘@needs_google_goa_account’:
   7        # remove google goa account here
Scenarios and features have similar procedures: before_feature, before_scenario, after_scenario and after_feature

The above has helped me alot in developing a clarity about how to go about writing tests in behave. My code for gnome-weather is HERE . I will be updating this as and when I code new features.

Advertisements

2 thoughts on “Making applications “behave”

  1. Hi!

    Really interesting article and good point in use Behave for testing GNOME applications. In my current job Behave is one of the more important tool when you develop (as developers we are responsible to develop acceptance tests).

    A couple of months ago, I gave a talk about testing GNOME applications with Behave+Dogtail (for Spanish people), you can check the source here: https://github.com/gnomehispano/cribar

    I’m very interested in Q&A for the GNOME environment, How can I help? I don’t know the current status, the information here https://wiki.gnome.org/GnomeQA is quite old, and the mailing list haven’t recent activity…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s