DESKTOP TESTING .. JHBuild Updated Setup


The world of open-source is ever evolving.. My previous posts in the series of desktop testing were conversant about starting from scratch .. the scratch in GNOME tests just got a little changed. So this blog would address a couple of pointers which people intending to contribute in our QA team might find useful. (The post assumes the knowledge of building JHBuild from HERE) The first thing one would need to have in place is the gnome-desktop-testing-runner inside of JHBuild so as to run tests on the  GNOME components build via JHBuild. This post thus, aims at gathering some of the most useful resources for jhbuild in one place and also, providing some hands-on reference point to get started! Beginners might find it frustrating to look for modulesets which include this respective module. And so as a part of this post I would be sharing my ~/.config/jhbuildrc file which should make life a little bit simpler. After we have our jhbuild and its config file in place. All thats left to be done is fire up that bash and run: “jhbuild build gnome-desktop-testing” Few Troubleshooting pointers: The error below can be fixed by installing libatrr-devel externally.

checking attr/xattr.h presence... no
checking for attr/xattr.h... no
configure: error: You must have attr/xattr.h from libattr
*** Error during phase configure of libgsystem: ########## Error running ./ --prefix /home/shivani/jhbuild/install -enable-installed-tests *** [6/6]

Before proceeding to anything else, you might want to do a quick

jhbuild sanitycheck

to make sure jhbuild is good to go and

jhbuild sysdeps --install

so as to install the dependencies we might need in the future! A good place to cross check the dependency installation would be HERE. Also, a look at the JHBuild troubleshooting archive could be helpful in solving the most common issues. Finally, with a basic setup in place referring to THIS post might just be what you are looking for. More to come! :)

The changes you might want to do to the sample.jhbuildrc before transferring it to ~/.config/jhbuildrc can be traced as follows:

moduleset = ['gnome-apps-3.14','gnome-apps-3.12','gnome-world']
# A list of the modules to build.  Defaults to the GNOME core and tested apps.
modules = ['gnome-weather', 'meta-gnome-core', 'meta-gnome-apps-tested' ]
#..<some more stuff that went here>

Navigating the UI by ipython


However brilliant an API is, one never discovers its wonders without really learning how to use it. While some API’s are developer friendly and let their utility be discovered with ease, I found one which has been a rare used tool of awesome in the hiding.

Behave and Dogtail (in ipython majorly) in conjunct have little documentation to guide devs willing to write tests. I ventured into the same wilderness and came out wise enough to put together the following tutorial.

My sample application is Evince-3.8 , which is the latest of its versions available on Fedora 19 (Ancient, yet something I kind of am stuck with at the moment). There are some very good outcomes of my ancientness which will be apparent in the following post.

While there are many ways to navigate the UI of an application , a lot of them with fancy UIs , at-spi browsers like Sniff, Accerciser et al, however nothing beats the simplicity yet efficiency of a terminal. So, I would be covering ipython as a part of this post, which in my experience was the most efficient to figure the most complex/non-a11y friendly UIs.

The modules I found coming in handy are dogtail.root, dogtail.rawinput and dogtail.utils.

Once I had all the above installed and ready I fired the following commands, ready to navigate the evince UI.

$ ipython
$ from dogtail.tree import *
$ from dogtail.utils import *
$ from dogtail.rawinput import *

Once we have this initialized, we would need to “run” the application in order for it to be a part of those listed for browsing via ipython.

#Initialize the app here, using the "root" from dogtail.tree
$app = root.application('evince')

At this point, everything to know about the application is contained in “app” and the magic of autocomplete in ipython, “blink”, and “dump”.
Standard tutorials online (say the one HERE) right away start describing as to how to access various “widgets”, “children”, “text”, “frame” of the given application, however at this point I am not interested in how to access this Node of the “app” called “x” , rather I want to know WHAT is this node of the “app” called “x” in order to access it finally. This is where I’d need to execute the following:


which would give you an output on the following lines:

[application | evince]
 [frame | ]
  [filler | ]
    [tool bar | ]
    [panel | ]
     [filler | ]
      [push button | ]
       [action | click |  ]
      [push button | ]
        [action | click |  ]
    [panel | ]
     [filler | ]
      [text | page-label-entry]
       [action | activate |  ]
      [text | ]
       [action | activate |  ]
    [panel | ]
     [filler | ]
      [push button | ]
       [action | click |  ]
       [push button | ]
       [action | click |  ] ...

And it is this output which spawns multiple things hereon..

What I expected out of my magic “dump” was an illustrative verbose of terminal trace telling me all about “roleName”, “name” – much like they said it would be HERE. However, owing to some depredations in dogtail (i guess) I got something like .. well above.
It took me a while to make out that not all magic of dump is lost.

  • The dump trace encapsulates the hierarchy of the nodes of the UI of the application (here evince) , so, frame is the immediate “child” of the application, filler is the child of the frame and so on. which makes it easy to access (say) frame as – $frame=app.child(roleName=’frame’)
  • Which brings me to my second point, the roleName is the left side of the bracket, for instance in a line [x|y], x would be the roleName, while y should be the name/description of the node. If y is missing, we would need to file a Bug (like I did for 3.8 ;)) to request better accessibility for the application (which btw is also better for impaired people using Orca etc.); If however, y is well defined then you can keep accessing the objects like mentioned in the above point – $var=app.child(roleName=’x’,name=’y’)A point to keep in mind while filing a bug for the same is that name should be there only with relevant nodes, like the menu items, buttons et al, since it is super normal to have only “roleName” with fillers, panels. frames etc. – the ‘meta’ objects functioning only to structure the UI together.

What did I do to write some tests for Evince 3.8 without the y’s. My mentor, Vita helped me figure a way around, by accessing the “indices” instead of the non-available names for the respective nodes in question. In case of evince the buttons were ordered in the tree by their positions from left to right , so it was easier to navigate.

By default with dogtail, we can use the Node.children property, which simply returns ‘list’ all the immediate children of the node. So starting with the ‘toolbar’ object we can get to specific button (lets say the last one shown in the above output) :
button = toolbar.children[2].children[0].children[1]
which can be simplified to:
button = toolbar[2][0][1]

which goes as (indexed from 0 up): toolbar’s third child – panel > panel’s first child – filler > filler’s second child – the desired button.
Herein it is important to note that “child()” searches for the children recursively not just the immediate ones.

At this point when I am happy I know how to extract stuff, how do I confirm if what I have in my “button” object is right ?

$ #Clicks the respective button
$button.blink() #Highlights in Red the button in the evince UI

Essentially, when using the ipython to investigate, child is the most common way of exploration, followed with obtaining the Node we want to – and once we have the required node we can do all sorts of stuff to do.Essentially executing actions  (from Node.actions) . For instance – the ones mentioned above like bink, click, typing into it (if its i.e. text field) et al. Its only when one can not child() ourselves onto a Node that the indexing be used as a backup.




GUADEC’14, Strasbourg



GUADEC’14, Strasbourg marked many things for me. My second GUADEC – check .
A QA Team kickstarted – check. Discussions with release team – check. Sightseeing and dining out with the most diverse geeks – check. Celebrating a grand birthday – check.

While the above is a brief summary of how the last 10 days were. There were some major things which stood out.

Work Stuff
We finally formalized the QA Team. Yes, GNOME components shall have tests now. We also had the “Screenshots automation BoF” which made me realize the pain documentation team goes through, clicking all those screenshots, so we finalized the following workflow, wherein QA team can help the documentation folks ease the same.

QA team writes Tests for component X → QA Team updates the status of tests for X HERE → QA team drops a ping in IRC #doc → Doc team files a bug on the component Bugzilla stating the required screenshots → QA Team updates the tests.

We are yet to try this proposed scheme but I am positive that it should take a lot of load off the designers.

I also got a chance to interact with the maintainers/developers of the various components I am planning to write tests for. Especially Evince and Weather. All of which are going to see some major UI changes in the near releases. Which is why discussing about upcoming changes played a good role in shapping my future test coverage.

Finally, I got to meet my mentors. We got some time to socialize over dinner and drinks , and talk about how to write tests effectively..

Non-Workish details
Right from Eiffel Tower and Louvre in Paris to the amazing sighseeing in Strasbourg we had an amazing time. A special thanks to these people who made dinner/drinks hunting, to Le Petite France trips so memorable.

Also, this time around we had one of the best women’s dinners I have been to. The GNOME girls know how to make a birthday special ^_^ (and nearly made up for the people I missed on the day back home), we sung in soo many languages, had great food over some heart warming conversation.
A big hug to Karen and Marina for buying me my birthday meal :D


And a hug to all the ladies for being there and making the day so special.
A snap of a special gift I got from the younger one of our guests at women’s dinner..


Signing off .. Thankyou to the organizers for doing a great job and to GNOME Foundation for having me there :)

sponsored-badge-simple Continue reading

Making Evince “Behave” & GUADEC!


Evince_logo_newThis is probably going to be the shortest post I will be writing, but then it’s GUADEC time!

My latest component is Evince, the document viewer. I am building up on the same infrastructre as we did for Weather testing, and have put together some basic tests for the application. These can be found HERE

On the lighter side of things, I could do with a baggage zipper (much like a file zipper) at a click to take care of my packing, and mail me and my zipped luggage to save us from the gruesome travel :P

Apart from these, I am helping the team organize Volunteers, and I realize I love organizational tasks for GUADEC. (Should take up more of these in the future too :D)

Anyways, returning to the old way of packing now. (I promise the next blog will be better!)

Making applications “behave”


After the previous post wherein we setup gnome-continuous, the next phase of my project involves writing tests for applications installed in it. I started with gnome-weather.
To start with one can upgrade the previously installed image to latest version from with “ostree admin upgrade” (restart required).
There are majorly 2 ways to approach writing tests.

  • By playing around directly with the installed tests located at “/usr/libexec/installed-tests/<application-name>
  • By writing “features” and “steps” for testing the application via “behave

In my case the former option doesn’t quite work as much as the later. A major could be that I find it faster to work on my local machine and then test the tests on the gnome-continuous platform. And also because the VM tends to undergo major connectivity issues every now and then. So, this post will majorly talk about “behave”.

We segregated a list of components which need work, along with the priority in which we will be working on them.
The compiled list can be found HERE.

Since, the resources which talk about behave are scarce, my mentors linked me to some good snippets for beginners. They helped me crawl through the first bits of my code for gnome-weather ( HERE ). And so I decided to document a short summary of how to make our applications “behave”. (TLDR)

What is Behave?

Behave is Behavior Driven Development in Python (BDD). This testing methodology doesn’t use ‘tests’ in its usual sense, but replaces them with scenarios, which are basically an equivalent of ‘use cases’. These scenarios are organized in features, which are bdd’s equivalent of test suites. These features describe how some particular feature is supposed to work. Note, that features can use BBD’s equivalent of ‘setup’ – Background, which describes the state, in which the application under test should be put in order to execute the following scenarios.


BDD’s core concepts are Gherkin and DRY


Gherkin is Business Readable Domain Specific Language. This is a subset of plain text language, which is designed to be both readable by non-technical folks and be parsable by test automation tools. This language can also be used be designer to describe desired behavior before the implementation starts. Another feature is that engineer can use failing Gherkin scenario as ‘Steps to reproduce’ The core Gherkin concept is a step – an instruction, which is both human-readable and can be parsed by test automation tool. Step examples are:

   Given main Evolution window is displayed
    When I click ‘New Message’ button on the toolbar
    Then new composer window is opened
Note the gherkin keywords – Given, When, Then. These core keywords are used to highlight the purpose of the step.

Given is used to describe prerequesites to be executed to put the system into a known state.
When is generally used to execute actions
Then is used to verify result or check some important clause
Checkout a tutorial on basic concepts of Given When Then

Though Behave uses and implements all Gherkin features, I suggest to use a simplified version of this syntax with these simple rules:

Use asterisk (*) instead of When (and Given). ‘When’ is the most usable keyword in scenarios, so we should keep it short
‘Then’ keyword is used in the last step of the scenario only. The last step should verify the result of the whole scenario and should contain assertEquals (etc.)
Gherkin steps are organized into scenarios:

Scenario: Create a contact with categories
   * Evolution is opened
   * Create a new contact
   * Set “Full Name…” in contact editor to “Adam Jones”
   * Set “Categories…” in contact editor to “Business,Key Customer”
   * Save the contact
Asterisk here means “any keyword”, as step definitions don’t care which keyword is used.

DRY Don’t repeat yourself

DRY principle in Gherkin is important in automation using BDD, as in order to execute Gherkin scenarios, automation tool has to refer steps to parts of code, called step definitions. As Gherkin steps can be used in various scenarios and not organized in packages, class etc. (in other words, have no hierarchy), each step must match one and only one step definition. As a result, steps should be written in the way to be re-used in various Gherkin scenarios, as step definition code doesn’t know about previous nor following steps.

Using scenario from the previous section, we can create the following step definitions:

Toggle line numbers
   1 from behave import step
   3 @step(u’Create a new contact’)
   4 def create_new_contact(context):
   5    ….
Note, that step definition is basically a Python function with @step decorator, which contain step matching string. Note context variable, which contains existing execution. Context should be used to store variables between steps, scenarios and features during test execution.

The step matching string can contain named properties, using parse format (think inverted format). This allows to pass parameters to steps:

Toggle line numbers
   1 from behave import step
   3 @step(u’Set “{field}” in contact editor to “{value}”‘)
   4 def set_field_to_value(context, field, value):
   5    ….
Using previous scenario, set_field_to_value here will be called twice:

Toggle line numbers
   1  set_field_to_value(context, field=’Full Name…’, value=’Adam Jones’)
   2  set_field_to_value(context, field=’Categories…’, value=’Business,Key Customer’)
Behave also allows to call steps withing step definitions:

Toggle line numbers
   1 @step(u’Set contact categories to “{value}”‘)
   2 def set_contact_categories_to_value(context, value):
   3    context.execute_steps(u”””
   4      * Set “Categories…” in contact editor to “%s”
   5      * Another step if request
   6    “”” % (value))
   7    ….
These simple features of Behave allow us to write generic, easy-to-maintain code, which doesn’t depend on actual scenarios.

Note, that some code can be shared across projects, for instance, handling file open/save dialogs, start/stop test for applications etc.

Scenarios = Test case, Feature = Test suite

Gherkin scenarios are organized into features, which contain separate scenarios. If some actions should be performed before each scenario, use Background keyword:

Feature: Contact categories

  * Open Evolution via command and setup fake account
  * Open “Contacts” section
  * Select “Personal” addressbook

Scenario: Create a contact with categories

Scenario: Set categories using Categories dialog

Scenario: Contact is not listed in unchecked categories
Note, that Behave doesn’t have ‘setup’ or ‘teardown’ concepts, but you still can control a group of scenarios/features using tags:

Scenario: Setup Google account in evolution via GOA
In step defintion code you can specify actions to be performed before and after execution of scenario/feature with a tag:

Toggle line numbers
   1 def before_tag(context, tag):
   2     if tag == ‘@needs_google_goa_account':
   3         # setup google goa account here
   5 def after_tag(context, tag):
   6     if tag == ‘@needs_google_goa_account':
   7        # remove google goa account here
Scenarios and features have similar procedures: before_feature, before_scenario, after_scenario and after_feature

The above has helped me alot in developing a clarity about how to go about writing tests in behave. My code for gnome-weather is HERE . I will be updating this as and when I code new features.

Desktop Testing, GNOME Continuous and VMs..


Before writing tests, one needs to figure whats wrong with the existing ones and fix them. Or better more get them to run first.
This is where I started testing those we already have gnome-weather. (Insert – This is where I realized my system needed a lot more packages, thrashing and fixing to run anything close to tests) .

Usually, by my experience prior to this project, I would simply run the python tests (via “make check”). However, as gnome-weather has quite a complicated way to get started, one needs gnome-desktop-testing runner installed via jhbuild.

[sourcecode language="bash"]
jhbuild build gnome-desktop-testing-runner
jhbuild run gnome-desktop-testing-runner org.gnome.Weather

Following this, to be able to run the above tests the app needs to be built with  –enable-installed-tests -> this to be added in autogenargs in the file “~/.config/jhbuildrc”.

I faced some at-spi issues, which I resolved by building  at-spi2-atk, at-spi2-core and pyatspi2-python2 with jhbuild.

In my case however, some packages clashed because of which depreciation errors kept intruding. To overcome this, I moved to a much saner approach. Install a pristine environment for the tests. (Read – create a gnome-continuous VM and work on installed tests).

The following post will talk all about setting up a VM for gnome-continuous. The image we will need is Here.
There are various options of VM Managers available , the easiest to use is Virt-Manager, the one I used.

The foremost thing we will do is, setting up a Bridged Network which the VM can use to ssh into.

  1. Go to “Edit – Connection Detals”
  2. Navigate to “Virtual Networks”
  3. Click “+” < Input any name for the Network < Then click Forward < Forward < Forward < Check “Forwarding to physical network”
  4. Choose your connection (em1 usually, if wired) and finally, method = NAT
  5. Click Finish.

Your final output should be something like the below:


After the network is setup, we will create the VM which will be using the above network to connect to the internet and for ssh connections. The steps to install the gnome-continuous image can be seen in the following screens.

ImageImageImageClick Forward < Finish.

Make sure the VM boots. A few troubleshooting tips:

  • Make sure the storage format is selected correctly, the default format is “raw”, while we are using “qcow2″ , which might cause problems in booting the VM. You can correct the same by changing the it from the IDE Disk settings tab as follows: Image
  • Another issue can be the boot loader settings. Make sure the Boot source is set to be from the Hard Disk. Image

After this hopefully, your VM should be up and running!
If not. you arent alone ! :P Although the VM should ideally boot up by now, there is a slight bug because of which you might be prompted to login. I was subjected to a blank gnome screen and was stuck after this. The workaround for the following awaits you in the escape terminal. The steps you need are :

  1. Start the vt2 terminal by sending in the combination Ctrl+Atl+F1/F2
  2. Login as root
  3. Use “useradd test”
  4. and finally ‘passwd test”

This should get the VM up and running with a VM :D


More on testing to follow :)



SoC 2014 , Automated Tests for GNOME Continuous.


The past couple of weeks have been a cocktail of emotions, from being thrilled (with SoC results), to amazed by all that I am going to be learning and have learned so far.. I am now on the phew lots to learn and implement flavor of my summer of code journey.
Being my first blog about my project I’d like to write about the utilities of testing and why this project. The next blogs which shall soon follow are about:

  • Setting up gnome-continuous with a VM
  • Dogtail, Sniff (and the issues I faced with them so far)
  • Introducing Behave


What is the project about? 

The ultimate Aim of the project is to improve test coverage for GNOME components. Me along with Martin Simon , under the mentorship of Vadim Rutkovsky and Vitezslav Humpa will be working on important components being built on gnome-continuous namely:  gedit, geary, gnome-documents, gnome-logs, evince, seahorse, gnome-maps, gnome-calculator, gnome-photos, gnome-software, evolution, gnome-control-center, gnome-weather, nautilus, gnome-terminal, gnome-clocks and totem.     The tests we envision to incorporate with various GNOME component would be majorly integration tests, which will thus witness the full application running in a pristine but complete gnome environment with most dependencies, and without mocking services or back-ends.


What did I learn so far? 

Although I had a brief exposure to writing code to test applications, however, the sanctity of writing a robust test, is something that I have learnt now.  I have realized that it probably might take you more time to setup things, learn to move on from systems you have tried to setup devoting time than the real coding part of the application.  (More about this in the following blogs!)


Signing off with this funny quote about TDD

Always code as if the person who writes tests for your program is a violent psychopath and knows where you live!