Navigating the UI by ipython

However brilliant an API is, one never discovers its wonders without really learning how to use it. While some API’s are developer friendly and let their utility be discovered with ease, I found one which has been a rare used tool of awesome in the hiding.

Behave and Dogtail (in ipython majorly) in conjunct have little documentation to guide devs willing to write tests. I ventured into the same wilderness and came out wise enough to put together the following tutorial.

My sample application is Evince-3.8 , which is the latest of its versions available on Fedora 19 (Ancient, yet something I kind of am stuck with at the moment). There are some very good outcomes of my ancientness which will be apparent in the following post.

While there are many ways to navigate the UI of an application , a lot of them with fancy UIs , at-spi browsers like Sniff, Accerciser et al, however nothing beats the simplicity yet efficiency of a terminal. So, I would be covering ipython as a part of this post, which in my experience was the most efficient to figure the most complex/non-a11y friendly UIs.

The modules I found coming in handy are dogtail.root, dogtail.rawinput and dogtail.utils.

Once I had all the above installed and ready I fired the following commands, ready to navigate the evince UI.

$ ipython
$ from dogtail.tree import *
$ from dogtail.utils import *
$ from dogtail.rawinput import *

Once we have this initialized, we would need to “run” the application in order for it to be a part of those listed for browsing via ipython.

#Initialize the app here, using the "root" from dogtail.tree
$app = root.application('evince')

At this point, everything to know about the application is contained in “app” and the magic of autocomplete in ipython, “blink”, and “dump”.
Standard tutorials online (say the one HERE) right away start describing as to how to access various “widgets”, “children”, “text”, “frame” of the given application, however at this point I am not interested in how to access this Node of the “app” called “x” , rather I want to know WHAT is this node of the “app” called “x” in order to access it finally. This is where I’d need to execute the following:


which would give you an output on the following lines:

[application | evince]
 [frame | ]
  [filler | ]
    [tool bar | ]
    [panel | ]
     [filler | ]
      [push button | ]
       [action | click |  ]
      [push button | ]
        [action | click |  ]
    [panel | ]
     [filler | ]
      [text | page-label-entry]
       [action | activate |  ]
      [text | ]
       [action | activate |  ]
    [panel | ]
     [filler | ]
      [push button | ]
       [action | click |  ]
       [push button | ]
       [action | click |  ] ...

And it is this output which spawns multiple things hereon..

What I expected out of my magic “dump” was an illustrative verbose of terminal trace telling me all about “roleName”, “name” – much like they said it would be HERE. However, owing to some depredations in dogtail (i guess) I got something like .. well above.
It took me a while to make out that not all magic of dump is lost.

  • The dump trace encapsulates the hierarchy of the nodes of the UI of the application (here evince) , so, frame is the immediate “child” of the application, filler is the child of the frame and so on. which makes it easy to access (say) frame as – $frame=app.child(roleName=’frame’)
  • Which brings me to my second point, the roleName is the left side of the bracket, for instance in a line [x|y], x would be the roleName, while y should be the name/description of the node. If y is missing, we would need to file a Bug (like I did for 3.8 ;)) to request better accessibility for the application (which btw is also better for impaired people using Orca etc.); If however, y is well defined then you can keep accessing the objects like mentioned in the above point – $var=app.child(roleName=’x’,name=’y’)A point to keep in mind while filing a bug for the same is that name should be there only with relevant nodes, like the menu items, buttons et al, since it is super normal to have only “roleName” with fillers, panels. frames etc. – the ‘meta’ objects functioning only to structure the UI together.

What did I do to write some tests for Evince 3.8 without the y’s. My mentor, Vita helped me figure a way around, by accessing the “indices” instead of the non-available names for the respective nodes in question. In case of evince the buttons were ordered in the tree by their positions from left to right , so it was easier to navigate.

By default with dogtail, we can use the Node.children property, which simply returns ‘list’ all the immediate children of the node. So starting with the ‘toolbar’ object we can get to specific button (lets say the last one shown in the above output) :
button = toolbar.children[2].children[0].children[1]
which can be simplified to:
button = toolbar[2][0][1]

which goes as (indexed from 0 up): toolbar’s third child – panel > panel’s first child – filler > filler’s second child – the desired button.
Herein it is important to note that “child()” searches for the children recursively not just the immediate ones.

At this point when I am happy I know how to extract stuff, how do I confirm if what I have in my “button” object is right ?

$ #Clicks the respective button
$button.blink() #Highlights in Red the button in the evince UI

Essentially, when using the ipython to investigate, child is the most common way of exploration, followed with obtaining the Node we want to – and once we have the required node we can do all sorts of stuff to do.Essentially executing actions  (from Node.actions) . For instance – the ones mentioned above like bink, click, typing into it (if its i.e. text field) et al. Its only when one can not child() ourselves onto a Node that the indexing be used as a backup.





One thought on “Navigating the UI by ipython

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s