simon meets AT-SPI-2

Over the last couple of days I have again been working on what I started during this years Desktop Summit: simons AT-SPI 2 integration.
What started as a GSoC project idea back in April is now beginning to take shape.

The basic idea is still the same: First, integrate sequitur in simon to be able to transcribe arbitrary words automatically. To facilitate this, sequitur first needs to learn the transcription rules from a large dictionary. So I integrated a feature that let's user turn their shadow dictionary (which already supports many different formats) into a regular sequitur model.

After this sequitur model generation process, the system is used to transcribe words for the ATSPI plugin but also for adding new words manually.

Thanks to sequitur, simon can now transcribe words automatically that are definitely not in the shadow dictionary:

With this as the basic foundation and some help from Frederik and Joanie I created a plugin that would analyze the UI of currently active window, create vocabulary and grammar for it and associate commands with the user interface elements.

It's still in an early development stage (as is the support for ATSPI-2 of GTK and Qt) but the basic stuff already works. To check it out, either build and install the current development version of simon from Git (atspi branch) or have a look at the demonstration video below.

For RSS readers: ATSPI demonstration on Youtube