simon: Open-Source Speech Recognition

Knock, Knock, KDE

2012-01-21T04:19:00.000-08:00

After using Sourceforge for the last couple of years, simon finally joined the kool kids on the KDE infrastructure!

As part of the move, we also united the Sourceforge- and github repositories - they were only separate for organizational reasons.
We then re-organized the codebase into two projects:
simon (containing the simon application suite) and simon-tools (consisting of smaller tools that we created for various projects like a small command line utility to control Skype, a tiny calendar, and even a touch-friendly media center).

So far (about one week in) I can only say that I'm already thouroughly amazed about how incredibly active and helpful the KDE community really is - but more about that in a later post!

Looking for a KDE related job? We are hiring!

2011-11-04T09:55:00.000-07:00

We, the non profit research organization simon listens e.V. are looking for qualified C++ / Qt / KDE hackers to join our team!

Initially, we would be looking to fill part time positions but they can be extended to full time afterwards.

While our projects mostly focus on speech recognition using our own, KDE based solution called simon, you do not need to know anything about speech recognition to join!

Interested? Contact me for more information or send me your resume right away: grasch at simon-listens dot org

simon meets MeeGo

2011-10-14T02:05:00.000-07:00

I'm happy to report that since August, I can now officially call myself a Qt Ambassador!

As an Ambassador, I had the opportunity to apply for a loaned Nokia N950 to develop / port applications to MeeGo/Harmattan. I took Nokia up on their offer and the result is simone - a trimmed down, mobile version of simon. In other words: "simon embedded" or "simone".

The client features push to talk or automatic voice activity detection (configurable) and because of simons client / server architecture uses little power on the device itself. Even with voice activity detection running you should get many hours of continuous speech recognition out of a single charge.

On the one hand, simone can be used to replace the headset of a "full" simon installation but also includes a couple of default actions on the device. For example, you can use a voice controlled quick dial feature or start / stop a turn-by-turn navigation.

For more information and a live demo, have a look at the youtube demonstration:

If you can't see the embedded video, try this direct link.

simon meets AT-SPI-2

2011-09-06T09:16:00.000-07:00

Over the last couple of days I have again been working on what I started during this years Desktop Summit: simons AT-SPI 2 integration.
What started as a GSoC project idea back in April is now beginning to take shape.

The basic idea is still the same: First, integrate sequitur in simon to be able to transcribe arbitrary words automatically. To facilitate this, sequitur first needs to learn the transcription rules from a large dictionary. So I integrated a feature that let's user turn their shadow dictionary (which already supports many different formats) into a regular sequitur model.

After this sequitur model generation process, the system is used to transcribe words for the ATSPI plugin but also for adding new words manually.

Thanks to sequitur, simon can now transcribe words automatically that are definitely not in the shadow dictionary:

With this as the basic foundation and some help from Frederik and Joanie I created a plugin that would analyze the UI of currently active window, create vocabulary and grammar for it and associate commands with the user interface elements.

It's still in an early development stage (as is the support for ATSPI-2 of GTK and Qt) but the basic stuff already works. To check it out, either build and install the current development version of simon from Git (atspi branch) or have a look at the demonstration video below.

For RSS readers: ATSPI demonstration on Youtube

Desktop Summit 2011

2011-08-09T09:39:00.000-07:00

I just arrived back home (I flew home after the talks) after this years Desktop Summit and it was awesome! In retrospect I kinda regret not staying the whole week... Next year... :)

Anyways, I met tons of interesting people and had a lot of productive meetings and discussions. It's amazing what can get done in just a few minutes if the right people are sitting together.

If we (the KDE accessibility team) can implement even half of what was discussed in the last couple of days, I'm sure we're looking at a big step towards a truly accessible free desktop.

Oh and Martin: I'm looking forward to all those KWin effects for simon :P

Benefit Project Completed

2011-08-09T06:56:00.000-07:00

After more than a year of hard work we - the simon listens Team - are proud to announce that the Benefit project to use simon among other open source technologies (XBMC, Ubuntu,...) to create an affordable, self contained, voice controlled multimedia solution especially suited for elderly people has been completed.
The created solution - including the speech model and scenarios - will be released under a free license very soon.

But in the meantime, you can already have a look at a short demo video on youtube:

(Planet readers, click here)

GSoC Guest Post: Context Detection

2011-06-08T00:59:00.000-07:00

This year, we have been given the opportunity to work with two students as part of Googles annual Summer of Code. Adam is working on context dependent speech recognition (see below) and Alessandro is working on the Voxforge integration. Moreover, another student, Saurabh, is working on the Workspace integration as part of the Season of KDE.

So as kind of a start to hopefully a series of blog posts of our new contributers, I asked Adam to talk a bit about his progress and future plans about the context dependent speech recognition. This is what he wrote:

As part of the Google Summer of Code, I have been working to add
context-based activation and deactivation of scenarios in the KDE speech
recognition program simon. The simon program allows users to create or
download scenarios which, when activated, allow them to control other
programs such as web browsers, text editors, and games with speech
commands.

When the number of commands that must be considered for speech
recognition in simon becomes too large (for example, if the scenarios
that are active have a large number of possible commands), the speed and
accuracy of the speech recognition can suffer to the point of
unusability. Context-based activation and deactivation of scenarios will
allow scenarios to be deactivated when they are not needed (for example,
when the program that they control is not opened, or when the program is
not the active window) so that the number of commands being considered
by speech recognition will be kept low enough to ensure accuracy and
performance.

The context gathering system has been developed so that scenarios have a
"compound condition" which is a group of conditions under which the
scenario should activate. The compound condition becomes satisfied when
all of its conditions (which gather contexts) are satisfied. When the
compound condition becomes satisfied or unsatisfied, it communicates
this to its scenario, which then indicates to the scenario manager
whether or not it should be activated.

Compound conditions will be created with a user interface similar to
simon's command adding and editing interface. A scenario with no
conditions in its compound condition will always be active. This means
that any scenario made before this feature was added will maintain its
former functionality, but can be easily changed to (de)activate under
certain conditions.

The conditions of which the compound condition is composed are developed
as plugins (similarly to the command managers in simon), so it will be
easy to add new types of conditions. For example, one of the currently
developed plugins gathers information about running processes, so a
scenario can be activated under the condition that some process is
running or not running (for example a Rekonq scenario could have the
condition "'rekonq' is running"). The extensibility allowed by this
plugin system means that conditions such as "'Firefox' is the active
window" or "The user is connected to the internet" or "Fewer than 3
scenarios are currently active in simon" or any other type of condition
that could be determined by simon can be easily developed and used to
guide scenario activation and deactivation.

The next steps of my project include making the scenarios actually
activate and deactivate in response to conditions, making a parent/child
scenario relationship so that a single scenario can have child scenarios
with independent grammars and conditions (so that parts of the scenario
can be activated and deactivated independently), making more condition
plugins, and exploring the possibilities of what else simon would be
able to do with the contexts that it will be able to gather (for example
switching speech models based on the microphone that is being used).

GSoC idea: Ubiquitous Speech Recognition

2011-04-05T10:44:00.000-07:00

The Google Summer of Code application period for students closes in a couple of days and I still have one last idea for simon for any student still looking for a project: Ubiquitous Speech Recognition.

Some of you might already know that simon already supports recording (and recognizing) from multiple microphones simultaneously. Sound cards and microphones are comparatively cheap and the server / client architecture of simon would even allow for input from mobile phones, other PCs, etc.

We also have gadgets and home appliances getting smarter and smarter every year. KNX is getting increasingly popular, is already included in many new electrical installations and allows home automation for a very fair price.

Voice control is an intuitive way to interact with all kinds of devices and - compared to alternatives like touch screens and the like - also quite cheap. simon already has more than enough interfaces to connect up your favorite home automation controllers / hardware interfaces. Something that people are already doing.

However, speech recognition has traditionally relied on controlled environments. False-positives are still a major issue and recognition accuracy depends on being optimized for a certain situation.

Still: Adapting the recognition to certain situations is already part of another GSoC idea (that fortunately already has a very promising student attached to it) so that leaves the voice activity detection part as the remaining hassle.

The voice activity detection (in short: VAD) tells the system when to listen to the user and tries to distinguish between background noise and user input. Normally this is just one comparatively minor part in a speech recognition system but when your whole apartment (or at least parts of it) are listening for voice input this becomes kind of important :).

The current system in simon just compares the current "loudness" to a configurable threshold. This is fine for headset users but almost useless in the above scenario.

And here is where it's your turn to get creative: Try to find a novel approach to separate voice commands from background noise.

For example: Use webcams and computer vision algorithms to determine if the user is even near a microphone at the time of the heard "command".

You could also define "eye contact" with a camera as the signal to activate the recognition. Or maybe you could deactivate the system unless the user raises his hand before he speaks?

Another idea would be to let different microphones work together and subtract the similarities (to filter out global noise).

You can also use noise conditioning to remove the music playing over the PC speakers automatically from the input signal.

Or why not use the reception strength of the users bluetooth phone to determine in which room he currently is?

Bonus points for coming up with other ideas in the comment section!

GSoC idea: Voice Control for the Linux Desktop

2011-04-04T05:55:00.000-07:00

As this has worked so perfectly the last time, I want to use this blog post to present another idea for the Google Summer of Code 2011 that has not yet found an interested student.

Motivation
The simon system currently has plugins to trigger shortcuts, simulate clicks and interact directly with applications through IPC technology like DBus and JSON. This makes simon perfect for interacting with a vast variety of applications as long as it is configured for each application beforehand.

To counteract this, we have the scenario system that allows users to exchange such configurations online. This repository already covers many of the "standard" applications.
Still: The user has to actively pick which applications to control. If there is no scenario available for an application, things get a bit more complicated.

So how could we create dynamic scenarios that allow the user to control new applications without configuring anything?

Well let's look at what's needed to voice control an application.

Commands
First of all, we need to know what options are currently available.

Let's look at KWrite as an example application:

Just looking at the screenshot a human can quickly tell that there are at least the following commands: "New", "Open", "Save", "Save As", "File", "Edit", etc.

Well if screenreaders can read those options to the user, why shouldn't simon parse them automatically as well?

With the upcoming AT-SPI-2 and the Qt accessibility bridge, the user interface (including buttons, menu items, etc.) are all exported over DBus.

As elements can also be triggered (clicked / selected) over this interface, simon can easily "read" running applications and create appropriate commands.

Best of all: Because screenreaders are well established, many applications already make sure that this will work properly.

Vocabulary and Grammar
Now that we have our commands in place simon still needs to recognize all those words ("New", "Save", etc.) that are probably not in the users active vocabulary.

As speech recognition systems need a phonetic description of each word that is not trivial.

...if it weren't for Sequitur. Sequitur is a grapheme to phoneme converter that translates any given text to a phonetical description.

The system can be compared to a native speaker: Even if you have never heard a word spoken out loud you still have at least a rough idea about how to pronounce it. That's because there are certain rules in any language that you know even if you aren't aware of them.
Sequitur works in much the same way that it learns those rules by reading large dictionaries. With the generated model it can transcribe even words that were not in the input dictionary.

In our tests, sequitur prooved to be very reliable, accurate and quite fast.

simon already allows the user to specify a dictionary large enough to act as the information source for sequitur: The shadow dictionary. Because there are already import mechanisms for most major pronunciation dictionary formats, there is more than enough raw material to "feed" to sequitur already available.

Now that we have the vocabulary, setting up an appropriate grammar is very easy. Just make sure that all the sentences of the created commands are allowed.

For static models no training data is required so that's all that'd be needed.

Summary
With a combination of AT-SPI-2 and Sequitur one could quite easily extend the current simon version to automatically create working voice commands for all standard widgets of running applications.

This allows the user of a static model to comfortably use any application-specific configuration at all.

Because AT-SPI-2 is a freedesktop.org standard, the resulting system would automatically work with both Qt and KDE applications as well as Gnome applications.

If you are interested in working on this idea, please send me an email.

GSoC idea: Crowdsourcing Speech Model Training

2011-04-02T02:20:00.000-07:00

There still is a week left for students to apply for Googles annual Summer of Code.

Following Lydias recommendation on the mailing list, I've decided to showcase some ideas for simon that are not yet taken by any student on this blog for the remainder of the application period: If you'd like to implement one of those ideas, please feel free to send me a mail at grasch ate simon-listens ° org.

The first idea that is still up for grabs is simons voxforge integration. Voxforge is an ambitious project to create free (GPL) speech models for everyone. With the current Voxforge models, simon can already be used without any training at all. Just download simon and the appropriate model from the Voxforge website for your language and start talking to your computer.

This works because the Voxforge models have been trained with lots and lots of voice recordings from people around the world. The resulting model is speaker-independent and works quite well for most people. If you need even more accuracy, just adapt the general model to your voice with a couple of training session and you are ready to go.

The current Voxforge model for English is quite good for command and control but nowhere near powerful enough for dictation. The models for other languages consist of even fewer samples. In the last five years, 624 identified users submitted voice recordings for the English model. Only 50 identified people submitted recordings for the German Voxforge model.

I think this is primarily because donating voice (through the Java applet on the Voxforge homepage) is only done by those who are actively searching for ways to improve open source speech recognition. There is also no immediate pay off for the donators.

simon on the other hand reaches a wide array of people interested in open source speech recognition: More than 24.000 in the past 12 months.

Many of those users train simon to get the most out of their system. But those trainings samples never get submitted to Voxforge to improve the general model because there is no easy way to do that.

I propose to implement an easy to use uploading system that allows the user to submit his training samples directly to the voxforge corpus with the press of a button.

Together with an automatic download of the voxforge model for a selected language when simon is launched for the first time this means that simon users can:
1. Get started with the general model even easier because they don't have to download it manually
2. If the recognition rate is too low, they can (and in our experience often will) train their model locally.
By submitting the recorded samples for the local training back to Voxforge, they not only submit valuable recordings - more often than not they would even submit exactly those recordings that train words that couldn't be recognized with the previous Voxforge model.

And because users can immediately see if their samples are helping or hurting (by checking if the recognition rate improves locally), the generated submissions should be fairly high quality. There is even an immediate advantage for the end-user (their recognition rate improves).

If you are interested on working on this proposal please contact me at grasch ate simon-listens ° org.

simon at GSoC 2011

2011-03-19T09:28:00.000-07:00

I've just been officially approved as a mentor for KDE. There are three ideas already for simon on the ideas page.

It's the first time I'll (hopefully) be participating in GSoC (as a mentor) but I am very much looking forward to it.

I hope to find a few students interested in simon so if you want to do anything that has to do with speech recognition at all (even if it's not mentioned in the ideas page), just contact me.

Want to voice-control your lawn mower? Talk to your Roomba? We can do that :)

Let your ideas run wild!

CeBIT 2011

2011-03-19T08:59:00.000-07:00

I've been to quite a few conferences in the past year including cooperate stuff like the AAL Forum in Denmark but also this years Akademy, the OpenSUSE conference and the LinuxTage in Graz (where I'll be again this year, btw.). However, I've never been to anything like the CeBIT.

We arrived a day early to inspect our booth space and to set up our equipment. I then had a quick stroll through the rest of our hall. That "quick stroll" took about an hour. And that was only "our" hall (hall 2) which isn't even the biggest one of the 17 - again: seventeen - halls in use.

But you all probably knew that the CeBIT, the biggest IT fair in the world, was quite big :). Still, it's something entirely different to walk through the halls yourself. Not that we had a lot of time to explore the exhibition - no, we were plenty busy :)

The simon booth had an ideal position in one of the most frequented halls right in the middle of the open source area. That meant we got a lot of foot traffic and had lots of interesting conversations. We also had a lot of people coming up to us telling us that they already use simon. This included stuff like home automation - a first for simon AFAIK.

We also met quite a few people how work in nursing homes that were quite impressed with simon and exchanged contact information with us. We got amazing feedback and of course a lot of feature requests so we won't be running out of ideas any time soon :)

As a little thank you to all the people behind simon we sent out invitations to simons many testers, contributers and translators including free tickets for the event. Many had to decline because they live too far away but some could make it and met us in Germany. Among them was John Ambeliotis, author of jaNET a personal assistant powered by simon.


John and me next to the simon listens booth

I think it's great to meet contributers up close instead of just communicating per e-Mail and want to thank everybody again who came to visit us and expressed interest in simon!

Waking from Hibernation

2011-02-05T07:13:00.000-08:00

After a very busy Jannuary I finally have some more time to work on simon again. Do expect more updates than usual from me this month :)

The first week(s) I will be working on getting the Akonadi integration in simon up and running. Using this new plugin you will be able to schedule simon commands on specific dates / times using your conventional groupware infrastructure.
For example, you can use simons dialog system to wake you up in the morning by scheduling an event in KOrganizer with a special (but configurable) prefix in the events summary: By default "[simon-command] Dialog/Welcome" would execute the welcome dialog at the start time of the event in your calendar.

The Akonadi plugin also provides the option to react on other, "normal" events by displaying a reminder about them - again by using the dialog system. This is meant as a replacement of the korganizer reminder system for voice controlled systems.

You can even use the Akonadi plugin to schedule command executions from within simon.

The dialog system has been spruced up and now sports support for user avatars and multiple texts per state (one will be randomly selected during execution to make the system feel a bit more natural).

Other than that I will also be working - again - on improving our speech model creation technique and have also been assigned to have a look at some problems in ssc / sscd again that seem to creep up in the windows version.

The other parts of our Benefit project are also progressing well and we will be starting the first prototyping tests within the next couple of weeks. I hope I can get some video demonstrations done before the end of the month.

Of course, next to all that serious business I couldn't resist to hack around a bit and developed a tiny proof of concept that demonstrates a novel approach on how to deal with false positives in a speech recognition system that will basically running 24 / 7.

The prototype uses opencv and the webcam of my notebook and runs a simple face detection on the current image. If no person is found sitting in front of the laptop and looking at its screen, simon is automatically disabled. As soon as the user looks at the computer again, the system is re-activated.

This is just a proof of concept at this stage but is already working quite well. I hope to extend this into a regular simon plugin that could use simons filter system to toggle recognition for simon 0.4. Yes, we are going multimodal :)

simon at the CeBIT OpenSource 2011

2010-12-21T02:18:00.000-08:00

Thanks to the generous support of the Linux New Media AG, the simon listens e.V. will have a stand at this years CeBIT OpenSource event.

The CeBIT is the biggest IT event worldwide with more than 400 thousand visitors last year so you can image that I'm pretty excited about this :)

I'm definitely looking forward to seeing some KDE folk there as well!

On a related note: Does anyone know a cheap place to stay at during the event? It seems like the superlatives of the event sadly extend to the hotel rates during the exhibition...

Robotics: Research Laboratories Tour

2010-12-08T04:47:00.000-08:00

A couple of days ago, we received a rather intriguing mail: Our project partner on the current ECHORD EU-Project told us that they would, in the process of getting an overview of current developments, undertake a tour through the most promising robotics research laboratories in America and Asia.

Now, I'm not exactly an expert when it comes to robotics but maybe you are?

Maybe you know a really great lab that works on cutting edge technologies related to robotics (especially human / robot interaction)? Maybe you even work at one?

If so, please post in the comment section or simply drop us a line at office (ate) simon-listens.org. Thanks!

CeBIT Open Source 2011

2010-11-25T11:57:00.000-08:00

Just a tiny update: This years CeBIT again contains a special section called "CeBIT Open Source". This dedicated section contains funded exhibition stands for selected open source projects.

I have just sent out the application form for a simon stand at the CeBIT 2011 :)

Whish us luck!

Integrating speech recognition with other applications

2010-11-23T01:14:00.000-08:00

As many of you will probably already know, we pay a lot of attention to make speech recognition actually usable by integrating it with existing applications. We do this by simulating conventional interaction patterns (mainly mouse and keyboard) through our command infrastructure.

simon 0.4, however, will also allow application developers to use the speech recognition much more effectively by providing plugins to call DBus and JSON functions.

If the application to be controlled has either of those interfaces, you can utilize these new command plugins to write simon scenarios that call methods in the application through the IPC layer. This way you can directly execute code with voice commands which makes the system much more robust and powerful than, for example, using global shortcuts for the same purpose.

Moreover, simon 0.4 provides a dbus interface to allow third party applications to execute simon commands directly as well.

How simon learned to talk

2010-11-07T04:58:00.000-08:00

Finally I find the time for a long overdue Blog update :). I already promised this in September when I blogged about the dialog system but I want to write a bit about simons text to speech infrastructure.

Because the next version of simon will be able to interact with the user through dialogs, we wanted to enable simon to actually "talk" with the user through the means of text to speech systems.

Of course we didn't reinvent the wheel but rather looked around at available open source solutions. We needed it to be cross plattform and work at least with English, German and Italian.

Naturally, Jovie (formerly KTTSD, KDEs text to speech system) is the obvious choice but it is not yet cross plattform as it uses speech dispatcher which only works on Linux. Also, it wasn't very stable when I tried it and had quite a few rough edges and missing features.

Furthermore the best (open) German voices I could find where HTS voices developed with and for the OpenMARY framework. They should theoretically also work with festival so they could be used with Jovie as well if someone wrote a festival configuration set for it. OpenMARY is cross plattform and provides very high quality synthesis but is a very big and heavy Java dependency which needs a lot of resources and is quite slow - even on current hardware (synthesizing a paragraph of text takes around 10 seconds on a Nettop).

So we decided to do what we always do and leave the final choice to the end user:

simons TTS framework now allows you to use Jovie (default), a generic webservice (like OpenMARY) or to record sound snippets yourself.

The last option is especially helpful if you are dealing with languages where no good open voices exist yet or your users who have trouble understanding them.

Simply create a new TTS set for your speaker (the one recording the sound bytes) and record the needed texts with him / her. When recording texts, simon will show you a list of recently synthesized texts so you can easily record whole dialogs quite quickly. Instead of using the Jovie or OpenMARY to synthesize the text, simon will then play back these recordings.

These TTS sets can be exported and imported so you can share your sound snippets with others - for example accompanying the scenario containing the dialog which uses them.

Multiple TTS backends can be used simultaniously which means that you can use pre-recorded sound bytes primarily but fall back to a TTS system for dialog paths you have not (yet) recorded.

You can find an online demonstration of the OpenMARY voices on their homeage and a demonstration of simons dialog system using Jovie on youtube.

openSUSE conference 2010

2010-10-24T03:46:00.000-07:00

This years openSUSE conference has sadly already ended. It was a nice event with lots of cool people.

As planned we arrived just in time for the conference party which of course was a lot of fun - I even got a genuine Jos Hug (tm).

People tried to steal the big plush Geekos all night long. The above attempt was - surprisingly - not successful :)

Altough we were only there for a couple of days, we got to see a bit of Nürnberg and it's a very nice city. Especially the K4 "Kulturzentrum" (literally translated: "centrum of culture") which hosted fridays open movie night was very interesting.

The simon talk yesterday was also very well received (we actually took about twice as long than scheduled because apparently nobody needed the room) and quite some people jotted down the contact details. Hopefully I'll be hearing from you soon!

simon at the openSUSE conference

2010-10-06T14:59:00.000-07:00

Just a quick update: I'm proud to say that I will give a talk about simon at this years openSUSE conference (20.-23. October in Nürnberg).

To make up for being the only simon listens guy at Akademy, I'm going to bring a colleague of the simon listens e.V. this time: Mathias Stieger.

The talk is scheduled for Saturday but we'll try to make it there as soon as possible - probably Thursday. After all we're not going to miss the party, right?

See you all in Nürnberg!

Monolog++

2010-09-29T12:48:00.000-07:00

simon 0.3.0 has been released about two weeks ago and this means that even tough temperatures outside disagree it's once again summer in trunk!

Our newest addition is actually one that has been in the works for quite some time (in a separate branch) and presents itself to the user as a new plugin: The Dialog plugin.

While simon 0.3.0 is ideally suited to silently execute the commands you tell him, the next version of simon will talk back.

The basic idea is quite simple: The dialog system lets the user define an arbitrary amount of states which each has some transitions to move on to other states of the dialog. Each transition can also execute other simon commands if configured to do so.

An example use case could look like this:
Every day at 10 am the system displays a dialog that asks the user if he has already taken his medication. Yes -> "Great!"; No -> "Do you need help?" -> etc.

But you could also create quite complex menus like this:
You: "Computer!"
Computer: "Hi! What do you want to do? Say any of the following options: Read e-Mails; Browse the web; Check calendar; Close"
You: "Check calendar"
Computer: "Alright. These are your upcoming events: Birthday at Susies Place in Graz"
You: "Where am I?"
Computer: "You are in Graz, Austria."
You: "Thanks"

Sounds ridiculous, right? Well, let's look at this example in a little bit more detail...

States, Options and Transitions
The above dialog is a quite simple state based dialog. You can see three states: "Welcome", "Calendar" and "Location". We can define them in the dialog configuration.

You can see that dialog options to continue to other states can be added there as well. The text of the state actually goes through a templating engine so you can define paramters for that in the "Template options" page.

But more interesting is probably the "Bound values" page. There you can define variables and bind them to values. Those values can be static, determined at runtime through QtScript (Javascript) or values from plasma dataengines.

For example you can bind $currentTime$ to the Local/Time of the date and time plasma data engine. And because there already lots of great plasma data engines this means that the dialog system is already quite powerful.

TTS
Remember that I said that simon will talk back to you in simon 0.4? Yes we also have an all new TTS layer but I'll cover this in a separate blog post as this one is already too long :)

Demo
So to show off the current state of development, I created a very short demo video displaying the dialog above. While the code is not production ready, I didn't cheat: both the upcoming events and the location is determined dynamically, at runtime, through plasma data engines (upcoming events use the calendar data engine to get data from your Akonadi calendar).

For RRS readers here a direct link.

simon at the AAL Forum 2010: Again

2010-09-19T09:32:00.000-07:00

The whole simon listens team attended this years AAL forum in Odense, Denmark.

The AAL Forum is a plattform for projects of the ambient assited living joint program of the European Union and related projects. After having attended the Akademy this year it was quite interesting to see the other side of software development with almost all the projects there being quite well funded :).

Despite the very steep attendance fee (€ 450) the exhibition was quite active. In just three days we collected more than 30 business cards of interested people - many of them looking for project partners for their next projects.

All in all it was very interesting so to see related projects, discovering similarities and potential synergies. Cooperations across projects - even in the same call - are still far too seldom in my opinion.

Quite some people were surprised to find out that simon is open source and completely free ("Where are the hidden costs?") so we also got to introduce some people to the concept of free software.

On Thursday we we then got to see Dj Ruth Flowers at the networking dinner. And it was just awesome seeing so many suits dance :P.

I really wouldn't have thought that such a formal event could be turned around into a wild party just with a good DJ. Suffices to say: The booths were quite empty the following day.

We then spent the last evening in Copenhagen before flying home which as it turns out is a great city - and they have great cocktail bars as well :)

Application centric speech recognition for your desktop: simon 0.3.0 released

2010-09-13T11:12:00.000-07:00

The new version 0.3.0 of the open source speech recognition simon has been released and boasts the all new scenario system allowing you to build your own customized speech recognition system with just a few mouseclicks.

With simon you can control your computer with your voice. You can open programs, URLs, type configurable text snippets, simulate shortcuts, control the mouse and keyboard and much more.

Because of simons architecture, it is not bound to a specific language and can be used with any dialect. It is also specifically designed to handle speech impairments which makes simon a viable alternative to conventional input methods especially for physically disabled people and senior citizens.

simon is based off the open source large vocabulary continuous speech recognition engine Julius.

New in simon 0.3

simon 0.3 introduces an application centric approach to speech recognition by using packaged use cases of the speech recognition called "scenarios". Scenarios contain the complete configuration for one specific task like controlling Firefox or using the voice controlled on screen keyboard. These scenarios can then be shared with other simon users and are collected in a central online repository which can be accessed directly from within the application.

Besides the scenario system the new version also provides the user not only with the possibility of creating his own model through training but also to use an existing acoustic model (base model) to get started even quicker - entirely without training.If the user wants more control or would like to improve recognition accuracy, personalized training is possible through the optional HTK (not included in simon due to license restrictions). simon then offers to adapt the used base model to your own voice or to create a new model entirely from scratch.

Additionally, we have been working hard to make simon even easier to use. Some of the more notable results of these efforts are the new introductory wizard that guides you through the initial setup as well as the speech model generation adapter that automatically fix a vast variety of common beginners mistakes for you.

Furthermore simon 0.3 introduces three new applications to the suite. Sam, an acoustic modeling tool is geared towards professionals who want to tinker with their speech model and get the best recognition out of it. It is also a great tool to create and test large models which can then be distributed as base models for other simon users. To create base models you also need a lot of speech data which can be easily collected through the newly introduced combo of ssc and sscd. ssc stands for simon sample collector and is the client to the sscd server. Together they provide a powerful, cross platform tool to collect samples from lots of different speakers - even allowing you to record with multiple microphones and / or sound cards simultaneously.

Demonstration

Readers of the RSS feed: Watch it on Youtube

Download

You can download simon 0.3 as source archive but there are also packages available for Windows, OpenSUSE and Ubuntu on our Sourceforge page. Up to date installation instructions are available on the simon listens wiki.

simon at the AAL Forum 2010

2010-09-12T01:37:00.000-07:00

I'm happy to announce that the simon listens team will attend this years ambient assisted living forum in Odense, Denmark!

The AAL Forum (15th-17th September) is an annual event as part of the Ambient Assisted Living Joint Program of the European Union. The main goal of this program is to improve everyday live for healthy seniors.

While this might not sound as the most exciting topic at first, this is a fast moving, exciting field of research that covers everything from home automation to assistive robotics.

We will be represented through a booth in the exhibition hall and Franz Stieger, our chairman, will give both a short introductory talk simon listens and another one about the project in the context of robotics enabled assisted living.

It isn't all work and no play, tough. In the spirit of the conference an internationally proclaimed 69 year old DJ called Ruth flowers will apparently rock the the social event of the conference. I can't wait to see that :)

Astromobile Kickoff Meeting or Creating a Terminator powered by KDE

2010-07-17T06:01:00.000-07:00

It's a very busy time for simon :). On Saturday I came back from a weeks worth of hacking and socializing on this years Akademy in Finland and just a couple of days later the whole simon team set off to the our next adventure: From Tuesday to Thursday we were attending the kick off meeting of our Astromobile project at the facilities of our project partner: The ARTS Lab of the Scuola Superiore Sant' Anna in Pisa.

In the Astromobile project we are trying to help seniors who are well able to take care of their daily routine but would benefit from a little assistance here and there.

Often these people don't want to have a full time care giver staying with them both for financial reasons but and to preserve their autonomy.

The Astromobile project tries to address this issue and bridge the gap between living autonomously and full on assisted living by creating a special robot that is able to for example remind seniors to take their medication, provide a way to call for help in case of an emergency as well as using modern technologies like video chats to set up a communication link with family and friends.

In this project, simon listens is responsible for the touch screen and the voice interface to the robot using our KDE4 based simon system.

Just after the introduction of the Scuola Superiore Sant' Anna by one of their professors our robot plattform - the SCITOS G5 from a german company called MetraLabs - arrived.

After coordinating the most important steps we received a short introduction to the internal workings of the robot plattform and were pleasently surprised when we booted it up for the first time.

Yes you are seeing the KDM login screen of Fedora 12 :)

(Full disclosure: The robot contains both GNOME and KDE and actually logs into GNOME by default - but still :)

We then got a quick walk through on how to talk to the many sensors, how to use the integrated path planning, mapping features and the script language, AngelScript, that is used to communicate to the robot platform.

During our stay we also got to see the living lab of the ARTS Lab which is basically a 200 square meter appartment containing lots of smart home prototypes. This is also the place where we will be testing our solution with elder people.

While we were in the area we also visited a pilot project which deployed automatic garbage disposal robots called "Dustbots" which were also developed by the Scuola Superiore Sant'Anna.

They are running Ubuntu, btw :)

simon says: Hello Planet KDE!

2010-07-17T03:46:00.000-07:00

This is the first blog post that is going to be aggregated to the planet and as such I feel that a short introduction is in order.

My name is Peter Grasch and for the past couple of years I have been working on an open source speech recognition software called simon.

With simon you can control your computer with voice commands.

simon uses the KDE 4 libraries, Julius and the HTK and is developed under the GPL license. You can find more information on our sourceforge page and in an interview on the dot.

Together with Franz Stieger, Mathias Stieger and Alexander Breznik I am also chairing the non profit research organization called "Simon Listens" which uses the simon software and other open source projects to research speech recognition and its applications through multiple research projects funded by the Austrian Benefit project and the European Union.

Through one of this projects I have also been lucky enough to be able to attend this years Akademy where I gave a talk and held a workshop about simon.

In Tampere I got the opportunity to, among many other interesting people, meet some of the KDE accessibility guys: Jeremy Whiting and Gunnar Schmidt. We discussed the current state of accessibility in KDE SC, the most pressing problems and how they could be tackled.

There is definitely a lot of work laying ahead of us but there are also some highly motivated people (yes Jeremy, I am looking at you :) working on this.

So lets join forces, buckle down and try to make KDE SC accessible to everybody!

Day 4 at Akademy

2010-07-06T08:34:00.000-07:00

Well the conference part of Akademy already ended on Sunday so for the past two days I've been hacking at Demola.

So far:

I met Jeremy Whiting, a KDE accessibility guy and had a very interesting discussion about kttsd and speech dispatcher.

I attended the KDE Accessibility BoF which was sadly more like a list of things KDE desparately needs to do. To all KDE developers out there: We need to do more!

I became a "Nokia Certified Qt Developer"! Yay :)

I broke my entire desktop because after seeing all those cool features in the current KDE trunk I just had to try it :P

And of course I am already working on new features in the simon suite (ssc at the moment, actually).

Also, my presentation is already online.

To everyone at Akademy or somewhere near Tampere: The simon Workshop is tomorrow from 9:30 to 11:00 in Area 2. Hopefully I'll see you there!

simon at Akademy

2010-07-04T13:31:00.000-07:00

Ok first of all: Akademy is awesome.

But today, for me, it kinda got serious: My first international talk :)

Well everything went fine - more or less - and even tough we were a bit pressed for time (aseigos keynote ran a bit longer but it was well worth the time), I think I got the most important ideas behind simon across.

The presentations were all recorded so those who where not able to be there in person can watch it online soon.

So far the feedback has been overwhelmingly positive and has resulted in many fruitful discussions.

In other ~~news~~ common knowledge, KDE devs are really nice and exceptionally smart people so I am really looking forward to the coming days and the upcoming hacking sessions, workshops and BoFs.

simon 0.3 alpha 3

2010-07-03T08:05:00.000-07:00

Just a short post because I am actually sitting in an Akademy presentation right now: simon 0.3 alpha 3 was released which contains some critical fixes (no new features).

You can download it on sourceforge.

New test version: simon 0.3 alpha 2

2010-06-24T10:02:00.000-07:00

Well, Qt 4.6.3 has been released and as promised, so has simon 0.3 alpha 2.

What's new with this release? Well: A lot.

The most visible change is probably the completely revamped soundstack which is something I was meaning to do for a long time. Because usually this would have been as dull as it sounds, we managed to spice it up with some serious features:

Internal sound server takes care of multiple recordings / playbacks at once; An especially developed priority system tells simon if other streams should be stopped or are supposed to run in parallel.
Can handle more than one device simultaniously! The UI and logic have been adapted as well and with the current version you can:
- Record your trainings samples with two (or more) microphones at once
- Use multiple devices for recognition (for example in different rooms)
- Use multiple playback devices for example to drive speakers and headphones at the same time
Volume calibration helps you to determine the optimal recording value to get the most out of your trainings samples.
Pause notifications ensure that you don't cut off important information in your trainings data to ensure best results with as little needed data as possible.
Even the sample collection engine and sam were adapted and the SSC / SSCD system even stores information about the used recording devices in it's database. This way you can reconstruct which device recorded what samples possibly finding faulty / bad quality samples in large databases even faster.

But don't think we "only" worked on sound handling...
We removed any trace of Julius on the client and replaced their adin system with our own sound streaming including our own level based voice activity detection which is of course fully configurable through the graphical interface.

The recognizer interface (simond) now only needs to be able to recognize wave files (which I guess most recognizers are able to do) so replacing Julius gets even easier.

Handling of alternate keyboard layouts was improved and simon is now able to "type" for example Greek letters.

This release also contains the announced welcome wizard to make the initial configuration even easier.

Packages are currently being built and uploaded.

Happy testing!

What happened to 0.3 alpha 2?

2010-05-15T00:24:00.000-07:00

It's been a whole month since the last release of the 0.3 series.

The reason for this is not because nothing happened since then (quite the contrary) but rather that we are waiting for the next patch level Qt release.

For 0.3 alpha 2 we finally switched from portaudio to QtMultimedia which fixed some long standing issues on Ubuntu. However, it also introduced a few new issues because Qt 4.6.2 still has some nasty bugs in this area.

Luckily, the Qt software team is incredibly responsive and the most important bugs have already been fixed in their current development version.

Because two of these bugs are blockers for simon (QtMultimedia always uses default output device on Linux, Not supporting 16 kHz recordings on Windows) we sadly have to wait for Qt 4.6.3 to be released before we can release a new test version.

simon at the Akademy 2010

2010-05-12T00:11:00.001-07:00

The conference program for this years Akademy is online and it looks like we got a pretty sweet timeslot.

Scenarios coming to life

2010-05-12T00:01:00.000-07:00

With simon 0.3 we introduce the scenario system: A package based recognition architecture allows users to simply choose which use cases they want the speech recognition to handle from a vast online repository.

That's the idea. The problematic part is that "vast online repository". We obviously can't anticipate all use cases of simon, nor do we have the necessary manpower to design all of those scenarios. But what we can do - and what we already did - was to make it very easy to create and share scenarios from within simon.

Now we depend on the community to pick up the concept and start creating / uploading scenarios.

This is why I was excited to see the first user contributed scenario uploaded to kde-files.org: The day before yesterday, Ken Maclean (of Voxforge fame) created and uploaded a scenario to control the music player rythmbox.

Considering a stable version of simon hasn't even been released yet I am looking forward to many more user contributed scenarios being uploaded soon!

Want to get involved? Contact us at support ate simon-listens dot org to find out more!

simon at the LinuxWochen 2010

2010-05-05T23:48:00.000-07:00

This Saturday, the 8th of May, we will present the current prototype of simon at the biggest IT event in Austria - the "Linuxwochen" in Vienna (program).

The admission is free.

Akademy

2010-05-05T11:11:00.000-07:00

Our talk proposal for the Akademy 2010 was accepted!

Still waiting for the "official" banner to insert at this point but:
I'm going to Akademy!

Goodbye Portaudio! Long live QtMultimedia!

2010-04-17T23:57:00.000-07:00

The sound stack of simon was long a source of many issues. This was mainly because it relied on portaudio which sadly isn't supported that well by the sound configuration of e.g. Ubuntu because it interferes with their Pulseaudio setup. Long story short: Users of Ubuntu often had completely unusable simon installations because it crashed often and seemingly at random. Because those crashes happened in portaudios code and not in simons, there was little for us to do.

In the last week, I finally found some time and threw out all the old sound handling code and replaced it by a completely new, QtMultimedia based system. QtMultimedia is still a very young library and too has issues but I suspect that those will get fixed pretty quickly.

While I was at it, I also implemented a much cleaner way to stream audio to simond. Older versions used Julius libsent to do this because of their voice activity detection implementation. We now implemented a similar system (configurable, level based voice activity detection) in simon and now have complete control over the audio stream. Because of the new implementation I also implemented the feature to keep recognition samples - complete with their recognition results - on the server. This could for example be used to gather training data during normal usage. All you'd need to do is check if the words were correctly recognized and add them to the model.

Because all sound in/output is handled through a central point, I implemented a quite primitive sound server that will handle multiple simultanious streams correctly. Recordings while simon is activated will now work much faster (because the sound device handle will simply stay open) and are of course completely stable. You even get automatic pausing / unpausing for interrupted streams (If you for example start to record one sample, while recording this one start to record another sample the first one will pause until you are done recording the second).

The new implementation also has a much better level meter integrated into the recording widget so you can check your current microphone volume while you record. If you start to clip, simon will now automatically display a warning message telling you to re-record the sample.

Btw, QtMultimedia also works e.g. on Symbian devices so a simond client on a mobile phone should be trivial now.

All this has already been merged to the master branch and works very well in my tests. However, just like any new code it might contain bugs so try it at your own risk :).

Usability

2010-04-11T07:49:00.000-07:00

Considering that simon was designed to be as easy as possible, someone who just downloads and installs simon might say that we failed.

To many new users the concepts behind simon are - at first - too complicated and simply getting the recognition to work seems needlessly hard.

However, those that stick with it seem to "get" the ui pretty soon and it proves very powerful for expert users.

This is why one of the goals for 0.3 was to make this initial learning curve as flat as possible at get simon up and running quickly.

After I released the first alpha of 0.3 about a month ago, I posted a review request to the KDE Usability mailing list asking them for ideas of how to improve our interface. I got great feedback (thanks!) and it quickly became clear that it would be best if simon provided an assistant on the first start that would guide new users through an initial setup.

Some users might remember this concept from 0.1. Back then we had a first run wizard but it took ages to complete because it consited of dozens of pages with quite complicated instructions.

In 0.3 however, with the introduction of scenarios and base models we designed another such wizard. It now includes just 5 pages (including welcome and finish pages) and includes short but precise instructions for every step. If the users follows through, they will be rewarded with a completely functional simon within minutes.

We are still fine tuning the wizard (updating the descriptions, everything is already fully functional) which is why it still resides in its own branch ("hci") for those of you who want to check it out.

The wizard will be included in the next release.

Thanks again to the KDE usability team for their valuable input!

simon 0.3: First alpha released

2010-03-09T14:13:00.000-08:00

The first alpha of simon 0.3 was just released.

simon 0.3 is not officially feature complete but this is basically what you will get when it is released.

simon 0.3 alpha 1 does not replace the current stable version (0.2) but should only be installed by interested testers.

simon 0.3 alpha 1 at sourceforge
Changelog

Cheers!

Benefit project

2010-02-24T01:26:00.000-08:00

On the 1st of February 2010 the friendly society simon listens started to work on a new project. For the next one and a half years, the simon listens team will investigate ways and means to make the simon speech recognition solution even more usable – especially for the elderly.

Abstract:
With the help of verbal control provided by simon using terms of everyday language, useful scenarios and areas of application shall be created to enable an easy use of new communication technologies such as the internet, telephone and multimedia applications for elderly people. Moreover, additional security can be provided, for example, a reminder for the user to take a medication.

In the course of this project we will join forces with the Signal Processing and Speech Communication Laboratory of the Graz University of Technology, the HTBLA Kaindorf/Sulm, the Rehabilitation Clinic Maria-Theresia, the KFU Research Center for Austrian German and the Huminatis Graz to ensure that we have the necessary expertise to tackle such an ambitions project.

The solution created in this project will be released under the GPL license. All code will be freely available to the community.

Thanks to the generous support of the bmvit (federal ministry of transport, innovation and technology) of Austria and the FFG (Austrian Research Promotion Agency) for making this possible!

Model Compilation Adapter

2010-02-23T07:54:00.001-08:00

In simon 0.2 we introduced some mechanisms to catch common errors during the compilation of the model and display nicer error messages to the user explaining ways to solve the issue manually. In simon 0.3 simon, however, simon will automatically repair some common mistakes without the user even noticing.

To explain what I am talking about, I first have to talk about simons architecture a bit so bear with me...

During normal operation, the simon client gathers the instructions (words, grammar, etc.) that will then be sent to simond. simond in turn compiles the model out of the given input files. To do that, simond first converts them to a format usable by the underlying tools (HTK, Julius). This conversion step was not needed in 0.2 because simon 0.2 only used the raw file formats of HTK / Julius. However, in simon 0.3 we need more control over the model and also want to give the user some advanced features that were not possible with just the information contained in those raw formats.

In simon 0.3 we introduced a new step between gathering the data and compiling it to a usable model: Adapting the input files.

This sounds like a boring but nescessairy conversion and indeed it is.

But what makes it interesting is that at the point of adaption we have all the input data that will be turned in to a mode in a format that is easily parsable. This means that it is an ideal place to do some last minute optimizations on the temporary files that are then used to generate the model.

The model adaption manager will for example automatically remove words from lexicon that have no training data associated. It will also clean the grammar of sentences that have no associated words. It will even remove samples containing words that are not in your dictionary. Basically, simon should be able to handle a lot of case that would cause an error in simon 0.2, automatically.

Git

2010-01-24T11:38:00.000-08:00

Like all the cool kids, simon moved to git.

And so far, while the syntax is a bit strange at first it works really, really well.

The cheap branches are great and svn2git made the transition painless.

Repository URL (read-only):

git://speech2text.git.sourceforge.net/gitroot/speech2text/speech2text

The old svn repository is still available and up to date but if everything works well enough it will eventually be removed in the next couple of days.

Model adaption

2010-01-17T05:55:00.000-08:00

Keeping in line with the last couple of blog posts that all were breakthroughs on their own, this one is definitely up there as well.

Since revision 1117 simon now supports to use static models or adapt speaker independent models to your own voice in addition to building a new, speaker dependent model from scratch (which is still the default obviously). This means that new users can set up a complete working speech recognition literally in seconds. Pick the scenarios you want, point simon to the voxforge speech model, press "Connect" and start talking.

Of course this only works if you have a fairly "standard" voice and the voxforge model is still not perfect. So if you want a little higher recognition rate go ahead, train a few samples and tell simon to adapt the voxforge base model with it. As little as one minute of speech will yield visible results (you will still need to install the HTK for this, tough).

The only user interaction needed is to click a radio button - simon will do all the work for you.

While I was at it I also improved the julius error reporting so that the recognition process now writes a log file (~/.kde/share/apps/simond/models/<user>/active/julius.log) so that you can easily debug low recognition rates, mic troubles etc. When the recognition fails completely, simon will display the log along with a short description of what simon thinks that happened.

Of course all of this is completely untested and will most likely contain bugs so try it at your own risk. By the way: Current trunk needs KDE 4.4 to compile.

SSC: Large scale sample acquisition

2009-12-26T09:22:00.000-08:00

Two new applications have joined the simon application suite: ssc and sscd.

ssc stands for Simon Sample Collector and is specifially designed for large scale sample acquisition.

sscd is the central data server which it's data in a MySQL database (no graphical frontend).

ssc is the graphical client that records the samples, manages the users and institutions and uploads the data to the server.

We use the software to collect samples in various medical institution (rehabilitation clinics mostly at the moment) for further analysis. Every user of the software has a quite extensive profile which stores birth year, education, mothers tongue, diagnosis and other relevant factors that may influence the speech.

The system allows the user to create "institution" (the clinic for example) and associate users with zero or many institutions.

ssc users are not interchangable with simon users (more information) and there is no attempt to creating speech models with ssc / sscd. Those tools are simple data-acquisition tools and probably only useful if you are a speech researcher who wants to gather data from a lot of speakers. Then, however, it is a neat little utility that works very well for our uses already.

simon scenarios

2009-12-26T08:51:00.000-08:00

First of all: Merry Christmas to everybody!

Some of you might still remember that back when simon 0.2 was released, I talked about modularizing the recognition into smaller "packages" or "scenarios". Well this features has been implemented now and is almost ready.

The idea is to provide pre-defined application packages (like default values on steroids) as well as the ability to create such packages and share them with the community easily. This should significantly improve our biggest weakness: The time it takes a new user to get started with simon.

To show you what I mean, I created a little screencast that shows the current development version of simon in action:

Automatic BOMP import

2009-09-16T08:46:00.000-07:00

Disclaimer: This Blog post is probably only interesting if you want to use simon for the German language.

If you do, then you most likely already know the problem with the BOMP dictionary. It is a great quality, huge dictionary that even contains terminal information. In short: It is the best shadow dictionary for simon there is - and I am not saying the best for the German language; it's the best. Period.

Only problem: The licence. It doesn't permit free distribution. Instead, you had to send an e-Mail to a specific address at the university of Bonn and wait for a reply that contained the dictionary as attachment. This process could take a couple of days.

So we contacted the team that is holding the copyright and they permitted us an exception in the licence. We are now allowed to distribute the dictionary to our users ourselves as long as we still gather their name and e-Mail addresses (for statistic purposes) and they still have to accept the BOMP licence terms. So we integrated this process directly into simon. When selecting to import a HADIFIX dictionary you can now select to manually specify a file or to download and import the HADIFIX BOMP automatically.

Calculator Plugin and Keyboard Plugin

2009-08-28T13:17:00.000-07:00

Thanks to the "Österreichische Forschungsförderungsgesellschaft" (literal trans.: 'Austrian Researchfundingassociation') the SIMON listens team has been expanded with the two summer interns Mario Strametz and Dominik Neumeister.

After some general testing and getting to know the system, they are now working on two promising command plugins: A calculator plugin and a keyboard plugin.

The calculator plugin is a natural extension of the existing input-number-plugin.

As seen it is still quite basic but already usable to a certain extend. However, it is under heavy development and we expect first stable versions by the end of next week.

The calculator is - beside the obvious - also targeted to school kids doing their math homework so upon pressing ok it provides the option to not only write out the result but also the calculation leading up to it (e.g.: "1+1=2" instead of just "2"). The finished version will also include formatting options like formatting the output as an amount of money, etc.

At the same time, the two are working on a keyboard plugin (no screenshot there yet as development has just started). However, our "keyboard" will not only be a regular on screen keyboard.

The keyboard plugin will not have a fixed amount of fields (keys), nor will their values be fixed to that of a qwerty keyboard.

Instead, the user will be able configure them as he likes in configuration sets (sensible defaults will of course be provided) and even spread the keys out across multiple tabs.

While this just seems overly complicated on paper it makes advanced configurations possible with e.g. a text-snippet tab that combines his most often used text snippets or allows the user to add - for him - important special characters (e.g. Currency symbols for an accountant) right where he wants them.

I will update this blog as the development progresses so check back!

Publicity

2009-08-28T12:34:00.000-07:00

Hi fellow readers! Long time no see!

As some of you might have seen, there was an article about simon on the dot. Thanks for Troy Unrau for making that happen!

The article spawned a lot of discussion and interested and several sites brought it up. Most notably the discussion on lwn focusing on the license issues. The article also hit digg (50 digs), osnews, several twitter/identi.ca feeds and a lot of blogs everywhere.

Of course this also showed on our download statistics. We had more downloads in the last week than we had in the whole month before that! The forum has also been noticeably busier than usual but the low number of support requests showed that the extensive documentation of simon 0.2 really helps a lot.

The simon homepage runs google analytics so there has been quite some interesting data about our (newly found) user base:

55% of all visitors were running GNU/Linux (Windows: 39%; Mac: 5%)

Our 5000 hits were spread out to 106 countries using 75 different languages; The most used languages were English (2500), German (1000), French (500), Chinese (200).

In the open source scene, firefox rules the browser battle (58%)

More people are using konqueror (9%) than Internet Explorer (7%) (of course this is because of the KDE-specific audience this month but I still found it interesting; konqueror was actually on 2nd place after firefox)

sam

2009-08-06T11:31:00.000-07:00

I already mentioned it in the last post: A new application has been added to the simon application suite: sam.

sam is targeted towards power users who want to tweak and improve their acoustic model manually to improve recognition rates even further.

sam will include a sophisticated testing framework to immediatly receive feedback on changes in the model configuration. In fact during optimizing models manually, I realized that IMHO a well working, automated model testing framework is the most essential part in manual optimization as it makes the impact of changes immediatly visisble.

In contrast to simon, sam will not hide any of the internal workings from the user (due to the different target group) so the logs of both the building and the testing of the model are displayed and the whole operation can be double-checked for errors or warnings.

An initial, working version is already available through SVN.

Selecting the input files:

Building the model:

Testing the model:

Test results:

As you can see, simon will run the recognition with the generated models on the trainingssamples to see if simon correctly recognizes their contents. The algorithm already recognizes and considers confidence scores of the recognition results which is why in the screenshot you can see the recognition rate of e.g. "NULL" not being 100% even tough every instance of it was recognized correctly (5/5).

Btw: This is a well trained, rather small model which really works very well in practice so don't be alarmed by the very high recognition rate...

Greetings,
Peter

Look out - cool stuff coming your way!

2009-07-30T08:30:00.000-07:00

Ok I have way to little time at the moment for simon development let alone regular blog updates.

However here is a quick overview of the latest updates:

simon can now import dictionaries to the active lexicon. While you obviously not want the whole BOMP or Voxforge Dictionary in your active dictionary it is a little step towards easy export and import of the speech model.

The URL to the BOMP has been corrected - they had moved.

simon can now import prompts files through the import training data wizard.

simon can now be launched through the ksimond context menu.

Some phoneme segmentation issues have been fixed.

And finally: A new application has been added to the simon suite: "sam".

sam stands for simon acoustic modeller and is an application targeted towards power users to tweak and test their speech models. Of course sam is nowhere near usable right now but the first lines of code have been written so I thought I should mention it here.

simon 0.3: One Week In

2009-07-20T10:10:00.000-07:00

About a week ago, I announced the simon 0.2 stable release. Fueled by this milestone and a lot of positive feedback all around, simon 0.3 development has already started ... and is already showing results!

I'll start small: simon now supports a "Power Training" mode which starts the recording immediatly as the text to say is shown. The recording is then, upon preceding to the next page, automatically stopped, saved and the next one starts. This simple change really makes training of large texts a lot faster!

Ok but that alone is not blog worthy, right? Right! One of the most awaited features has made it's appearance: Confidence scores.

The recognition server now provides information about how confident it was on the recognition result

Moreover it also not only provides simon with the most likely result but with the ten most probable ones. simon now ranks them based on the recognition confidence and can ignore them if the recognition was just not sure enough (with a configurable threshold).

Now the cool part: If two results (or more) are very likely and simon can not determine which one you meant, simon will simply display a nice list from which you can select (of course with your voice) what you meant.

This looks like this:

The feature is already quite stable and works well in combination with other plugins. There are of course safeguards in place to prevent recursive "did-you-mean-popups".

Of course the confidence scores of the results are also relayed to the plugins and if they want to they can even retrieve the whole list of recognition results including the phonetic transcription of the result. This brings even more flexibility to the plugin developers without making plugin development more complicated (the base classes have appropriate implementations that you don't need to overwrite if you don't want the additional information).

If you are running a svn snapshot and are upgrading: You will need to manually copy the julius.jconf file from `kde4-config --prefix`/share/apps/simond/default.jconf to ~/.kde/share/apps/simond/models//active/julius.jconf (overwriting the old one) as simon(d) will not do that automatically.

simon 0.2 released

2009-07-10T09:48:00.000-07:00

Almost three years after the start of the development, the first stable version of the open source speech recognition suite simon has finally been released: simon 0.2 is ready for download.

With simon you can control your computer with your voice. You can open programs, URLs, type configurable text snippets, simulate shortcuts, control the mouse and more.

Because of simons architecture, it is not bound to a specific language and can be used with any dialect. It is also specifically designed to handle speech impairments which makes simon a viable alternative to conventional input methods for physically disabled people.

simon 0.2 is based off of the open source Julius speech recognition engine and the HTK (which - due to licensing restrictions - has to be installed seperately).

In comparison to the 0.1 series that never made it past alpha quality, simon 0.2 does not only bring stability improvements.

simon 0.2 is now based on KDE 4 and thus perfectly integrates in every KDE setup. This move also brings KIO to simon which allows for network transparency, transparent compression and more.

The seperate Juliusd application has been discontinued and replaced by the much advanced simond which features network audio streaming, centralized model management with automatic backups and more. simond is a command line application which makes it easy to set up a central simon server without the heavy X dependencies. For users of graphical environments the front-end ksimond has been introduced.

Moreover, the command architecture has been completely overhauled and now uses a much more flexible plugin architecture and supports individual triggers per plugin. New plugins include the list plugin (which can be used to display options), the composite plugin (similar to "macros"), a number input plugin and an artificial intelligence. Combined with the improved commands of previous simon versions this makes a total of 10 command plugins out of the box!

The import of the shadow dictionary now also supports PLS and SPHINX dictionaries which opens the door for dictionaries like the German GPL dictionary from Voxforge.

Because of the growing user base simon has been translated to English, German and French and also partly to Spanish, Dutch and Czech.

simon 0.2 is also the first version of simon ever to ship complete with an extensive user manual - available in English and German.

Next to the source package, the release is also available in convenient binary packages for 32-bit and 64-bit users of both GNU/Linux (Ubuntu and OpenSUSE) as well as Microsoft Windows operating systems and can be downloaded from the sourceforge project page.

Two Final Issues

2009-07-09T12:21:00.000-07:00

The last round of testing of the simon 0.2 codebase only resulted in two found bugs.

The first one is quite annoying in that it essentially limits simon functionality. The HTK does not like words that start with the character "'". That makes "words" like "'em" (short version of "them") fail during the model compilation with a confusing error message.

As I really don't want to mess with the wordlist code (we would have to escape special characters under certain conditions) so late in the development process, I delayed that fix for the 0.3 series. In the mean time just stay away from 's at the beginning of the word, please. Words like "that's" are no problem, tough (as the "'" is not at the beginning of the word).

The second bug was a rather strange one: Some people reported that over time, the recognition became slower and slower for them. All of the users that reported that bug were using Windows. During testing, I found out that using the pseudo device called "SoundMapper" (or similar) caused this - when using the hardware device everything was working. So if you experience this issue, please check that you use the appropriate hardware device instead of meta-devices.

For users that don't read the blog, I added entries for both problems in the troubleshooting guide on our wiki.

And yes, I know that these are hardly the last two bugs in the 0.2 code - but they are the last to be fixed before the stable release which makes them kinda special ... for me anyways :)

simond: Thread Termination

2009-06-14T09:23:00.000-07:00

Some users told us that when simon crashed / closed under certain conditions, the simond would be trapped in an infinite loop.

To accept any new connections, simond had to be restarted.

This was caused by the implementation of the TCP/IP adin. Once a client would be connected (data connection to synchronize the model), a new socket would be opened for the audio stream. This was implemented using the accept() method which blocks until a connection is made.

While this was in a different thread as the main event loop this was no problem most of the time. However, when the client connected through the data channel but didn't establish an audio connection this socket would be blocking in accept(). When the client exited, the thread should by terminated by the main event loop but the thread event loop would still block waiting for a connection.

This resulted in the debug output "QThread::start: Thread termination error" which was printed over and over again.

Because I really don't want to modify too much of Julius to keep it synchronized to the SVN version I added a little workaround in the stop() routine: I simply connected to the socket and disconnected immediately.

So starting with revision 862 simond should not hang any longer and should be more stable in general.

Manuals

2009-06-13T12:42:00.000-07:00

The manuals for simon, simond and ksimond are now also available as PDF files at our wiki.

Also, I managed to get khelpcenter on windows working. Sadly this really blows up our windows package as this adds the whole khtml library to it... Anyways: Say hello to F1 in the simon 0.2 final!

Overcoming Limitations

2009-06-12T12:39:00.001-07:00

Remember the post a couple of days ago about the limitations of the event simulation?

Well they are history now :)

To fix the problem with the dead keys, I added a hash table to the CoreEvents class (the plattform independent part of the event simulation backends) containing the unicode characters ('â', 'é', etc.) and what characters it consits of (in case of 'â': '^' and 'a').

After implementing this and finding all the common dead keys (there are a lot of them!) I had to restructure things anyway in the XEvents backend (to allow for AltGr+Shift at the same time) so the event simulation on X11 should now work better and faster too!

Long story short, simon now supports the following dead keys:

ŵêẑûîôâŝĝĥĵŷĉ

ŴÊẐÛÎÔÂŜĜĤĴŶĈ

ẃéŕźúíóṕǘáśǵḱĺýćńḿ

ẂÉŔŹÚÍÓṔǗÁŚǴḰĹÝĆŃḾ

ẁèùìòǜàỳǹ

ẀÈÙÌÒǛÀỲǸ

ȩŗţşḑģḩķļçņ

ȨŖŢŞḐĢḨĶĻÇŅ

ẽũĩõãỹṽñ

ẼŨĨÕÃỸṼÑ

ẉẹṛṭẓụịọạṣḍḥḳḷỵṿḅṇ

ẈẸṚṬẒỤỊỌẠṢḌḤḲḶỴṾḄṆṂ

ẇėṙṫżıȯṗȧṡḋḟġḣẏẋċḃṅṁ

ẆĖṘṪŻIȮṖȦṠḊḞĠḢẎẊĊḂṄṀ

ẅëẗüïöäḧÿẍ

ẄËTÜÏÖÄḦŸẌ

ẘůåẙ

ŮÅ

ēūīōāḡȳ

ĒŪĪŌĀḠȲ

ěřťžǔǐǒǎšďȟǰǩľčň

ĚŘŤŽǓǏǑŠĎȞǨĽČŇ

űő

ŰŐ

ęųįǫą

ĘŲĮǪĄ

I told you there are a lot of them...

French Translation

2009-06-09T04:15:00.000-07:00

Quick update: The user yanncantin translated simon to French.

Thanks!

The translations are not yet checked in but I hope to get that done by tonight.

Limitations of the Event Simulation

2009-06-09T04:00:00.000-07:00

simon provides the possibility to simulate user input like keystrokes or mouse clicks.

This functionality is provided by the simoneventsimulation library. The library internally uses plattform dependant backends for X11 (xtst) and Microsoft Windows (WinAPI).

The backends need to provide a common sendKey(unsigned int /*unicode*/) method mandated by the shared interface. Since neither Xtst nor WinAPI provide a way to "write" such a character directly we first need to find out how the user would generate such a key.

Internally both methods use a simple switch / case to first determine if this is a special key (like the "Home" key) and if not try to dertmine the keycode by extracting the "base character" (for example: The base character of € is e) and then trying to find out which modifiers (shift, altgr, control) were pressed to get to the keycode given.

This works fine for "normal" characters and special characters like €. However one group of "keys" is missing: The dead keys.

Dead keys are used to generate characters like â or é. They are generated by first pressing one key like ^, releasing it, and then pressing the next key. This is substantially different from modifier keys as there the modifier key is not released until the combination is complete.

This is why simon in the current form does not know how to handle characters that need dead keys. I hope to get this integrated before simon goes stable as this is a substantial feature for languages like French.

Portaudio and Pulseaudio

2009-06-05T05:27:00.000-07:00

So as I already said yesterday, I seemed to have missed the gamechanging "make-portaudio-work-with-pulseaudio" patch. So I went back, updated the sources and created new packages for the portaudio snapshot of yesterday.

Guess what? It didn't help a bit. simon still crashes seemingly random and nowhere (no backtraces) when you use portaudio and pulseaudio is active.

I give up. There is apparently no real way to have portaudio and pulseaudio coexist together peacfully.

So if you are using pulseaudio (which is the default of Ubuntu and apparently also on Fedora) expect ugly crashes from simon.

The only workaround I could find is to launch simon over padsp and select OSS devices in simon. That seemed to work fine (no crashes there).

Using pasuspender doesn't help at all.

If you do experience a crash due to pulseaudio it is usually bad enough (catastrophic) that it somehow manages to get simond in an infinite loop so you will want to restart that too if simon crashes.

I am sorry for the inconvenience but sadly there is little that I can do within a realistic timeframe. I don't want to get involved into either portaudio nor pulseaudio development so I keep hoping for phonon to develop a recording API.

Maybe for simon 0.3.

Bad Timing

2009-06-04T11:29:00.000-07:00

Ok this has to be bad karma or something.

I just finished packaging a portaudio snapshot because the stock Ubuntu-Version of portaudio is just too old.

So I upload them to the ubuntu bugtracker after careful testing on Kubuntu 32-bit and Kubuntu 64-bit. Everything works fine.

Yesterday I tested the whole setup (simon and the portaudio snapshot) on Ubuntu 32-bit and with it the Pulseaudio soundserver. simon crashes instantly when I deactivate it (which happens to happen all the time).

Back to square one. First I try to debug the problem but even when building every package I can find with debug information I get no valid backtraces at all. Valgrind doesn't help. Pasuspender? Crash. OSS devices? Crash. Nothing. The only thing that kind-of-works is using padsp and then using OSS devices in simon. But thats not good enough IMO.

So I start to look around and after quite some discouraging "there was some effort but now it is dead" posts on mailing lists and a lot of pushing the blame around (portaudio support in pulseaudio or pulseaudio support in portaudio?) I finally find light at the end of the tunnel. Apparently the problem annoyed enough audacity users that they integrated a patch into the audacity fork of portaudio that allows to use non-mmap devices which is apparently needed by pulseaudio.

David, It works!!!!!!!!!

Good enough for me. So I download the patch and a new portaudio snapshot and try to apply it. It fails. Of course.

So after going through it line by line and double checking the portaudio code I realize that the patch was already integrated. So why doesn't it work? Is the Audacity fork of portaudio just so much more advanced?

But then I look at the portaudio svn log and see this:

r1412 | aknudsen | 2009-05-24 18:54:22 +0200 (Son, 24. Mai 2009) | 2 Zeilen
Apply Kevin Kofler's non-mmap patch

Checking back to the date of the snapshot I used for packaging: 2009-05-19. I made the packages over the course of the last 3 days but used the "old" code, mind you.

So in an effort to update the stock version that is 2 years too old, I updated it with packages of a newer portaudio version that is ... 5 days too old. Oh the irony.

Ok I am off packaging portaudio again. And this time I am using the SVN version.

simon 0.2 and KDE 4.1

2009-06-02T11:11:00.001-07:00

simon and KDE backward compatibility has always been an issue. Because I use KDE/trunk to develop simon, I tend to use new classes as they appear.

However, OpenSUSE still has KDE 4.2 only in the factory respository and *buntu 8.10 only has it in its backports repository from jaunty.

So while we will not support 4.0 (Ubuntu Hardy), we decided support KDE >= 4.1. This means that users of all major distributions should have no problem installing simon 0.2.

Portaudio Snapshot Packages

2009-06-02T04:37:00.001-07:00

After I realized that a lot of the sound issues on *buntu can be traced back to the very old portaudio version used.

I packaged a snapshot from a couple of days ago and uploaded it to the Launchpad bugtracker.

Just uninstall the libportaudio2 and portaudio19-dev packages if they are already installed from the universe repository and install the new packages. This should fix a lot of sound issues (all of them if you are running Kubuntu and thus don't use pulseaudio) instantly.

Ubuntu 9.04 and simon

2009-05-20T12:47:00.000-07:00

Ubuntu Jaunty Jackalope and simon are not exactly best friends. If you tried it, you most likely experienced some very strange sound issues.

simon hangs, something else hangs, everything hangs...

I installed Kubuntu 9.04 today and had the very same issues.

Turns out the problem is a pretty old portaudio version.
The current snapshot fixed all those problems for me. You can get it on the portaudio website. Download the pa_snapshot.tgz, compile and install.

I haven't yet found a pre-packaged debian package to spare you the compiling part. If you know one, please let me know in the comments.

Also, it worked for me but obviously YMMV. Still issues? Again: let me know in the comments.

XML - Again...

2009-05-15T01:49:00.000-07:00

Ok it seems my last blog post has triggered quite a reply.

However, I think there still seems to be a bit of confusion. Let's try again...

Why am I so into XML-based standards? Because I understand them.

Even if this was not your intention, this is a bit misleading as it suggests that I don't understand XML files and if I would, I would share your points of view.

I can assure you that I understand the principles of XML files very well. Hell, a lot of parts of simon even use XML files (take a look at how the commands are stored for example).

But at the moment it just doesn't make sense to use a custom PLS modification (adding terminal tags). We need the lexicon to be readable by julius and HTK. This implies that we have to store the lexicon in that format (or add support to Julius; HTK is essential closed-source so we would still have to keep a seperate HTK lexicon around). It essentially does not matter what I believe is the better format for the job.

So the only possible way to incorporate a XML based dictionary storage format would be to add an additional layer. This, however, means that the features supported can only be the smallest common denominator of both formats. So no fancy IPA (no support in HTK), no nice multiple-graphemes per word (HTK could be compared to 1NF if you are familiar with database normalization), etc. In the end this additional layer would bring nothing beneficial to the table because we can't use it's nice features as long as we have to keep HTK compliant too. All it would do is introduce another source for errors.

All these considerations are irrelevant when we take Julius and HTK out of the equation. Then, adopting and modifying PLS is not such a bad idea (altough I would like to store commands and the dictionary in the same file for the upcoming package-based structure). Removing the dependency on HTK is something I would like very much but it doesn't seem feasible right now and in the near future.

And, by the way: I don’t like to read SAMPA. I prefer the IPA when editing the pronouncing dictionary.

The HTK does not support UTF-8. However, I would prefer using the SAMPA even if it did. I find that it is much easier to read and learn the SAMPA (especially if you speak german). Also, I do prefer to be able to transcribe my words with the keyboard instead of using sign-tables to pick out the symbols.

As the IPA and X-Sampa can be converted to and from each other without loosing anything I don't really see a problem there.

Sometimes, I ask myself the question: why don’t they switch from SAMPA to the IPA? Why don’t they switch their homepage from ISO-8895-1 to UTF-8?

This somehow confused me a bit. Our homepage is UTF-8 encoded? So are all the files produced by simon (except where it is not possible because of third party products that don't support it)...

Then I saw that you linked to the SPHINX homepage and not to our homepage...

export functionality is a low priority feature

OK. From my point of view, Voxforge needs an export functionality.

This is especially confusing as I was talking about export functionality of simon. You talk about an export functionality from Voxforge which would be an import feature from simons point of view. And as I stated in my previous blog post this is something that I am indeed very interested in.

I don’t know about the exotic BOMP standard, I couldn’t find an entry in the Wikipedia. So I assume that BOMP is not a relevant standard.

BOMP is no standard at all. It is a dictionary following the HADIFIX "standard". HADIFIX is a speech synthesis project that uses phonetic dictionaries to know how to pronounce the words. Those dictionaries have to follow a specific format which could be called the "HADIFIX standard" (I have not found a definition of it anywhere).
The import functionality was implemented because a very large, high quality phonetic dictionary (the "BOMP" dictionary) exists following that format.

Simon allows me to record just single words, not utterances. I am not convinced by that concept.

I wouldn't be either. Fortunately, this is not true. Take a look at the Training module. You can easily import "normal" Texts. Try to input a text file containing this: "I am an utterance. And here comes another.". Even the standard examples shipped with simon contain sentences and not individual words, btw.

You see, there are several aspects. The world is not just about simon. It is about Voxforge, too.

Please don't lecture me. It is disrespectful and unnecessary. I am very grateful of the effort that Ken and all the contributers put into Voxforge and actively promote participation when people ask me about dictation with simon.

I am also investigating how to best use the voxforge model with simon and have stated on several occasions that I have intention to integrate the possibility to contribute to voxforge from within simon.

Followup 16.05.2009
Today I have been contacted by ralfherzog by e-Mail where he explained the misunderstanding.

XML Standards: Clarification

2009-05-14T05:26:00.000-07:00

One of the largest contributers to the german voxforge acoustic model and one of the main contributer to the german GPL lexicon called ralfherzog keeps posting about simons (missing) import / export functionalities in his "testing simon"-blog. Normally I answer him directly per mail but I think this warrants a blog entry as this might be interesting to other readers as well.

First off some facts:

simon does support importing PLS dictionaries

simon does not support any explicit export functionalities what-so-ever. There are no export functions for the training data, the lexicon, vocabulary or anything.

simon does not support the import of training data based on a supplied prompts file - be that in plain text or XML.

None of those missing features are due to idelogical reasons but mostly due to time constraints. However, I am not as convinced as ralfherzog that they are that essential.

As far as I know, simon is the only application using PLS dictionaries so an export functionality is a low priority feature. The same goes for the training data. An integration with voxforge is planned for the future which would in my opinion be the only practical use case for export features right now anyways.

Some might wonder why we don't use PLS as the default dictionary format in the first place but the answer is very simple. The PLS standard does not allow for any terminal information to be stored with the dictionary. The current storage format is a standard Julius vocabulary file and an accompanying HTK dictionary. Those are the respective file formats of the underlying components and as they are not (yet) exchangeable I see no reason to introduce new file formats.

The import of training data is something that is included in simon 0.2 but only in a very basic form. Its current state is usable if you have training data gathered by a previous simon installation. However, everything else is not yet supported. I would personally like to see importing of a "normal" HTK prompts file but don't see the advantage in SSML. SSML is not designed for that paticular usage and just introduces unnescessairy overhead. Yes, content validation is a nice thing that makes XML a very good choice for many, many things but prompts are imho not one of them. So maybe we might see a import function for SSML formated prompts for data that is already gathered and stored in that format but making it the primary storage format of prompts in simon is probably not going to happen anytime soon. Its the same as with PLS: HTK expects the prompts in that format so why introduce an additional source of errors by introducing another conversion step?

Homepage

2009-05-14T05:19:00.000-07:00

We finally decided to bring our homepage up to speed.

Much of the information on it was (and still is) outdated and sometimes even plain wrong. So we decided to restructure it a bit to make the core information more accessible to the new use and to get rid of the outdated content.

Obviously, there is still a lot to do. But even just after the new menu was implemented, it is already much easier to find what you are looking for.

Personally, I don't like the external links to the wiki. The howtos, tips & tricks, etc. are still on the simon wiki but are linked from the main homepage. This of course is a bit confusing when you click a link in the sidebar and suddenly end up on a completely different homepage. However, we don't want to miss the advantages of having the (ever changing) content on our wiki as it is much easier to update.

So dear lazyweb: Is there a convenient solution (like a mediawiki plugin) for embedding wiki content in a typo3 page?

Ideas and of course feedback for the homepage would be much appreciated!

Microblogging

2009-05-14T05:17:00.000-07:00

Yes I finally gave in to all that peer pressure :)

You can now follow me on identi.ca/bedahr.

Linuxtage 09

2009-04-27T04:45:00.000-07:00

On Saturday we gave a presentation at this years Linuxtage in Graz.

In short: It was awesome :)

We had the largest of the three lecture halls and it was nearly full which means we had about 30 - 40 people listening. The presentation of the software worked out perfectly which just goes to show that the current version is very usable already.

Sadly we could not get into all the discussion points presented during the presentation (after all we had just 45 minutes) so we continued the discussion in the cafeteria afterwards.

All in all, it was a great way to get people interested in the project and it was a pleasure to meet and talk to other free software enthusiasts.

On a related note: One of the people I talked to at the conference, Sven Guckes, pointed me to an Austrian phonetic database called Adaba. This database also comes with a lot of sound samples spoken from professional speakers so this could even be a starting point for a speaker independant german acoustic model. However, the licence is commercial so we have to see what we can do...

Correction (11.05.2009)
Seems like I mixed up two conversations I had. Astrid Gruber was the one that pointed me to Adabe. Thank you very much!

A little fun with the AI Plugin

2009-04-24T02:14:00.000-07:00

I just set up a little conversation model which allows for a handful of simple sentences to be recognized. I then loaded the AI Plugin, turned up the speakers and had a little conversation with my notebook :)

This is how it went:

Me: Hello
Computer: Hi there!
Me: What is your name
Computer: My name is alice.
Me: My name is Peter
Computer: Your name is Peter?
Me: Yes
Computer: That's cool.
Me: Yes
Computer: Pardon me?
Me: Yes
Computer: Ah.
Me: What are you
Computer: I am alice. The female chat robot-electronic brain...
Me: Cool
Computer: Who is the best robot?
Me: you are the best
Computer: Thank you very much human...
Me: you are welcome
Computer: The pleasure was all mine.
Me: Bye
Computer: Thanks for chatting, Peter!

While certainly not very helpful - this is pretty cool :)

A little Story about Accidents, Releases and Conferences

2009-04-21T00:17:00.000-07:00

Ok the update schedule on this blog is getting a bit ridiculous :)

However, I have an excuse this time: About a month and a half a go I had a pretty bad accident (no cars or other persons involved). I broke both legs and am currently sitting in a wheelchair. As a result, the simon development has been a bit lower on my todo list(s) than usual. I put some explanations up on the sourceforge page and on the german ubuntu forum (actually while still in the ICU) but forgot the blog somehow. For those of you that though that the project might have been halted: I am sorry.

I am expected to learn to walk and subsequently begin to work full time again in July.

On the plus side: We still managed to release the first release candidate of simon!
It is pretty much just a packaged version of what we had already finished before my accident but it still is pretty stable and defenitely a lot better than beta 3.
There are still two issues (which are pretty bad: one crash and one bug that just stops the recognition without any visual indication) but both are related to julius and are hopefully fixed soon. Anyways, they are non-blockers - the problems only occur during restarting the recognition which doesn't happen once you set up your model.

For our readers from Austria: We are presenting the current prototype at the Grazer Linuxtage 09 in Graz. If you live near Graz: Come by and take a look - admission is free!

Practical Tests

2009-03-02T00:46:00.000-08:00

While this blog is defenitely not updated nearly enough, things have still progressed at an amazing pace. The software is maturing quickly and altough the windows version still crashes from time to time, those crashes seem to be completely random and mostly happen when opening menus and such so I get to blame KDE or Qt for that ;). Anything else seems alright and works really well in practical tests.

Speaking of practical tests: The current prototype has been tested with two teenagers suffering spastic paralysis. One of them had only a mild speech impairment and after just one hour of training reached already a recognition rate of astonishing 97 % percent on a lexicon with 26 words. Those 26 words are enough to control MediaPortal, Firefox, Skype as well as sending pre-defined Text snippets as e-Mails.

The other test person had a severe speech impairment which meant that even as a native speaker I couldn't understand him at first myself. However, with just a few trainings sessions we already reached impressive 73 % recognition rate on the same 26 words.

This proves that simon really is usable for people who could never use commercial offerings because of their speech impairment.

Tomorrow we have an important presentation to doctors and therapists to demonstrate simons usefulness for handicapped people and it's opportunities when used as a therapeutic tool.

Wish us luck!

Documentation Update

2009-02-04T07:40:00.000-08:00

Yes! I did it! I managed to devote a whole day to documentation - getting to know docbook, writing installation instructions on the wiki and generally updating and adding documentation all around.

But I am not done yet: We actually have plans for a real manual. With pages and content and everything :)

On a less cheery note: Someone reported problems with our binary packeges. Both the debian package and the generic binary package apparently have some issues. I have installed a clean Ubuntu 8.10 in a virtual machine and am going to double-check everything tomorrow.

simon in action

2009-01-28T07:45:00.000-08:00

As promised, I created a little teaser video that shows what you can do with simon 0.2 beta 3 and a little training.

The video is available for download here.

Btw, I use the mousless browsing addon to controll firefox.

simon 0.2 beta 3 released

2009-01-26T08:03:00.000-08:00

Finally. Unlike the second beta which wasn't really that exciting this is new release is IMHO again a huge leap forward.

Why? Well three new command plugins and one old one that actually works for the first time in this release? In my book that effectively doubles our plugin-count.

Composites
Composites take other simon commands as their arguments and execute them in order. Moreover you can introduce delays between them. A typical use-case for such a command would be:
Open a Chat with Anna.
This takes three actions: Open Kopete, type "Anna" to find her in the contact list and press return.
Those three actions have already been possible with old versions of simon but they had to be executed by the user. With the new composite plugin you can create a command that launches the executable command "kopete", the text-macro command to type "Anna" and the shortcut command to press return and associate a meaningful trigger to the combination.

Lists
List commands also take other commands as their parameters. However, instead of executing them one by one, lists - when triggered - provide the user with a small dialog pop-up that presents the commands that make up the list and let the user choose the command they want with numbers from 1-9. Of course lists can contain other lists so you can easily generate sub-menus if you want to. Composites in lists are no problem either.
This enables simon uses to effectively limit their vocabulary much more if they don't want to invest much time into training or just can't speak very well.
An example of a combination of those lists and composites would be a "Contacts" list-command which contains place commands to e-Mail addresses (mailto:bla@blub.com works great as place-command) and a composite plug-in to start a chat with Anna (see the composite part).

Number Input
While not as exciting as lists and composites this plug-in was important too: It let's you input complex numbers easily by providing a calculator-like interface.

Desktop Grid
The desktop grid was even part of the 0.1 series but it was never controllable by voice and just included as a teaser for... well this version :)
Using the desktop grid the user can click any point on a desktop by saying a combination of numbers. When the desktop grid is activated, it devides the desktop in nine areas. Saying a number of 1-9 will then divide this region again in 9 parts, etc.
That way the user can precicly click any point on the screen - again only using numbers from 1-9.

Of course there have been other advancements too. For example the recording widget now displays the current "loudness" using the progress bar while recording and a lot of bugs have been fixed.

All in all 0.2 beta 3 is a really cool release and, if all goes well, might just be our last beta release...

UPDATE 27.01.2009
I seem to have forgotten one final commit which, sadly, fixed some critical bugs (ksimond crash on startup and some plugin-loading stuff). I committed the patch now and am rebuilding the packages as I write.
Anyone who downloaded the release yesterday will have to download and install again. Don't forget to uninstall any previously installed 0.2 versions first!
Sorry for the inconvenience!

A Tale of Poor Recognition Rates

2009-01-23T00:44:00.000-08:00

Some of our german speaking users who tried the 0.2 series of simon might have noticed that the recognition rates are extremely poor. No matter how many trainings-samples, simon doesn't recognize a word.

While I never had that problem myself, I saw it myself on Mathias' notebook. Interestingly, using the "real" julius on the same model worked very well. The problem had to be in simon somewhere. And so the digging began...

The first thing I did was to compare julius log produced by simond and the one generated by the "real" julius. The only difference between them was the comma seperator: Julius used "." and simond used "," (which is correct as this was a german windows xp). Well that can't be it, can it?

After a ~~bit~~ lot of fiddling, I gave up and changed the locale to English/USA. And just like that things worked fine.

It turns out, that Julius respects the locales decimal point even when parsing the hmm model files. And as the HTK uses "." as it's seperator and julius expects "," the model is parsed incorrectly. That never happened to me, as my system locale is en_US.

So if you use any version of simon 0.2 with a system locale that uses any other comma seperator than ".", you will have mediocre recognition rates unless you open up the hmmdefs file (Windows: %appdata%\.kde\share\apps\simond\models\\active\hmmdefs; KDE: ~/.kde/share/apps/simond/models//active/hmmdefs) and replace "." with your locales decimal point or change your locale to English.

Dutch translation

2009-01-15T08:56:00.000-08:00

Quick update: Thanks to Daniël Heres, the third beta of simon will also be available in Dutch!

Second Beta

2009-01-15T08:35:00.000-08:00

Well on monday I pushed out the second beta release of 0.2.

I really didn't want to make a lot of fuss about it which is also why I didn't put announcements on all the "cool" places (like kde-announce). The second beta is only a bug-fix release (Changelog) and as such does not introduce any new features.

To be honest, the main reason for releasing the second beta was that we got a couple of new testers on monday and I wanted them to test the latest version of simon. As there was no reason to keep the compiled binaries to myself I put them on sourceforge.

Thanks to everyone who reported bugs since the first beta! Keep 'em coming...

Happy New Year

2009-01-02T14:20:00.000-08:00

Well I guess I am a bit late but still ... Happy new year to everyone!

As some of you might know, I stayed with an old friend over new years so my responses to forum posts, opened bugs etc. might have been a bit slower.

But I am back and - as a sign of good will - already fixed some of the open bugs on the way back home (at least the hours on the train were useful for something). The fixes are not yet available through SVN but I will commit them tomorrow after some testing.

For now I just want to thank all the users and supporters of simon who helped to make simon what it is today.

Looking forward to another great year!

ps.: Besides some bug-fixing I started to work on new command plugins that might *cough* pop up in the near future - stay tuned...

simon 0.2-beta-1 released

2008-12-27T11:44:00.000-08:00

Ok what better way to start a blog about the development of a software product than it's first beta release?

So without much further ado, you can download the version from sourceforge.

This is pretty much a feature complete 0.2 release (only some command-stuff is missing which I have planned for the next beta) and at least in my tests it has proven to be quite stable.

This is also - as pointed out earlier - the first beta release of simon ever (0.1 only got to alpha 2.1 before being discontinued) so this is kind of a big step forward. However, that does not mean that simon is ready for productive use yet. It needs a lot more testing - which is something you could help us with! Download a version for your plattform from the download page and try it out!

Along with pre-built debian packages and of course a complete source package (which contains an updated build-script) this also marks the first time we release a binary installer for Windows (yes a real installer built with NSIS) - and it contains all the needed dependencies too (except HTK because of their licence).

Please report any bugs or feature requests on our bug tracker.