Voice control suggestions

[Hugolp]

Hi

I am going to start working on a voice control system based on speech recognition for linuxmce. I have a basic and clear idea of how I want the system to behave but I am open to suggestions. I am going to explain my idea so you can critice or add to it. Any idea is welcome. Also, if anyone wants to collaborate is welcome as well.

Basically the system will have two ways to interact. One through the menu and the other by direct order.

Menu: Basically you say menu and the menu displays just like if you were using a mouse. Then you can say another option from the menu and that submenu will open, and so on until an action from a menu is chosen. You can go back by saying back.

Direct order: Basically a list of orders will be stored with the sentence that activates it and the action or actions to be done. It should be totally configurable by the user, letting you change any order. Also, it should work with templates, being able to export your actual configuration, download "popular" order templates from the internet, and so on. Once in your system the template should be totally reconfigurable.

There are two issues here. There should be room to let the system "understand" numbers and maybe for the future understand dates and hours. Something like "Turn off the tv in %time" where %time would be a number of minutes or hours.

Also the system should recognize and alert to the user for incompatible orders. For example, you configure the sentence "switch on" to turn on the lights and you configure the sentence "switch on tv" to turn on the tv. That can cause problems because when you say "switch on tv" the system could understand that you said "switch on" and turn on the lights. I dont know if it will be for the first version, but I want the system to alert the user for this.

The area where I have more doubts on how the system should behave is where you have to select a movie or a song from a large list. I am waiting to use more the system to decide how is the best way to do it.

Ok. So basically this is it. If anyone has any idea on how to improve this model or has any experience with a voice control system and wants to give his/her opinion is welcome.

Hugo

[darrenmason]

Certainly very interesting and something that I have thought about myself.

Have you worked with voice recognition technologies before? From what I have read a lot of complexities come from picking up voice reliably and what microphones to use. This could be overcome somewhat by incorprating the telephony system, so that you dialed a certain extension to issue a command. Not as simple to use, but probably more reliable plus the bonus that you could conceivably control it from anywhere you can find a phone.

I think activating a scenario (assuming that you have the recognition part working) should be quite straightforware. You are essentially doing a messageSend. This could be done by creating a "voiceOrbiter" device or something.

Controlling the menus might be better approached by simulating a user interface device. The Orbiter can be controlled by keyboard, mouse, infrared remote etc. so I imagine this should be quite doable. Potentially you could even write a LIRC driver that works on voice???

Configuration should be a ble to be added to the standard configuration screens I guess.

Let us know how you get on. You might want to take a look at MisterHouse as I believe that has voice control already.

Regards
Darren

[thynk]

I can confirm that MH does support voice control - tho I never did it working in linux, setting it up in windows is fairly trivial.  I've never done any media control with MH, it's more to control the lights and watch the security system than to play movies and such.

One suggestion I can make - be sure to put a keyword control in to your system - something you use to put to sleep the voice control and to wake it up again unless you want it doing the things it thinks it hears off the tv or movie your watching :-)

Stu

[Hugolp]

I can confirm that MH does support voice control - tho I never did it working in linux, setting it up in windows is fairly trivial.  I've never done any media control with MH, it's more to control the lights and watch the security system than to play movies and such.

One suggestion I can make - be sure to put a keyword control in to your system - something you use to put to sleep the voice control and to wake it up again unless you want it doing the things it thinks it hears off the tv or movie your watching :-)

Stu

I was thinking on giving the computer a name, so to give an order you have to call the name before. Something like "Franky, turn on the lights" and the lights will turn on. But if you say "turn on the lights" nothing will happen. As I said, until I get the system working I wont be sure what will be the best solution, thats why I open this thread, so you all can give your ideas and opinions.

Hugo

[Hugolp]

the systems of voice recognition at the present time still are not very good and usually fail with facility, since all the people do not have The same tone of voice nor pronounce equal, the way more simple it is using numbers since they defer in his pronunciacion enough.

try this as an example
http://www.winsite.com/bin/Info?500000019196

Thanks for the advise. But that program doesnt work on my operating system, and also seems to use his very own speech recognition software, wich may bee primitive. I plan to use a good speech recognition software with more than 97% of acuracy for an untrained system. There are alredy comercial companies using it.

Hugo

[Hugolp]

Thanks for the links. I knew them all. Last version of Dragon Naturally speaking doesnt need to be trained for each user anymore since last release (I thinks is the 9). Thanks for the links again.

Hugo

[MasterC]

I like this idea and have also been entertaining it for a while.  One of the problems that I thought of as well was random discussions.  By calling the machine by name you would reduce that significantly, but I can still see the situation where a movie would call out "Eduardo, halt!" and your machine would shutdown.  I really like the idea of implementing telephony, and to take that a bit further, implement it via Bluetooth.  By using Bluetooth to connect to the system I could similarly "talk" to the system through a Bluetooth enabled phone.  The problem I can't seem to get over is ambient noise.  The telephony, and by extension the Bluetooth would somewhat take care of it, but any other Mic type voice control would probably require some loud shouting to ensure the system could hear your commands clearly enough to execute them.  Alternatively you could place X10 or wireless Mic sets (even wired I suppose) sporadically throughout your house (or at least the rooms in which you may want to control a PC) to help out, but this might make things actually worse.

One other "problem" is response time.  Obviously as the system matures it will be faster, but if it's not minimally fast enough to respond to voice, there won't be enough adoption to entertain continued development.  Once you speak, the expectation is that the computer will respond, but first it has to translate your voice to the command, wait X amount of seconds to determine if the command is finished, and then execute it. 

Overcoming those 2 (well maybe that's 3 ) moguls will probably be difficult but I think will help with wide adoption of voice control (not just with LinuxMCE).

Do you have any ideas on which API you intend on using, or any basic development (research maybe?) in place that an average joe like myself could click on over to take a look at?  I'd be happy to offer my assistance in any capacity I can if the project takes off at all.

-Chad

[Hugolp]

MasterC thanks for your comments. You are right in a lot you say. First of all, voice speech recognition has gone a long way, and now is able to be speaker independent with good rates, but stills needs a good recording to begin with, so placing mics all over the room its a no way. To use voice recognition a headset is needed (you could use a good mic as well, but I am sure people wont go carrying a mic arround the house). So a bluetooth headset, I think, is the way to go. Also, as linuxmce uses bluetooth to detect where the person is, carrying a bluetooth headset would allow the system to know where you are at any moment and act accordinly. I think it can be a powerful combination.

About the time the system would take to respon, there are options to overcome that, but I wont take a decision until I see the system acting. And be sure the project is going to take off. If you want to help in any way, send an email to hugo.calleja@gmail.com and when I have some testing to (it will take a while) Ill contact you.

Hugo

[bulek]

Hi,

I'm speech recognition researcher and gave a lot of brainstorms on that field and heard a lot of feedbakc info from users. I'll try to add few my opinions :

1. I think that users generaly don't like to have house filled with mics. Also there is a problem, when system shouldn't react to same words.

2. The basic approach IMHO should be that user can decide when to talk and when not to talk to house. Curently house phones and whaever smart device that can be used for that is appropriate. Maybe speech recognition is currently more suitable for remote access (call and tell the house what you need).

3. In current architecture, I see one obvious and convenient way to use speech recognition. Currently we have Asterisk as VOIP PBX, and we can talk to Asterisk in all different kinds (Mobile, ISDN, Orbiter phone ,etc...). There is also majority of other features implemented with the help of Asterisk and its features (talk to the house (it basically mean that conference call is made with caller and all Orbiter phones)).

4. So from 3, connecting speech recognition engine to Asterisk would IMHO be the most efficient way to start digging in this area. Asterisk 1.4 already supports one of commercial SR Engines (Lumenvox), maybe we can connect something from mentioned projects too. But we need Asterisk upgrade and more transparet setup (currently Pluto-Asterisk setup looks more like spaghetti than well organized code).

HTH,

regards,

Bulek.

[robertsaron]

I wont go to home automation until it is all voice recognition. Imagine this scenario: I come home from work, computer play track 3, in folder media, music selection enya. Then later when I have my food and beer, computer stop music, tv on, load movie blade 3.

No remotes, lets say I need to use the bathroom from eating dinner in the middle of the movie, computer pause movie, come back from using bathroom, computer lights off unpause movie.

No remotes needed. You would not need mics all over the room, or all over the house. One shotgun mic or a decent mic, in select rooms would be enuf, and if it is done right, most people would never even know the mics are there.

[Hugolp]

robertsaron placing mics all over the house can be very tricky to make it work and probably unreliable. Would you have any trouble having a bluetooth mic in you. Not a headset if you dont want to, but a little bluetooth microphone, something like what the tv presenters have (but without the big packet in the back). Just a little clip, so you can give orders to the house. Would that seem like a good solution to you? Also, that would allow the system to know in wich room you are and turn on the tv, lights, music, etc of the room you are at and not another one.

Hugo

[MasterC]

Hugo,

Love the idea of the Bluetooth headset.  I think it's definitely a step in the right direction.

-Chad

[Niels]

I like the idea off remote voice control to and i was already thinking of creationg something like the Star Trek communicators badge. This badge needs to by used for both communicating to other people by calling his/her name, or commanding my computer. One of the possiblities i was already thinking off was a Bluetooth headset for the mobil phone, but i'm thinking of other kind off radiocommunication too.
The idea's from Bulek are very nice, and in my opinion the way to go for my idea and also for LinuxMce i think.

What i understand from you is that you like to have a Bluetooth responder (is this the right name?) in each room, but how do's this work? Becose when i need to communicate between my pc and my headphone i need to register the device first. Do we need to this on all the respondersystems?

[Hugolp]

The idea's from Bulek are very nice, and in my opinion the way to go for my idea and also for LinuxMce i think.

What i understand from you is that you like to have a Bluetooth responder (is this the right name?) in each room, but how do's this work?

Pluto alredy has implemented a system that detects wich room the user is in by detecting the bluetooth signal, and changes the media to that room in response. So the bluetooth detection system is alredy done. I want to use that to get a personal bluetooth headset or just a bluetooth clip-in microphone to send the audio signal of the user voice to the server so the speech recognition system can do his job and send (or not send) an order to the system.

Once my idea is done, either Bulek or your idea shouldnt be hard to implement.

Hugo

[Compaq]

there is a similar project here
http://rubyhacker.com/domo.html