LinuxMCE Forums

General => Feature requests & roadmap => Topic started by: DragonK on January 13, 2012, 12:41:48 pm

Title: Virtual 3D AI
Post by: DragonK on January 13, 2012, 12:41:48 pm
Hi Guys,

This is not a feature request, but more like a wishfull thinking. But just had to share a thought.... If we had something similar for LMCE.....

Part 1 : http://www.youtube.com/watch?v=7W52TL9Akv4  
Part 2 : http://www.youtube.com/watch?v=ZmlrkQBjVok
Part 3 : http://www.youtube.com/watch?v=FxK-pOFFx1o&NR=1&feature=endscreen


Karel

PS. Her name would be Sarah and not Denise....  ;D
Title: Re: Virtual 3D AI
Post by: Marie.O on January 19, 2012, 12:37:23 pm
We are expecting a proof of concept by you by April 23rd, 2012.
Title: Re: Virtual 3D AI
Post by: _if_ on January 19, 2012, 05:45:53 pm
wow that would be killer  :o
propably there is nothing similar as opensource around, right?
Title: Re: Virtual 3D AI
Post by: tschak909 on January 19, 2012, 06:07:27 pm
Do you guys truly not understand, that if you don't write it, it doesn't get done, and therefore does not exist?

This is an open source project. Do it.

-Thom
Title: Re: Virtual 3D AI
Post by: DragonK on January 20, 2012, 06:33:56 am
Thom,

Where would the best place be to try and start. What would you recommend?
I would need ALOT of guidance.

Karel
Title: Re: Virtual 3D AI
Post by: grind on March 07, 2012, 03:00:49 pm
I also like that very much! And the best would be if you could customize the face.

Btw. yours would be Sarah? Maybe because of Eureka? ;)

Nicolai
Title: Re: Virtual 3D AI
Post by: WhateverFits on March 07, 2012, 11:09:25 pm
The face is customizable and it is $100 to $850 per license. It has an API but the biggest integration part comes with the expensive license.
Title: Re: Virtual 3D AI
Post by: Sigg3.net on June 16, 2012, 01:10:57 pm
Just searched for open source 3D talking head and found Xface. The project died in 2008, but they reached version 1.
http://xface.fbk.eu/index.htm

There are probably heaps of other "3D butler" projects out there if you look for it:P

Problems are not tying it into LMCE, but how it would play with other media playing on the instance..
Title: Re: Virtual 3D AI
Post by: JaseP on June 17, 2012, 03:26:40 am
Blender and Makehuman to make your 3D mesh, then just pre-render video that delivers a video scene for every use case,... Ought to take no more than 50 to 60 GB of hard drive space to store all the animations... And maybe 9 months working 16 hour days to do all the animation.

Sorry for the sarcasm... But, I'm just not all that impressed by this kind of tech. Now, a neural network AI (minus the Final Fantasy-esque digital sex doll) to interface with LinuxMCE,... complete with visual and speech recognition, with predictive capabilities, and maybe a personality,... That would impress me. But we'll have to wait another 15-20 years for that kind of thing.
Title: Re: Virtual 3D AI
Post by: Armor Gnome on September 12, 2012, 10:11:23 am
I have actually considered doing a similiar idea in my own set up.  My ideas though are much more simple and use looped or very long video files with previously recorded text-to-speech statements.  It would be ugly but just nerdy enough for me to love. The other idea besides a human persona was a robotic face or HAL glowing orb on-screen as the audio is played.

My roadblock to this was time and experience to develop it, the slight load time for xine after receiving a play command and most critical, I couldn't play a "blank" video file and an audio file at the same time with the amount of time I put into this.


Implementation idea if anyone is interested in putting time into this for themselves:
Code: [Select]
Event = press a category button on an orbiter
Response = loop interface video on local on-screen orbiter, play audio file "Here is a list of your videos.  Which would you like me to retrieve for you?", stop looped video

Event = selecting a video file from data grid and then pressing play
Response = loop interface video on local on-screen orbiter, play audio file "Now loading [filename] please wait while I prepare it for you", stop looped video,  play selected video file.

Note that this would not function with UI2 as the selections are not separate screens, I am sure with enough effort it could be incorporated but for the short-term a way to play video and audio at the same time would be a huge step for this.
Title: Re: Virtual 3D AI
Post by: _if_ on September 12, 2012, 06:31:10 pm
I've started learning blender some time ago to be able to put together a 3D head for that purpose. It turns out blender being a damn complex program, so I'm still doing babysteps and am not even close to modeling a head but I continue trying.

The plan for the begin was like this:
first learn blender
build a 3D head
use rigging to make the lips move (blender can record my actual lips with a webcam and sync the models lips - once I find out how that works...)
sync to already recorded audio
render animations.
start replacing Sarah in the first step of AVWizard and see if that makes sence
If I ever get to this point, we can think about integrating it into the whole wizard and popup screens like the "you have added a new device..."

I know this is a very hard task and maybe I will give up some time, but at least I'm trying.
What are your thoughts....doable?


IF

Title: Re: Virtual 3D AI
Post by: JaseP on September 12, 2012, 08:23:28 pm
Try MakeHuman;

http://www.makehuman.org/

It creates the mesh, etc. for the whole body, but you can export to Blender to do the animation, just focusing on the head. It also creates controls for animating the face (and body), but those can be challenging to use (like operating a marionette). The hair typically does not export (you probably would need to do the hair yourself, or find a mesh you can import).

Title: Re: Virtual 3D AI
Post by: _if_ on September 12, 2012, 09:54:50 pm
cool thanks, I will look into it!
Title: Re: Virtual 3D AI
Post by: Armor Gnome on September 12, 2012, 11:16:44 pm
Sounds good _if_ ,

I hadn't thought about the Sarah wizard since its already pulling from pre-rendered video and audio files.  Let me know how you are coming on this.  It is not something I am still actively working on but it is a cool idea.
Title: Re: Virtual 3D AI
Post by: _if_ on January 06, 2013, 02:31:47 pm
Hi,

just wanted to give a little update how far I came until today.

http://youtu.be/uxaLhACVKV4 (http://youtu.be/uxaLhACVKV4)

the whole model may totally change and the lipsync will be improved, as well as the hair (I know you can still see the short hair).
I'm learing blender more and more and will keep improving the whole model when I find time.

Use your imagination and let me know what you guys think.

greetz
IF
Title: Re: Virtual 3D AI
Post by: ardirtbiker on January 06, 2013, 03:23:05 pm
_if_,
  Thats kinda spooky <grin>

Dennis
Title: Re: Virtual 3D AI
Post by: _if_ on January 14, 2013, 10:13:03 pm
@ardirtbiker: don't worry, it just takes time to find out how to make it more human and less zombie ;-)
Title: Re: Virtual 3D AI
Post by: ardirtbiker on January 15, 2013, 12:25:57 am
hopefully... dont wanna get people scared!
Title: Re: Virtual 3D AI
Post by: tschak909 on January 15, 2013, 12:35:12 am
I don't fucking care what it is, JUST BUILD IT!

-Very Frustrated and Distressed Developer Thom

Title: Re: Virtual 3D AI
Post by: Sigg3.net on January 17, 2013, 01:40:06 am
Use your imagination and let me know what you guys think.
In my imagination she's too naked :D

Great work!
But is it interface-able (like Xface) so we won't be back at the point of origin, viz. pre-recordings. It would be nice to have "it" _read_ a text-file, using Festival or something for voice, and learn it some basic mimicry based on the input.
Title: Re: Virtual 3D AI
Post by: tschak909 on January 17, 2013, 02:08:32 am
Make something. Make a prototype. Don't talk it to death. MAKE THE PROTOTYPE, AND TALK ABOUT THE F****** PROTOTYPE! JESUS PEOPLE, HOW MANY TIMES DO I HAVE TO REITERATE THIS?!

-Thom
Title: Re: Virtual 3D AI
Post by: _if_ on January 17, 2013, 05:47:18 pm
@Sigg3: I had plans for that before, but it turned out to be much much more complex, and if you look at the lipsync in the videos the original poster posted - that is not even close to any real speaking even if it's done by a professional company.
Once the model is done it's not that much of effort to add new animations because all you need is new spoken text from the TTS and a camshot from my lips moving for tracking. Also, we don't have that much cases which need that much of flexibility I think. For example the Install Wizard, that is not a thing which will change every day...

@thom: I DO work on it and still I'm interested in ideas, so it's alright if you don't care but why don't you just ignore this threat instead of spaming it all the time?

thanks
IF
Title: Re: Virtual 3D AI
Post by: tschak909 on January 17, 2013, 05:50:27 pm
Boy, understanding subtlety isn't one of your strong points, is it? :)

Ok, I will be plain. I am interested!

You're working on it! COOL! OK! UPLOAD SOMETHING THAT WE CAN FIDDLE WITH! EVEN IF IT IS NOT COMPLETE! Otherwise, it does not exist, there is no point in talking about flavored air, and therefore, you are full of shit. Don't want to be? upload something. :)

-Thom
Title: Re: Virtual 3D AI
Post by: _if_ on January 17, 2013, 05:58:15 pm
I did upload my first try a few posts ago but I know that it is a very early stage. As soon as I have more to show I will upload it...
Title: Re: Virtual 3D AI
Post by: Sigg3.net on January 17, 2013, 10:03:39 pm
I guess tschak is just getting excited :D

Yeah, I checked the videos last night. The lipsync does NOT have to be "correct", only moving when there IS speaking. Add to that some prerecorded gestures (laugh, smile, whoops aso) and you have an API. Continue the good work!
Title: Re: Virtual 3D AI
Post by: _if_ on January 22, 2013, 05:37:45 pm
made a little bit of progress: http://www.youtube.com/watch?v=sWywOmytW4s&feature=youtu.be (http://www.youtube.com/watch?v=sWywOmytW4s&feature=youtu.be)
Title: Re: Virtual 3D AI
Post by: hari on February 09, 2013, 10:41:35 am
can you make her look like cylon six?
Title: Re: Virtual 3D AI
Post by: _if_ on February 09, 2013, 11:33:39 am
in some years when my blenderskills have improved that may be possible...but don't forget: you can't marry a 3D model!

btw: little progress http://www.youtube.com/watch?v=bEVmxuJGlU0&feature=youtu.be (http://www.youtube.com/watch?v=bEVmxuJGlU0&feature=youtu.be)
Title: Re: Virtual 3D AI
Post by: phenigma on February 10, 2013, 10:16:18 pm
Looking good!  Very neat stuff!

J.
Title: Re: Virtual 3D AI
Post by: _if_ on August 30, 2013, 05:52:37 pm
as it's been a long time since the last update, I just wanted to show progress (I did not have much time so its not much).
I played with creating hair and found out that this is crazy hard to do...anyway: http://youtu.be/DUQy14fcHbQ (http://youtu.be/DUQy14fcHbQ)

greetz
IF
Title: Re: Virtual 3D AI
Post by: ardirtbiker on August 31, 2013, 01:21:04 am
looking good!

Dennis


Im getting excited now!!!!