UI3 Discussion

[tschak909]

Yes.. this has been discussed a bit on the list, but I paraphrase here as well... we all know and understand that this is for UI3..and some affordance needs to be investigated for the old orbiter UI, either in the form of a replacement Orbiter that runs on the small devices, or a separation of dev tools so that traditional orbiter designs can be manipulated, alongside the Plasma layouts in UI3.

-Thom

[Matthew]

On the developer tools, I've talked with Aaron B (Pluto) and Aaron Seigo (TrollTech KDE) and the plan is very much to have a high end design tool to make theming easier.

As for the underlying toolkit, this would probably be based on libplasma. Plasma is a KDE library initially written to support the next generation desktop for KDE, but which has since been recognized as a portable library with a good design for writing sexy widgets. Depending on the software design chosen it may even be possible to place KDE desktop widgets in the new UI and have those widgets be aware of the 10' UI layout and font size restrictions. Theming would include not just be adjusting the look but also allow control over how the UI elements behave and even allow adding or subtracting applications within that layout without any need for a programmer to be involved.

That strategy would seem to make LMCE dependent on KDE in a way that is almost negligible now. Does it really make the most sense to make MDs dependent on an entire desktop when it's supposed to be a vastly simpler paradigm than a desktop, and supposed to run on little devices, including mobile phones? And again, why reinvent the wheel with new high end design tools when there are already so many, and so many people using them, to create SWF?

[tschak909]

Plasma is desktop environment independent.

-Thom

[Ender]

Hi everybody,

What about using Gnash? http://www.gnu.org/software/gnash/ Seems to have almost everything we need.

[totallymaxed]

Hi everybody,

What about using Gnash? http://www.gnu.org/software/gnash/ Seems to have almost everything we need.

GNASH has already been suggested earlier in the thread ;-)

It looks good... but my take was it might be a little early in its dev cycle... but on the other hand maybe not

[Matthew]

Plasma is desktop environment independent.

OK, so compare Plasma to, say, GNASH (which is not the only alternative, just the first one that I found which seems right).

GNASH is already just about to release, includes evidently all the features UI3 requires, comes with its own developer community, and uses SWF format. SWF of course has a huge content developer community as well as mature tools they use, open source alternative tools, is a default format for practically every desktop and browser in the world, and lots of existing content. SWF/GNASH also already plays video in FLV, which is what YouTube and most Internet video sites like it use. Adobe's pushing FLV (as is Google with YouTube), so it will be a primary "Internet TV" format for years, which will mean LMCE will have to support it somehow anyway. And GNASH uses ffmpeg for that video, which is already part of LMCE (though LMCE would probably need to wrap ffmpeg in a new UI module to integrate it fully), showing that LMCE already has consistency with GNASH, and common code reduces LMCE's (considerable) bloat. GNASH also runs on embedded/small devices, like really lightweight MDs will be, and mobile phones. The same source code generates either SWF or DHTML.

Plasma has some (new) libraries. It's unsuited to anything but a standalone app on the local machine (ie. no Web browser). There's no design tools, content management, developers, existing content, established techniques. It will take a lot more work to get a UI3 in it than it would applying that work to GNASH.

Maybe this is a false choice. Maybe the way to get the best UI3 the fastest with the best extra perks (community, tools, content) would be to spend the UI3 development time adding Plasma features to GNASH, where GNASH needs it. That would be the best for everyone, including the broader GNASH community (which would help them become contributors, directly or indirectly, to LMCE).

FWIW, substitute any equivalent to GNASH for GNASH in this discussion. As I said, GNASH is just the first alternative that I found that looked better than the Plasma route. There are likely others, though we have to find only one that works.

[aaron.b]

I'll divide my comments in 3 parts: 1) the design tool, 2) the player, 3) financial issues.  Both the design tool and the player are *almost* what we need, but, lacking in a couple areas.

--Design tool--

I really would like to find a way to somehow base this on Flash because it will be impossible to reinvent Flash's design suite.  I have never used it, but just from looking at the demo video on Adobe.com, it is clearly a very rich, robust, polished design suite that allows the designer to create gorgeous UI's that do almost anything with sophisticated illustration and 3d tools.  The thought of us saying "Let's rebuild something that's as good as Adobe's flash designer" is daunting; that software was 10 years in the making with hundreds of full time developers.  We just can't make something as elegant as it.  And we'll never be able to reproduce having tens of thousands of developers who know it intimately so they can create cool eye candy effects in our UI.

The one area that Adobe's design tools may be lacking in (I don't know for sure) is the cross-platform stuff.  Pluto's HADesigner is crude as can be, but, it was designed to target lots of platforms.  For example, the 'play' button in UI1 on windows ce pda, and the 'play' button in UI2 on the TV, as well as on the Cisco phone and mobile phone, are actually all the same object.  You just assign different attributes for each of the variations (like object oriented inheritance), so you can make it be animated and round on a more powerful box, and a small static square on a mobile phone.  You can also cluster the 'play/pause/stop' buttons into a 'media playback group', and say that the group itself is also different on a tv vs. a mobile phone (ie object containers).  Then, when you're creating a media remote control, you just drop the 'media playback group' on the screen and whether you're using a mobile phone or a tv it still works.  I'm not sure if Adobe's flash designer is object oriented like that, supporting inheritance and overrides, so you can create, say, a button that looks and behaves differently on various platforms.  I've never seen Flash used like this; the general goal of flash is the opposite: to have a player that renders the flash file exactly the same on all platforms, rather than having one UI that behaves differently on the different platforms.

While this may be a weakness, it's not necessarily a deal breaker since, after all, if Flash designer is not object oriented with inheritence, the person designing the UI could work around this by just creating separate flash files for the mobile phone, the pda, the cisco phone, the TV, etc.  Most designers would probably prefer to create 5 UI's using Adobe's flash designer vs. 1 UI with Pluto's HADesigner because our HADesigner is that bad and it's unlikely we'd have the manpower to create something that rivaled Adobe's flash designer.

The other thing I'm not sure of is how well flash scales.  In general all the flash files I've seen are created for a target resolution (say 800x600).  With Pluto's HADesigner, OrbiterGen scales the UI to any resolution, rotates it (ie some devices are vertical not horizontal), and stretches it (16:9 vs. 3:4).  This would be a big deal breaker if Flash couldn't do this because, although we could tell designers they need to create 5 different UI's for all the target platforms, telling them to create each of those 50 times (5 UI's at 10 different resolutions) would be ridiculous.  If they wanted to change the command the 'play' button sent, rather than doing it in 1 place like you do with HADesigner, you'd have to make the same change to 50 UI's if Flash didn't support scaling/rotating/stretching and didn't support object oriented inheritance with overrides.

--Player--

Here what's basically lacking is the ability to blend 2 different applications together.  Flash is designed to work stand alone, not bolt on top of and blend into something else.  We're looking for a UI that can bolt on top of MythTV, or Xine, or MPlayer, or Firefox (for a 10' web browser), etc.  If, however, we commit to using Adobe's flash designer as a back end, then that means we've committed to the flash file format and we're better off to put resources into adding the alpha blending capabilty to something that already plays back flash files, like Gnash, than to start from scratch reinventing a flash player.

--Financial Model--

Here is where it gets tricky.  Pluto has investors, and when we put in the budget expenses for development, we have to show that there's some way that we can at least make enough revenue to cover the expenses, and ideally make a profit, otherwise it's considered throwing money away.

Normally the way companies do this with FOSS is to offer software under the GPL that, in order to be used commercially, needs to link with non-GPL code, and the company then licenses it outside the GPL for this commercial use.  For example, with MySQL, the client libraries were originally LGPL, but that meant commercial companies could use them in non-GPL apps without paying a license fee.  So, in order to generate revenue, MySQL changed the client libraries to GPL.  Trolltech did the same thing with QT.  Except for cases where the software is offered b2b as part of support contracts (think SuSe, Asterisk, etc.), which doesn't apply to us, licensing the software outside the GPL for commercial use seems to be the standard way that a company can offer software that is open source for the FOSS community, but that they can still make money for with commercial use.

That was our intention with UI3.  We want all the FOSS projects to be able to use it for free under the GPL, since they will release their UI's under the GPL too.  But, since we will be hiring a lot of developers for this, and probably sponsoring several more, our justification to our investors for spending that money is to be able to license the same software outside the GPL, like MySql and Trolltech do.

So if the back end design tool is Adobe Flash Designer, and the front end player is either Adobe or Gnash, we have no way to explain why we should invest a lot of money on development.  If the approach of using Flash Designer + Gnash was close enough to being a workable solution, or if there were enough volunteers to do the work, then it's not an issue; we don't need a revenue model because we won't need to invest a lot of money in it anyway.  However, my gut feeling is that this likely will require a substantial investment because, as an example, MythTV's devs would obviously like to have a UI as pretty as TiVo or Apple or Vista, and if it really was that easy to create a beautiful UI in Flash Designer and bolt it on top of Myth, Xine, MPlayer, etc., then they all would have been doing it already.  Since all the media center apps need pretty eye candy UI's, and it seems nobody has figured out how to do it with Flash or some other off the shelf toolset, it would seem to indicate that it's really not so simple and therefore a significant investment will be required to make it happen.  If that's the case, then perhaps the only way we could get approval on the investment is to develop everything from scratch so we could hold the licensing rights.  But as a developer myself, the last thing I want to do is rewrite code or reinvent something that's already done and working, and I'd much rather start with something that exists.

What would be great is if we could get some people involved in the discussion on the 19th who know Flash, Gnash and the other existing software really well so they can let us know what exists already and how much work would be required to make it work for UI3.  If these devs are free on the 19th and, ideally, not too far from California, Pluto will cover the cost of bringing them over.  If not, we plan to setup a web cam conference anyway.

[tschak909]

I would like to be present on the discussions, however my health (I have paroxysmal atrial tachycardia) will not permit me to go to SF (even when you and I had talked on the phone Aaron, I had just had an attack an hour earlier..)... so I would like an alternative means to participate.

-Thom

[Matthew]

I really would like to find a way to somehow base this on Flash because it will be impossible to reinvent Flash's design suite.

I'm glad we agree on that general premise. And I'm even more encouraged that we're thinking this all out loud in the community, and that it's not just some "done deal" between Pluto and TrollTech, with the rest of us just watching as we get to decide whether to just "go along for the ride" or not. And in case it's not obvious, I'm also very grateful to Pluto (and everyone else involved) for all the excellent work done so far. Not just the final supercool product, but all the design in it that makes it so expandable while so complexly featureful and integrated. And of course the "open mind" to use whatever works when it's available.

The one area that Adobe's design tools may be lacking in (I don't know for sure) is the cross-platform stuff.  Pluto's HADesigner is crude as can be, but, it was designed to target lots of platforms.  For example, the 'play' button in UI1 on windows ce pda, and the 'play' button in UI2 on the TV, as well as on the Cisco phone and mobile phone, are actually all the same object.
(...)
I'm not sure if Adobe's flash designer is object oriented like that, supporting inheritance and overrides, so you can create, say, a button that looks and behaves differently on various platforms.  I've never seen Flash used like this; the general goal of flash is the opposite: to have a player that renders the flash file exactly the same on all platforms, rather than having one UI that behaves differently on the different platforms.

GNASH is designed to be better than Adobe's Flash player in this regard, the support of many very different platforms with different presentation and computation resources (also a more efficient VM especially to benefit embedded devices). I'm not sure how GNASH supports "robustness failover", though I believe you are correct in seeing the Adobe Flash paradigm as the same as their Acrobat paradigm: the point is to present the same bitmaps on every platform. But keep in mind that GNASH supports SWF and ActionScript, not just Adobe's "Flash", so it has other language features as well. I don't know whether inheritance and overrides are supported techniques. But I do know that OpenLaszlo (again, an SWF/ActionScript tool, not necessarily Flash) generates each of SWF and DHTML by compiling the same source code with different output format requests. So there's at least one technique for targeting differing GUI toolkits on different classes of device, but retaining all the other benefits.

While this may be a weakness, it's not necessarily a deal breaker since, after all, if Flash designer is not object oriented with inheritence, the person designing the UI could work around this by just creating separate flash files for the mobile phone, the pda, the cisco phone, the TV, etc.  Most designers would probably prefer to create 5 UI's using Adobe's flash designer vs. 1 UI with Pluto's HADesigner because our HADesigner is that bad and it's unlikely we'd have the manpower to create something that rivaled Adobe's flash designer.

In fact I saw an interview with GNASH's originator, Rob Savoye in which he implies the approach for handling "degraded" GUIs on lesser devices is typically to produce different SWF content targeting what's supported there. But I expect there's a way to balance the two techniques, especially if there's going to be coding the player involved.

The other thing I'm not sure of is how well flash scales.  In general all the flash files I've seen are created for a target resolution (say 800x600).  With Pluto's HADesigner, OrbiterGen scales the UI to any resolution, rotates it (ie some devices are vertical not horizontal), and stretches it (16:9 vs. 3:4).  This would be a big deal breaker if Flash couldn't do this because, although we could tell designers they need to create 5 different UI's for all the target platforms, telling them to create each of those 50 times (5 UI's at 10 different resolutions) would be ridiculous.  If they wanted to change the command the 'play' button sent, rather than doing it in 1 place like you do with HADesigner, you'd have to make the same change to 50 UI's if Flash didn't support scaling/rotating/stretching and didn't support object oriented inheritance with overrides.

Well, one of SWF's central features is that it's primarily vector graphics, which are inherently scalable. And I've certainly seen plenty of little Flash widgets for as small as 32x32 animated icons. And Sun's incorporation of OpenLaszlo in its JME for little devices indicates that's a design requirement for its use in UIs.

Here what's basically lacking is the ability to blend 2 different applications together.  Flash is designed to work stand alone, not bolt on top of and blend into something else.  We're looking for a UI that can bolt on top of MythTV, or Xine, or MPlayer, or Firefox (for a 10' web browser), etc.  If, however, we commit to using Adobe's flash designer as a back end, then that means we've committed to the flash file format and we're better off to put resources into adding the alpha blending capabilty to something that already plays back flash files, like Gnash, than to start from scratch reinventing a flash player.

I think making GNASH do alpha blending is exactly the way to go. Since SWF (even with its recent 3D techniques) is designed to render into a plane, and Adobe's many other products that are designed in terms of bundles of layers, this could be relatively straightforward within the overall architecture. Also there is likely room to add some 3D support beyond blended layers to GNASH, where UI3 requirements exceed GNASH's current capabilities. I think adding those features to GNASH would also make a lot of friends with users and developers who would be assets to the LinuxMCE community. Of course it would be faster to market to add those features to GNASH than to invent a new player that competes with it and with Flash. Leaving more time to actually create the GUI "content" itself that executes in the GUI platform.

Here is where it gets tricky.  Pluto has investors, and when we put in the budget expenses for development, we have to show that there's some way that we can at least make enough revenue to cover the expenses, and ideally make a profit, otherwise it's considered throwing money away.

Normally the way companies do this with FOSS is to offer software under the GPL that, in order to be used commercially, needs to link with non-GPL code, and the company then licenses it outside the GPL for this commercial use.  For example, with MySQL, the client libraries were originally LGPL, but that meant commercial companies could use them in non-GPL apps without paying a license fee.  So, in order to generate revenue, MySQL changed the client libraries to GPL.  Trolltech did the same thing with QT.  Except for cases where the software is offered b2b as part of support contracts (think SuSe, Asterisk, etc.), which doesn't apply to us, licensing the software outside the GPL for commercial use seems to be the standard way that a company can offer software that is open source for the FOSS community, but that they can still make money for with commercial use.

That was our intention with UI3.  We want all the FOSS projects to be able to use it for free under the GPL, since they will release their UI's under the GPL too.  But, since we will be hiring a lot of developers for this, and probably sponsoring several more, our justification to our investors for spending that money is to be able to license the same software outside the GPL, like MySql and Trolltech do.

So if the back end design tool is Adobe Flash Designer, and the front end player is either Adobe or Gnash, we have no way to explain why we should invest a lot of money on development.  If the approach of using Flash Designer + Gnash was close enough to being a workable solution, or if there were enough volunteers to do the work, then it's not an issue; we don't need a revenue model because we won't need to invest a lot of money in it anyway.  However, my gut feeling is that this likely will require a substantial investment because, as an example, MythTV's devs would obviously like to have a UI as pretty as TiVo or Apple or Vista, and if it really was that easy to create a beautiful UI in Flash Designer and bolt it on top of Myth, Xine, MPlayer, etc., then they all would have been doing it already.  Since all the media center apps need pretty eye candy UI's, and it seems nobody has figured out how to do it with Flash or some other off the shelf toolset, it would seem to indicate that it's really not so simple and therefore a significant investment will be required to make it happen.  If that's the case, then perhaps the only way we could get approval on the investment is to develop everything from scratch so we could hold the licensing rights.  But as a developer myself, the last thing I want to do is rewrite code or reinvent something that's already done and working, and I'd much rather start with something that exists.

Well, if the platform is SWF+ (including some extensions that let UI3 do a few things SWF doesn't already do), then it won't cost the investors as much to produce. The main financial benefit to the upgrade investment is that the better product will sell more. To the point, more competitive with Windows MCE, no longer held back by comparison to the familiar (and therefore "easy", at least sold as such) Windows GUI. The SWF architecture also creates the opportunity for Pluto to create separate GUI content that doesn't have to be GPL (or Creative Commons, or anything else other than simple private Pluto copyright). Pluto is free to create GUIs better than anything else. To create branded GUIs in private deals with equipment manufacturers or retailers/installers, that the community can't make, because it's not a simple corporation. Pluto's expertise in the way the new GUI works puts it in the lead to make the best GUIs. The community can make its own GUIs, perhaps with Pluto's help once Pluto recoups its investment on the "private label" GUI. All of which will show the power of the platform, which will grow the whole ecosystem, with Pluto still at its center.

In short, Pluto and the community can collaborate to cut the costs of the extra development on top of reusing the FOSS already incorporated. And position Pluto to make money from the new GUI and overall sales in ways that are more profitable than before.

What would be great is if we could get some people involved in the discussion on the 19th who know Flash, Gnash and the other existing software really well so they can let us know what exists already and how much work would be required to make it work for UI3.  If these devs are free on the 19th and, ideally, not too far from California, Pluto will cover the cost of bringing them over.  If not, we plan to setup a web cam conference anyway.

I'm in NYC, and I'm not exactly an expert in GNASH. But I've used the public SWF tools before (and I used to develop developer tools for Apple), as well as IDEs for targeting multiplatform (and crossplatform) content and executables. And I think I have a handle on the issues here, at least from the perspective of an outside developer who wants to help LMCE get as far as possible as fast as possible. If you want to fly me out, I'm game, but I'm OK with a webcam and speakerphone. And I'm really enthusiastic about the way that Pluto is going about exploring all the dimensions of this new chapter in LMCE, which is consistent with it's history on the project.

[aaron.b]

Matthew,  That's great.  I sent a private email to you about joining the meeting.  My email is aaron.b  [at] plutohome (dot) com

I also contacted Rob Savoy, lead dev of Gnash, and have invited him.  We already exchanged emails and he sounds interested.  We might be able to get him to work with us on this.

[aaron.b]

I had a nice chat with Matthew.  One thing that's clear is perhaps I didn't correctly state what I meant by needing a UI with object oriented properties, like inheritance.  Here's a real world example...

1.   Someone is creating a new screen in the design tool.  In the tool, there exists style sheets with all the font styles.  The style "Mid-size button" is defined as Times Roman 10 point. I create a button called 'play'.  It has a blue background that pulsates with an animated effect 3 times per second, and has the text "PLAY" on top of it in that "Mid-size button" style.  I cluster this together with a "pause" and "stop" button in a container called "Media Controls".

2.   A different designer doesn't like the blue background, and instead wants something with wood grain.  He doesn't want to recreate the screen, or change the layout or have to duplicate the commands associated with the buttons.  Just change the background.  So, he creates a new skin called 'wood' in the design tool, and creates a wood grain bitmap, and in the design tool associates this new bitmap with the skin.  Now, when the user selects the 'wood' skin he sees the same button with 'play' in Times Roman 10 point, but the background is wood grain.  The only thing is the pulsating animated effect doesn't look good on top of the wood, so he chooses another effect called "scroll" which makes the background wood grain image scroll a bit.  There may be 100 buttons the original designer created with the blue background and the pulsating effect.  This designer shouldn't have to open the properties for each button one at a time, he should be able to make a global change that says for this new skin, use this wood grain bitmap and scroll effect instead.

3.   Now there's a user running the UI on a PDA.  He finds that with the original blue skin it looks ok except that the pulsating effect is using all the cpu, and with the wood skin, the scrolling is just too much for the pda.  So he opens the 'play' button in the design tool, clicks on the 'effects' tab, chooses the 'blue' skin and sees the 'pulsate' effect, he clicks 'criteria' and under "target device" selects "PDA", and then moves to the property field "times per second" and changes it from 3 to 1 so it goes slower on the pda.  Then he chooses the 'wood' skin, again sets the criteria to 'target device=pda', and this time sets 'effect' to 'none'.  This doesn't effect either user #1 or #2 since they don't use a pda.

4.   Now there's another guy using a Cisco VOIP phone.  He sees that on his phone all the effects are too much for the phone and need to be disabled.  And although the 'Times Roman 10 point' text is legible with the 'blue' skin, when he choose the 'wood' skin which has more contrast, the text is unreadable because the Cisco phone's lcd display isn't that sharp.  So, he opens the design tool, selects 'platform = cisco voip phone' and checks off "turn off all effects".  Then he opens the style sheet tool and selects the "mid-size button" style, and clicks on "criteria" and sets "skin" = "wood", and changes the font size from 10 point to 12 point.  Except, he notices, when the language is German.  Then the words are just too big for 12 point font so the user has to live with 10 point, so he edits his criteria and adds to "skin=wood" a "and language<>german".

5.   Another user decides he wants to create a new variation called "sideways" that aligns the clusters vertically rather than horizontally.  He doesn't want to change any of the other stuff.  He just wants to open "Media Controls" cluster, which contains the play/pause/stop buttons in a horizontal group, and he re-arranges them so they are vertical.  They retain all their current properties; the play button still pulsates when it's the blue skin, except on the Cisco phone, and the font size is still 12 point instead of 10 with the wood skin on a cisco phone.  It's just arranged differently.

Note that whatever file format this master stuff is stored in must be something that supports collaborative development, so different users can make changes to the same object, and then those changes are merged back into the 'master'.

This is how Pluto's current HADesigner works (or at least how it was designed to work).  This is what I mean by it's object oriented and objects inherit properties, which you can override based on criteria.  There's a criteria builder (it's not very functional, but it's there), so you can build a criteria "skin=wood, device=cisco phone, resolution=less than 640x480".  And then you can change the properties of a button for that particular criteria.  You can also build a criteria for a text style, so that when 'device=cisco and skin=wood' the font size is 12 point and not 10.  The file format for this 'master' definition is an sql database.  And sqlcvs was created with HADesigner in mind so that the 5 users described above could all be changing the properties of the 'PLAY' button at the same time and have their changes get merged together.  Then, the program OrbiterGen runs which reads the 'master' definition from the sql database, and outputs a .info file which is the native format that Orbiter reads.  Orbiter doesn't read the master definition.  If you are running Orbiter on a cisco phone with the 'wood' skin, when it reads in the 'play' object, it just sees an object that has no pulsate/scroll effects, and is already scaled for the cisco phone, with the wood background and the word PLAY in 12 point text.  OrbiterGen had the responsibility for each object of evaluating the "criteria" to pull out the properties for that object given the target platform, resolution, skin, operating system, ui version, etc.

From a conceptual standpoint, I don't think HADesigner was bad.  The idea that everybody could edit the same objects and, without re-creating those objects, just change certain properties for specific situations made sense.

The challenge is that (a) HADesigner just awful to use, and (b) really limited in it's eye candy.  All the images are bitmap based (not vector based).  You can make them animated by using the MNG format (an animated bitmap, like animated gif), but this hasn't really been used at all.  There's a bunch of other ways that it's totally lacking.  The only thing I like about it is the concept behind the object properties and the collaboration possibilities.

Now let's say we want to let graphics designers use Adobe's Flash Design tools.  It has lots of great widgets for creating pretty objects that are animated, etc.  But, if it didn't support the concept of overriding object properties like I described in those 5 examples (which I don't think it does) it would be nearly impossible to create the kind of UI's that we do that target all those platforms.  Sure, the objects may have properties and the user could attach action script code which changed them.  But having a huge cluster of if statements in action script is not maintainable.  If skin=wood and platform=cisco and <blah blah> then object.text.style.pointsize=12; else if skin=blue and <blah> <blah>..  Just for those 5 steps I described above, you'd have dozens and dozens of nested if statements in a mangle of spaghetti code.  It would be a nightmare for the flash developers to have to attach hundreds of line of 'if' statements to each of the objects in the UI (there are many thousands).  And the developer would go crazy if, for example, every time he wanted to change something for the 'wood' skin he had to edit the action script of 1,000 buttons and add an 'else if skin=wood' to it.  And if collaboration would be impossible since you couldn't merge all that action script code; if 5 developers were each working on 5 skins, and all 5 were changing this huge block of nested else if statements, they'd have conflicts for each of the 1,000 buttons.  So rather than the design tool giving the user action script for each object, or button, the design tool has to support concepts of styles and properties which are inherited across all objects, with global criteria that you can change properties in one place and it propagates across all objects.  And to allow designers to work freely, they shouldn't have to edit code to do this; it should be done wysiwyg in the design tool.  I don't know Adobe's Flash designer, but I think that it doesn't work like this.

Also, I'm even sure the SWF file format is by itself sufficient.  I'm not sure that the swf format supports the ability to have that criteria/object properties.  After all, this isn't what flash and swf were designed for.  And, even if the swf format did support this, you probably can't pass that directly to the target render device (ie the Cisco phone).  As stated in my example with the 5 steps, to determine what the 'play' button should look like on a cisco phone with the 'wood' skin you have to evaluate a bunch of criteria factors; what the skin is, what the platform is, etc.  You don't want the player in the Cisco phone (which has a weak cpu), to have to go through that whole evaluation process to determine what the properties are to display the 'play' button.  It would be way too slow.  Rather there needs to be some other intermediary process that takes the master file that defines how a play button should work on all the various skins/platforms/languages/etc., and pre-processes the play button for the Cisco UI, pre-scaling any bitmaps, pre-rendering any fonts, etc., so that the player in the Cisco phone just gets the finished 'play' button and doesn't need to do a bunch of complex calculation.  This intermediary process is what OrbiterGen currently does.  So there would likely still need to be an intermediary program which reads in the master file, evaluates everything, and outputs a pre-rendered/scaled UI for a specific box.

If we wanted to use the swf format, and if the swf format is, as I suspect it is, really targeted for a certain platform/skin, then likely the 'master' format which defines all the objects and which the design tool uses would need to use another format, and this intermediary program that replaces OrbiterGen is what would need to read this master format, evaluate all the criteria for whatever UI it's targeting, and output a generated swf that could be played on it.  I think this is similar to what Laszlo does.

So this raises a bunch of issues and questions.  What is the design tool?  Adobe's Flash Designer is closed source.  So unless it did everything we need it to do out of the box we couldn't extend it to suit our needs.  Is there another design tool that's really polished and could be extended?  Does swf support all the object inheritence we need so that it can be the master format?  If so, should we feed that 'master' swf to each UI, even though that master swf defines every criteria for each object, or should we do some pre-processing so each UI just gets an swf that's pre-rendered for it?

Does anybody know if there exists a UI design tool that works like HADesigner does with object criteria/properties/inheritance to target multiple platforms and skins like Pluto does?  Is there another way to target all those platforms that doesn't require this approach?  If there's no way to get flash to do what we want with the multiple platform stuff, but if flash will let us create an eye-candy rich UI, maybe we need to consider scrapping the idea of having one unified UI/design tool that targets all those platforms and just use flash for the 10' UI on the TV, but leave HADesigner/OrbiterGen/Orbiter as-is for the other platforms like the Cisco phones...

[totallymaxed]

I'll divide my comments in 3 parts: 1) the design tool, 2) the player, 3) financial issues.  Both the design tool and the player are *almost* what we need, but, lacking in a couple areas.

--Design tool--

I really would like to find a way to somehow base this on Flash because it will be impossible to reinvent Flash's design suite.  I have never used it, but just from looking at the demo video on Adobe.com, it is clearly a very rich, robust, polished design suite that allows the designer to create gorgeous UI's that do almost anything with sophisticated illustration and 3d tools.  The thought of us saying "Let's rebuild something that's as good as Adobe's flash designer" is daunting; that software was 10 years in the making with hundreds of full time developers.  We just can't make something as elegant as it.  And we'll never be able to reproduce having tens of thousands of developers who know it intimately so they can create cool eye candy effects in our UI.

The one area that Adobe's design tools may be lacking in (I don't know for sure) is the cross-platform stuff.  Pluto's HADesigner is crude as can be, but, it was designed to target lots of platforms.  For example, the 'play' button in UI1 on windows ce pda, and the 'play' button in UI2 on the TV, as well as on the Cisco phone and mobile phone, are actually all the same object.  You just assign different attributes for each of the variations (like object oriented inheritance), so you can make it be animated and round on a more powerful box, and a small static square on a mobile phone.  You can also cluster the 'play/pause/stop' buttons into a 'media playback group', and say that the group itself is also different on a tv vs. a mobile phone (ie object containers).  Then, when you're creating a media remote control, you just drop the 'media playback group' on the screen and whether you're using a mobile phone or a tv it still works.  I'm not sure if Adobe's flash designer is object oriented like that, supporting inheritance and overrides, so you can create, say, a button that looks and behaves differently on various platforms.  I've never seen Flash used like this; the general goal of flash is the opposite: to have a player that renders the flash file exactly the same on all platforms, rather than having one UI that behaves differently on the different platforms.

While this may be a weakness, it's not necessarily a deal breaker since, after all, if Flash designer is not object oriented with inheritence, the person designing the UI could work around this by just creating separate flash files for the mobile phone, the pda, the cisco phone, the TV, etc.  Most designers would probably prefer to create 5 UI's using Adobe's flash designer vs. 1 UI with Pluto's HADesigner because our HADesigner is that bad and it's unlikely we'd have the manpower to create something that rivaled Adobe's flash designer.

The other thing I'm not sure of is how well flash scales.  In general all the flash files I've seen are created for a target resolution (say 800x600).  With Pluto's HADesigner, OrbiterGen scales the UI to any resolution, rotates it (ie some devices are vertical not horizontal), and stretches it (16:9 vs. 3:4).  This would be a big deal breaker if Flash couldn't do this because, although we could tell designers they need to create 5 different UI's for all the target platforms, telling them to create each of those 50 times (5 UI's at 10 different resolutions) would be ridiculous.  If they wanted to change the command the 'play' button sent, rather than doing it in 1 place like you do with HADesigner, you'd have to make the same change to 50 UI's if Flash didn't support scaling/rotating/stretching and didn't support object oriented inheritance with overrides.

--Player--

Here what's basically lacking is the ability to blend 2 different applications together.  Flash is designed to work stand alone, not bolt on top of and blend into something else.  We're looking for a UI that can bolt on top of MythTV, or Xine, or MPlayer, or Firefox (for a 10' web browser), etc.  If, however, we commit to using Adobe's flash designer as a back end, then that means we've committed to the flash file format and we're better off to put resources into adding the alpha blending capabilty to something that already plays back flash files, like Gnash, than to start from scratch reinventing a flash player.

--Financial Model--

Here is where it gets tricky.  Pluto has investors, and when we put in the budget expenses for development, we have to show that there's some way that we can at least make enough revenue to cover the expenses, and ideally make a profit, otherwise it's considered throwing money away.

Normally the way companies do this with FOSS is to offer software under the GPL that, in order to be used commercially, needs to link with non-GPL code, and the company then licenses it outside the GPL for this commercial use.  For example, with MySQL, the client libraries were originally LGPL, but that meant commercial companies could use them in non-GPL apps without paying a license fee.  So, in order to generate revenue, MySQL changed the client libraries to GPL.  Trolltech did the same thing with QT.  Except for cases where the software is offered b2b as part of support contracts (think SuSe, Asterisk, etc.), which doesn't apply to us, licensing the software outside the GPL for commercial use seems to be the standard way that a company can offer software that is open source for the FOSS community, but that they can still make money for with commercial use.

That was our intention with UI3.  We want all the FOSS projects to be able to use it for free under the GPL, since they will release their UI's under the GPL too.  But, since we will be hiring a lot of developers for this, and probably sponsoring several more, our justification to our investors for spending that money is to be able to license the same software outside the GPL, like MySql and Trolltech do.

So if the back end design tool is Adobe Flash Designer, and the front end player is either Adobe or Gnash, we have no way to explain why we should invest a lot of money on development.  If the approach of using Flash Designer + Gnash was close enough to being a workable solution, or if there were enough volunteers to do the work, then it's not an issue; we don't need a revenue model because we won't need to invest a lot of money in it anyway.  However, my gut feeling is that this likely will require a substantial investment because, as an example, MythTV's devs would obviously like to have a UI as pretty as TiVo or Apple or Vista, and if it really was that easy to create a beautiful UI in Flash Designer and bolt it on top of Myth, Xine, MPlayer, etc., then they all would have been doing it already.  Since all the media center apps need pretty eye candy UI's, and it seems nobody has figured out how to do it with Flash or some other off the shelf toolset, it would seem to indicate that it's really not so simple and therefore a significant investment will be required to make it happen.  If that's the case, then perhaps the only way we could get approval on the investment is to develop everything from scratch so we could hold the licensing rights.  But as a developer myself, the last thing I want to do is rewrite code or reinvent something that's already done and working, and I'd much rather start with something that exists.

What would be great is if we could get some people involved in the discussion on the 19th who know Flash, Gnash and the other existing software really well so they can let us know what exists already and how much work would be required to make it work for UI3.  If these devs are free on the 19th and, ideally, not too far from California, Pluto will cover the cost of bringing them over.  If not, we plan to setup a web cam conference anyway.

Hi Aaron,

As you know from our conversations over the last 18 months or so we also have been looking at building a UI for LinuxMCE in Flash. This work has been underway for about 6-7 months now... but back in October we decided that we had to focus on UI2 so that we could get to market with our customers earlier.

However we do believe that Flash is the way to go because of the great tools for designers that already exist and because the designers out there already understand how to build great UI's in Flash. So I 100% agree with Mathews comments here about Flash/Gnash etc.

So we would be very interested in a UI3 that was Flash based as our UI team is already developing in Flash. Some of our UI code would be redundant under the proposed UI3 architecture but in other areas we would have little or no work to do as much more of the "glue" underneath the skin would be part of the framework as opposed to something we would build on-top.

We would be happy to commit some resources to help support the Open Source effort to allow a full scalable & portable Flash UI3 to be delivered as this would allow us to focus back on our design ideas.

[Matthew]

Alternately, if OpenLaszlo, like Flash, is just too big and complex, what about SVG? SVG with ECMAScript can make lightweight but featureful GUIs that run in browsers and can be encapsulated in runtimes (Java or native x86/etc binaries). Again, why reinvent the wheel, rather than just build a better car or road for it?

There are also tools to convert SWF to SVG, including one SWF2SVG that's actively developed and just released a version this month. These tools (AFAICT) all render individual frames from SWF to SVG, not animated SVG. But they're at least a starting point for generating SVG from a single codebase that, with OpenLaszlo, could then generate SWF, DHTML and SVG, depending on the device targeted by developers at compile time.

[kir]

My thoughts:

- I don't think that Flash is a good choice for the UI3 basis: it is based on different ideas, it is used for different purposes and it is not open. In fact, the biggest common thing with UI3 in Flash is that it is sometimes used for building user interfaces In fact very often, it is not: I've seen much more sites based with UI based on pure HTML, HTML+JavaScript, etc. etc. than based purely on Flash. I.e. even for the UI-building task, Flash is not the 'best' tool. Web 2.0 sites for example, are based on AJAX and other similar approaches, not on the Flash itself, although they can embed Flash into them. For the desktop applications (i.e. running outside of the browser), I even cannot remember any _big_ application with Flash-based interface. So - selecting Flash is like selecting an old somebody's else SUV car and trying to use it on races: it was designed for different purpose, it is old and you cannot heavy-tune it.


- The same applies to the Gnash - with addition that Gnash (in the sense of development direction) is a tail of Flash pony: Gnash has to follow the Flash features and evolution. It will never be the separate.


- Regarding the design of UI3: I am not sure that it is a good idea to create it from scratch, because of the following:
0) most of the time it will be not feature-ready and incomplete, so everybody will still use the UI2 and see the "stall in the development"

1) if you want to design it from scratch, you will most probably have to hack all and every piece in whole LinuxMCE system, to propagate new ideas/design approaches - or you will restrict your "open mind" to some subset of things (say, "UI3 has to coexist nicely with applications A,B,C") thus losing half of "from scratch" benefits

2) parts of discussion, especially "big list of all required features and possible uses" reminds me about the well known "second-system" effect ( http://en.wikipedia.org/wiki/Second-system_effect ) - I am afraid this attempt can fail because of too big initial expectations/design features. It should be really lightweight

3) I don't see the list of "problems" with UI2 except that it sometimes lacks developers' attention - wouldn't it be better to start refactoring the existing code, creating "UI evolution roadmap", adding features, switching to new approaches/implementation, etc.


- Regarding the IDE: don't care too much about the big and powerful IDE at the beginning of journey: if the development be successful, the IDE will appear. Of course, certain ideas should be taken into account, such as - it is always easier to modify human-readable file that to issue SQL queries. Like with websites - webdesigners doesn't use IDE 100% of time - they spend a lot of time in notepad-like text editors, tweaking the HTML,CSS,JS,etc.


- Personally I think that "evolution" is better than "revolution" and would vote for the creating UI2.1, UI2.2 etc. with extra features/improvements


- For enhancing the usability, I would suggest to ask someone who interested to modify LinuxMCE current UI and see what are the common issues/things-hard-to-do - and compose a list of them and eliminate/optimize them one-by-one.


Now, couple of random thoughts related to the above description of "desired features" and logic of UI design:

- Description of objects properties inheritance and conditional properties evaluation, reminds me about HTML+CSS style of UI implementation - does anybody else see this analogy? Maybe we could utilize it and define the engine layouts in kind of XML-ish format. And use CSS-analogue approach for defining stylesheets, too.


- Targeting the same interface to all possible devices doesn't look as a good approach to me: compare, e.g. big 50" plasma TV and ~160x120px cellphone screen: trying to have the same screen for them, even styled differently doesn't seem feasible. Or the result will be non-optimal. So, why not divide all devices into few categories, based say on screen size, CPU power and network speed:
   screen size: high (TVs and displays), medium (PDAs, Cisco-like phones), low (mobile phones)
   cpu/rendering power: high ( PCs ), medium (PDAs, Cisco-like phones), low (mobile phones)
   network speed: high (100/1000Mbit), medium (WiFi), low (Bluetooth)

The ranges can be more flexible of course Of course this doesn't mean that there should be 3*3*3=27 different interfaces: the main could be the screen size (so there should be 3 different interfaces), the CPU power will determine which effects are on/off and if the device is 'active renderer' (like Orbiter) or passive (like plain bitmap display with off-screen renderer sitting e.g. on core). The network speed will allow to decide what media can be streamed, if the device can work as "bitmap display" or should be even falled back to text-only interface (if network is too slow).

So it is kind of "enumeration of capabilities" for different target devices and tuning/selecting proper UI for them.


- Scripting: imho it would be nice to have kind of scripting engine for the GUI. It can be either JavaScript or QTScript depending on what is better. - this will allow to extend UI w/o doing direct coding. Again, many people are familiar with scripting languages so this would be a useful feature. Scripting support was an overkill 5-10 years ago, but now PCs are faster, so this is not an issue. For scaling down to the slow devices, the script code can be executed on core in the way similar to off-screen renderer for bitmap rendering.


- Finally, if creating UI3 from scratch will be selected, I would suggest to utilize 3D-games developers experience - there are many nice tricks/ideas that can be used to create nice and extensible interface. "World organization" (for UI it is about widgets, and binding the actions to them), scripting, skins and even audio/video playback - all this is present in modern games. Including need of scaling to different resolution (within some range).

[Matthew]

- I don't think that Flash is a good choice for the UI3 basis: it is based on different ideas, it is used for different purposes and it is not open. In fact, the biggest common thing with UI3 in Flash is that it is sometimes used for building user interfaces In fact very often, it is not: I've seen much more sites based with UI based on pure HTML, HTML+JavaScript, etc. etc. than based purely on Flash. I.e. even for the UI-building task, Flash is not the 'best' tool. Web 2.0 sites for example, are based on AJAX and other similar approaches, not on the Flash itself, although they can embed Flash into them. For the desktop applications (i.e. running outside of the browser), I even cannot remember any _big_ application with Flash-based interface. So - selecting Flash is like selecting an old somebody's else SUV car and trying to use it on races: it was designed for different purpose, it is old and you cannot heavy-tune it.


- The same applies to the Gnash - with addition that Gnash (in the sense of development direction) is a tail of Flash pony: Gnash has to follow the Flash features and evolution. It will never be the separate.

Maybe that's true for "Flash", but Adobe's Flash is just their branded instance of the open standard SWF. GNASH is an open SWF player (and FLV server) that doesn't actually have to follow Flash, just SWF. It does, however, follow Flash without LMCE doing anything, so that's not a problem. However, SWF does indeed reflect quite a lot of what UI3, and LMCE in general, need to do, and the ways to do it. As I've posted in this thread, Sun has included OpenLaszlo in its new Java platforms that are already parts of the Blu-Ray, DVB and mobile phone specs. So lots of devices already work according to its design, and even include support for features that OpenLaszlo scripts.

FWIW, just because HTML is used for a lot more GUIs than is Flash doesn't mean Flash isn't good for a GUI, or for specifically UI3. Just because most cars on the road are sedans doesn't mean that pickup trucks are inappropriate for working construction sites. LMCE is much more complex than most software, especially the Web software that's pretty crippled in its UI paradigm and widgets (and which has to get downloaded with every new page).

1) if you want to design it from scratch, you will most probably have to hack all and every piece in whole LinuxMCE system, to propagate new ideas/design approaches - or you will restrict your "open mind" to some subset of things (say, "UI3 has to coexist nicely with applications A,B,C") thus losing half of "from scratch" benefits

Certainly not. LMCE is a very properly tiered architecture. Only the GUI needs to be touched for UI3 to take over. We're talking about the Orbiter, in various platforms (standard Orbiter, Web Orbiter, mobile Orbiter, IP phone Orbiter, etc).

- Regarding the IDE: don't care too much about the big and powerful IDE at the beginning of journey: if the development be successful, the IDE will appear. Of course, certain ideas should be taken into account, such as - it is always easier to modify human-readable file that to issue SQL queries. Like with websites - webdesigners doesn't use IDE 100% of time - they spend a lot of time in notepad-like text editors, tweaking the HTML,CSS,JS,etc.

It's irresponsible to replace a GUI with a new system without including a GUI designer (or allowing use of an existing one). Especially when one major limit of LMCE is that its old GUI designer is acknowledged totally unusable by both its users and its authors. This requirement is a driving force in producing a UI3. "It'll come" is no way to plan a project, especially when it'll have to come from us. We have to pick the framework that will most reliably include a GUI design/composition system, like a data format that already has one, or plan to write one ourselves.

- Description of objects properties inheritance and conditional properties evaluation, reminds me about HTML+CSS style of UI implementation - does anybody else see this analogy? Maybe we could utilize it and define the engine layouts in kind of XML-ish format. And use CSS-analogue approach for defining stylesheets, too.

Whatever we decide to do to produce the UI improvements, it will probably include runtime configs that style the GUI. HTML+CSS is one example of that model, but is very limited (owing mostly to the limits of HTML).

- Targeting the same interface to all possible devices doesn't look as a good approach to me: compare, e.g. big 50" plasma TV and ~160x120px cellphone screen: trying to have the same screen for them, even styled differently doesn't seem feasible. Or the result will be non-optimal. So, why not divide all devices into few categories, based say on screen size, CPU power and network speed:

No one is saying that a 50" TV and a cellphone should each show the same GUI. Just that it's valuable to have the most common code/content as far down the development stream towards the user and runtime as possible. Orbiter already does that, but could improve its common base to be shared further up the stream towards the developer. This paradigm is already evident in LMCE in GSD and handling devices that are controlled, not that are used as controllers. Making all of LMCE work that way for users, no matter what the role of the device in the LMCE ecosystem, is another advance that UI3 can bring, if we do it right.

I'm not really sure that you really understand GUI development, or LMCE's architecture. You also seem to be wrong about SWF and how it actually does relate to UI3 (or to Flash) on LMCE. Where are you getting these perspectives from?

- Scripting: imho it would be nice to have kind of scripting engine for the GUI. It can be either JavaScript or QTScript depending on what is better. - this will allow to extend UI w/o doing direct coding. Again, many people are familiar with scripting languages so this would be a useful feature. Scripting support was an overkill 5-10 years ago, but now PCs are faster, so this is not an issue. For scaling down to the slow devices, the script code can be executed on core in the way similar to off-screen renderer for bitmap rendering.

I am working on scripting LMCE (not just the GUI) by integrating Perl. Which is best done by exposing LMCE state to a script engine, and sending DCE messages to the DCErouter. Scripting the GUI itself seems like a redundant, and hard (given the various GUI HW) method that ignores the strength of the DCErouter.