UI3 Discussion

[Matthew]

Aaron B (Pluto CEO) posted a long examination of his thoughts about the next version of the Pluto/LMCE GUI, that he calls "UI3" (not the UI2/alpha-blended that people sometimes call UI3), in the wiki article "UI3". We can discuss it in this forum topic. Here's the current version of the article:

UI3
Background

Macromedia Flash has become the defacto standard for creating user interface's for the web, with a rich development environment to create an appealing UI, and an efficient player to do the rendering with lot's of eye candy and visual effects. Media centric applications with 10' user interfaces ("MCE apps") have the same needs, but there really does not exist any software that can do this for desktop applications, like Flash does for web applications.

A few MCE apps try to use Flash for their user interfaces. But this does not work very well since this isn't Flash's intended purpose. For example, the UI in a MCE app cannot be stand alone and isolated, rather it must be a UI that floats on top of whatever media the MCE app is delivering. The Flash player is not able to blend it's rendered output on top of the media application, like Xine or MythTV. So although you could create a really attractive EPG program guide in Flash, you cannot blend that guide on top of the live TV in MythTV.

Therefore, nearly all the MCE applications, like Xine, Myth, Windows MCE, etc., typically code their own UI's. This is inefficient; it is like re-inventing the wheel each time. And, most of the time, the developers are more focused on their core application, not the UI, and cannot create a rich UI engine. For example, Xine, MythTV, Mplayer, Freevo, etc., all have their own UI's, but none are rich with eye candy and visual effects. In general, only Microsoft has been able to develop a really eye-candy rich UI with lots of visual effects, and most other media center applications have fairly boring, static UI's, even if the core media application itself is rich. For example, while MythTV developers began implementing a rich OpenGL UI engine over two years ago, this is still only used for some of the menu system and had not been extended to the video overlay or plug-ins, and because there is no UI editor the capabilities of it's graphics engine go largely unused.

The goal, therefore, is to create (1) a UI player, like the Flash player, which is efficient and can render high-res, 3d, user interfaces with lots of eye candy on a variety of platforms (x86 Linux/Windows, PDA's, cell phones, etc.), and (2) a comprehensive development environment, like the one Macromedia developed for Flash, that is well documented and not overly technical so creative designers can use it to build innovate UI's.

The intention is that this can be used to create UI's for a variety of MCE apps, like MythTV, MPlayer, etc., and that unlike Flash which is designed to run as an isolated application, this UI will be designed from the ground up to bolt on top of the core MCE app. In other words, you can create a beautiful EPG, like you could with Flash, but it can be blended on top of MythTV's live tv feed and used to control MythTV. The UI tool should be modular and flexible so graphics designers can create interesting UI's without needing the help of the coders of the core MCE app. For example, a designer could create, say, an innovate 3d TV program guide on a rotating sphere and add it to MythTV without needing the MythTV developers to write new code. Having a unified UI will also allow us to address some of the vexing issues, like text entry using remote controls of various types, with new UI widgets which only need to be learned once for the entire class of 10' applications.

It is understood that this is a big, complicated task which hasn't really been done before. However, we feel it is an important gap that needs to be filled. The first thing most end users notice is the eye candy in the UI and how pretty it is, and only Apple and Windows Vista have had the resources to create such UI's. This will allow other MCE apps to have UI's that are just as catchy and the developers can focus on the core functionality.

Target user / financial model

Naturally this tool would be an asset to the FOSS MCE apps, like MythTV. However, there are many commercial applications as well. For example, only TiVo and Moxi have created rich UI's for their PVR's. There are tens of millions of PVR's out there from Motorola and Scientific Atlantic distributed by the cable operators, but they cannot justify the huge expense of creating a rich UI, so in general they have ugly, plain, static UI's. If there were an inexpensive, easy way to add a beautiful UI to these PVR's it would be worth it. Also, most TV's have a user interface for on screen menu's that is static and boring even though the TV's themselves have graphics processing chips in them that have the power to render a rich UI. There are numerous DMA's and set top boxes as well that could benefit from a rich UI as well. Mobile phones and PDA's also have media players and mini MCE apps.

In general these UI's are powered by a few single chip solutions, such as Sigma Designs, Broadcom and Marvell, and specialty chips for TV's, like AMD's Xillion. Macromedia Flash's player has been ported to many of these platforms so you can, for example, navigate a flash UI on a mobile phone. But, again, since Flash doesn't work well with MCE apps, in general these MCE apps all have crude, basic UI's, or they look to specialized middle-ware providers like Syabase, Mediabolic, Digion, etc., to create a UI for them.

The UI player will first be developed for X86, probably Linux and Windows in parallel. The software would be developed in such a way that the UI itself was subject to the GPL license, so there is an immediate potential to license the UI player for some commercial MCE apps. There's concern, for example, that SageTV will have a hard time remaining viable since it just doesn't have the pretty UI that Windows MCE does. They would be a potential customer to license the UI player outside the GPL since they would not want to release their UI under the terms of the GPL.

Over time the UI player could be ported to other platforms, like the Sigma, Broadcom, etc. Then the makers of set top boxes could use the UI generator to build rich UI's which could be played back on those set top boxes. By being cross-platform, we would have a strong advantage because the UI developer could target any platform for which the player existed, like Flash developers do now. Also, by being open source, the barrier of entry would be quite low. Anybody could start using the tool set, build it upon it if needed to add new visual effects, and only need to license it when they wanted to deploy a commercial application. And there would not be a concern of being locked into a closed, proprietary solution like the existing middle ware providers.

This relates to Pluto's core business of licensing an interoperability platform. Once a company adopted the use of the UI, Pluto would be able to offer a variety of modules, like lighting control, telephony, etc., that could easily 'bolt on' to the UI delivering a lot of extra functionality without writing any specialized code. If SageTV, for example, used the UI as their front end, Pluto could then offer lighting control modules, some to view security cameras, make phone calls, etc. which could be added to the SageTV package without them writing any new code. Pluto will be willing to sponsor the development of this new UI in exchange for receiving a license allowing Pluto to commercially license the UI software outside the GPL.

If QT is providing the underlying graphics library and hardware abstraction, then this benefits Trolltech as well because every commercial user who didn't want to release the UI under the GPL would also need to obtain a license to QT. The hope is that QT would also have an interest in furthering the development of this software.

How this relates to LinuxMCE

The UI in LinuxMCE, Orbiter, was developed to accomplish this same task. It has succeeded in some ways. For example, LinuxMCE's UI can be played on both Linux and Windows pc's, web pads, pda's, mobile phones, desktop phones, and web browsers. All those platforms are rendering the same UI and same set of screens created with a single design tool, called Designer. The problem is that Orbiter and Designer were created nearly 5 years ago, and were not very well designed. They are still quite crude. Designer is nearly impossible to use and does not appeal to graphic designers. The limited eye candy and visual effects in LinuxMCE, like the 3d cube media browser, are hardcoded into the player, Orbiter, rather than being part of the design as they should be. So, Orbiter and Designer are not general purpose enough to be useful to other MCE apps.

Skill sets needed

Here are 4 groups we need:

• First we need people who really understand how to design a great UI. These could be Flash designers who have created rich UI's before. They need to outline what the new UI software must do to appeal to the designers that will be building the UI's. That is the most essential thing since, if designers are creating, say beautiful, rich TV program guides with this software, it will be easy to approach the commercial PVR makers and show them why they need to use this UI in their product. So this team is more composed of graphic designers, not C++ coders.
• Next we need to know how to develop the UI design tool. There will be a lot of overlap with #1 since the people in #1 will be the users of the code written by this group. The UI design tool will not need to be cross platform. Windows/Linux x86 is acceptable. But it will have to be logical and similar to other design tools, like Macromedia's flash development tools, so designers can start using it without a steep learning curve.
• Next we need people who create MCE apps and know what the player must do to integrate well with those apps. For example, what are the low level C-language hooks the player must have to handle alpha blending the UI on top of MythTV. What is the socket-based control needed to allow the player and MythTV to seamlessly communicate as though the player were developed as part of MythTV, so the user is not aware that the UI is in fact a separate piece of software.
• The player needs to be highly efficient using a lot of the tricks that the video game industry uses to create high res 3d effects with smooth animation. For this we need some people who understand the Windows Direct X architecture, as well as the OpenGL fragment programs used by Linux and OS X, and some of the single chip solutions, so the player can be designed in such a way that as much code as possible is shared across all platforms, yet the player takes advantage of the unique hardware acceleration tricks each platform offers to be lean and efficient.

Proposed next steps

Some of the people needed will be in the Bay Area anyway for the KDE/Google event on Jan 18. We propose having a mini developers conference on Jan 19th. Pluto has agreed to sponsor this and provide airfare and accomodations to some key people, as well as pay change fees for some already at the KDE/Google event who want to stay an extra day.

The goal is to get representative people from the four skill sets to brainstorm some of the general concepts and come up with an organizational structure and delegate responsibilities. Also, key developers could be chosen to be sponsored full time.

We would also like to finalize an agreement with Trolltech so that if a committment is made to use QT as the underlying library for this new UI, it can be determined how Trolltech and Pluto will work together on the commercial licensing, and what resources Trolltech is willing to commit to the project.

[blackoper]

wow that's a pretty thought out strategy. Has then been posted to places like the mythtv mailing lists?

[Matthew]

I'm very excited about the prospect of a new & improved UI3. Especially the prospect of a single GUI across all devices and modes (eg. X app and browser). And a themeable (whether stylesheets and/or skins) GUI paradigm that is also tiered properly into layers encapsulating their proper independent/interdependent functions.

My first question about this proposal is just why is Adobe Flash so unsuited to be the platform for UI3? If its proprietary nature is a problem, there's OpenLaszlo. Which also now has the benefit of generating not only both SWF-compatible code and DHTML from the same source project, but is also being aggressively distributed on set-top boxes, PVRs and mobile phones (including iPhone) in a joint venture with Sun, making OpenLaszlo part of JME (so no additional SWF player is needed). JME is also the basis for the BD-J that is part of Blu-Ray (BD menu systems are all in BD-J) and also DVB.

OpenLaszlo can run on all of those devices, and present effectively the same GUI in either standalone apps or a web browser, and is inherently networked for distributed execution. It's already got not only the execution environment (including Adobe's Flash player) but also design/development environments, existing code, and - maybe best of all - an existing developer community.

If OpenLaszlo doesn't quite serve (eg. 3D and maybe alpha-rendering are lacking), its source is open, so it can be revised. If UI3 is going to undertake some serious development, why reinvent the wheel, rather than just extend something like OpenLaszlo that's already mostly there?

Alternately, if OpenLaszlo, like Flash, is just too big and complex, what about SVG? SVG with ECMAScript can make lightweight but featureful GUIs that run in browsers and can be encapsulated in runtimes (Java or native x86/etc binaries). Again, why reinvent the wheel, rather than just build a better car or road for it?

(Note: I have no vested interest in OpenLaszlo or related companies, I just think it's the best fit for these requirements.)

[tschak909]

my only concern is to make sure that the traditional orbiter devices are supported. This includes:

* Cisco 7970
* Nokia 770/N800
* Windows Webpads
* Symbian/Windows Smartphones

because, I do _NOT_ want to lose retargetability to other devices, in the process of moving across platforms. I will scream bloody murder, if that happens.

-Thom

[Matthew]

my only concern is to make sure that the traditional orbiter devices are supported. This includes:

* Cisco 7970
* Nokia 770/N800
* Windows Webpads
* Symbian/Windows Smartphones

because, I do _NOT_ want to lose retargetability to other devices, in the process of moving across platforms. I will scream bloody murder, if that happens.

I agree. I was attracted to LMCE partly because the Orbiter already runs on such a variety of devices. Including the Cisco 7970 - seeing that in the demo video is probably what clinched it for me (along with seeing the XL-1B jukebox supported). OpenLaszlo/SWF doesn't currently run on the 7970 (how about those other devices you mentioned?) But OpenLaszlo is XML, and the 7970 does support XML apps, so maybe porting an OpenLaszlo GUI to 7970 XML is an option. Perhaps even a faster/better option than rewriting a whole new UI3 Orbiter for it from scratch.

[totallymaxed]

I'm very excited about the prospect of a new & improved UI3. Especially the prospect of a single GUI across all devices and modes (eg. X app and browser). And a themeable (whether stylesheets and/or skins) GUI paradigm that is also tiered properly into layers encapsulating their proper independent/interdependent functions.

My first question about this proposal is just why is Adobe Flash so unsuited to be the platform for UI3? If its proprietary nature is a problem, there's OpenLaszlo. Which also now has the benefit of generating not only both SWF-compatible code and DHTML from the same source project, but is also being aggressively distributed on set-top boxes, PVRs and mobile phones (including iPhone) in a joint venture with Sun, making OpenLaszlo part of JME (so no additional SWF player is needed). JME is also the basis for the BD-J that is part of Blu-Ray (BD menu systems are all in BD-J) and also DVB.

OpenLaszlo can run on all of those devices, and present effectively the same GUI in either standalone apps or a web browser, and is inherently networked for distributed execution. It's already got not only the execution environment (including Adobe's Flash player) but also design/development environments, existing code, and - maybe best of all - an existing developer community.

If OpenLaszlo doesn't quite serve (eg. 3D and maybe alpha-rendering are lacking), its source is open, so it can be revised. If UI3 is going to undertake some serious development, why reinvent the wheel, rather than just extend something like OpenLaszlo that's already mostly there?

Alternately, if OpenLaszlo, like Flash, is just too big and complex, what about SVG? SVG with ECMAScript can make lightweight but featureful GUIs that run in browsers and can be encapsulated in runtimes (Java or native x86/etc binaries). Again, why reinvent the wheel, rather than just build a better car or road for it?

(Note: I have no vested interest in OpenLaszlo or related companies, I just think it's the best fit for these requirements.)

I tend to agree with you on the issue of not re-inventing the wheel. But the issue I think in technical terms is that if you want to have effects like transparency in the UI so that UI objects can blend correctly with the live picture from TV or some other video source then Flash (or any other stand alone player) will not know how to do this. The problem therefore is at the Player or runtime engine end of the solution... the runtime engine would need to know how to composite itself with transparency and other effects in the way described... Flash player does not do that and is closed source so we cant make it do that either. As OpenLaszlo relies on the Flash player at the target machine for its runtime engine this causes the problem that Aaron describes.

[craasch]

my only concern is to make sure that the traditional orbiter devices are supported. This includes:

* Cisco 7970
* Nokia 770/N800
* Windows Webpads
* Symbian/Windows Smartphones

because, I do _NOT_ want to lose retargetability to other devices, in the process of moving across platforms. I will scream bloody murder, if that happens.

-Thom

A balance would be good though, to support the existing devices (or a reasonable subset of existing devices) but eventually new features/ desires/ capabilities will simply eclipse the capabilities of what we have now.  I would hate to see future development restricted to support devices that have are out of production.  Don't get me wrong, I have no desire to purchase new equipment, In fact I'm trying to see if it is possible to create a DCE template for my hauppauge MVP running mvpmc software.

I wonder if the discussion could start with looking at the layout description language (probably xml but not absolute) and then choosing the appropriate tools to use for the different devices.  I realize that this could end up fragmenting development, but it doesn't need to.

I like the idea of open laszlo (or similar), and I would also suggest maybe the use of xforms or similar for layout as there are already reference implementations across many platforms and languages.  I think the use of an international standard like this could entice more corporate development / sponsorship as well.  I think the CORE will stay a restricted development environment, but bringing in something like webservices and xml based layouts will allow for some exciting interfaces.  I could easily see adding something like Red5 with laszlo and having game consoles like the PS2/3, Wii, nintendo DS as orbiters, or a number of flashlite 3 enabled devices like the Chumby

-Chris

[Matthew]

I tend to agree with you on the issue of not re-inventing the wheel. But the issue I think in technical terms is that if you want to have effects like transparency in the UI so that UI objects can blend correctly with the live picture from TV or some other video source then Flash (or any other stand alone player) will not know how to do this. The problem therefore is at the Player or runtime engine end of the solution... the runtime engine would need to know how to composite itself with transparency and other effects in the way described... Flash player does not do that and is closed source so we cant make it do that either. As OpenLaszlo relies on the Flash player at the target machine for its runtime engine this causes the problem that Aaron describes.

Flash (SWF) content does not depend exclusively on the Adobe Flash player. There is also GNASH, which is GPL and in active development - and it's not the only alternative. It would most probably be less effort to extend GNASH to use compositing etc than to rewrite from scratch. Aside from the lesser costs (especially time to market), the benefits of using SWF would include the huge library of existing reusable content, including lots of recent GUI components, consistent integration with video like YouTube (and lots of other Net video) players, and probably most important the existing developer community and tools for SWF. Plus, SWF is already an open format (despite Adobe's dominance), not a new format that we'd have to introduce as yet another GUI format standard.

[tkmedia]

http://www.rebol.org/cgi-bin/cgiwrap/rebol/art-display-article.r?article=jgp446wq#section-1.2

Excerpts from above.

In fact, there's absolutely no simpler solution for cross-platform GUI creation, anywhere (you can create complete, functional graphic applications with 1 line of code).

Rebol's multimedia functions are easy to use, and it's built-in graphic tools are powerful enough to handle 3D animation without any external requirements (no OpenGL, DirectX, etc. needed)

Don't know if I am way off base,  but maybe add this to the brain storm.

[totallymaxed]

I tend to agree with you on the issue of not re-inventing the wheel. But the issue I think in technical terms is that if you want to have effects like transparency in the UI so that UI objects can blend correctly with the live picture from TV or some other video source then Flash (or any other stand alone player) will not know how to do this. The problem therefore is at the Player or runtime engine end of the solution... the runtime engine would need to know how to composite itself with transparency and other effects in the way described... Flash player does not do that and is closed source so we cant make it do that either. As OpenLaszlo relies on the Flash player at the target machine for its runtime engine this causes the problem that Aaron describes.

Flash (SWF) content does not depend exclusively on the Adobe Flash player. There is also GNASH, which is GPL and in active development - and it's not the only alternative. It would most probably be less effort to extend GNASH to use compositing etc than to rewrite from scratch. Aside from the lesser costs (especially time to market), the benefits of using SWF would include the huge library of existing reusable content, including lots of recent GUI components, consistent integration with video like YouTube (and lots of other Net video) players, and probably most important the existing developer community and tools for SWF. Plus, SWF is already an open format (despite Adobe's dominance), not a new format that we'd have to introduce as yet another GUI format standard.

Hmmm... well at some point in the future I am sure that Gnash will be a viable runtime for UI3 but as you can easily see from the Gnash site http://www.gnashdev.org/?q=node/25#release this time has not come yet. They have no stable version yet... they are still in alpha mode and they still have significant gaps in the repertoire of content and formats supported let alone the fact that Gnash is not currently supported on  many other OS's. We cant build a strategy based on technology that may or may not meet our requirements at some point in the future.

Sorry the above sounds all negative... but I am just trying to look at this with deliverable working code in a reasonable period of time as my criteria. I have no axe to grind here at all... but we need a stable runtime engine for this to be viable in Flash otherwise we still hit all the issues that Aaron alludes to in his Wiki article.

[colinjones]

I understand the concerns about not loosing the "retargetability" to other (existing working) devices - but isn't that the whole point of the different UIs being selectable by Orbiter? Adding UI3, means that UI2 and UI1 are still available for other device Orbiters - I don't think that anybody was really expecting to have a new UI3 that works with all the cool alpha trans, 3d and animation effects on a mobile phone, or web orbiter, were they? Goodness, UI2 only works with AB on a specific set of nVidia chipsets reliably!

Surely that UI1 and 2 (AB and non-AB) will continue to be available for those orbiters, whilst we don't hamstring the capabilities of UI3, and really go for the "eye-candy"?

I understand the concern about re-inventing - what aaron.b described seems like the ideal world, but sounds like a hell of a lot of coding to get there from the ground up. At the same time, we probably don't want to limit or hack-in something less ideal for the sake of expediency. Does anybody have a feel for how long the plan as described by Aaron would take to implement? Does Aaron still read these forums? Haven't seen much from him recently - you out there aaron.b?

[tschak909]

well the difference also lies in the proposed TOOLS to make such UI's.. none of you have had to deal with the hell that is HADesigner... I have... (excluding Totallymaxed)...and it's the only thing that can do UI1 and UI2 at the moment.

-Thom

[colinjones]

Very true - I assume if it goes the way aaron.b outlines, then one of the design deliverable bullet-points would have to be " * develop descent design tools"!

[Matthew]

Hmmm... well at some point in the future I am sure that Gnash will be a viable runtime for UI3 but as you can easily see from the Gnash site http://www.gnashdev.org/?q=node/25#release this time has not come yet. They have no stable version yet... they are still in alpha mode and they still have significant gaps in the repertoire of content and formats supported let alone the fact that Gnash is not currently supported on  many other OS's. We cant build a strategy based on technology that may or may not meet our requirements at some point in the future.

Sorry the above sounds all negative... but I am just trying to look at this with deliverable working code in a reasonable period of time as my criteria. I have no axe to grind here at all... but we need a stable runtime engine for this to be viable in Flash otherwise we still hit all the issues that Aaron alludes to in his Wiki article.

Compare the effort necessary to take a alpha GNASH (or some other GPL SWF player) to where it needs to be for UI3 (only, not necessarily generic SWF support at first), against the effort necessary to write UI3 either from scratch, or from some other codebase even more rudimentary than Gnash's current incarnation. I'm not saying we shouldn't undertake basic development to get UI3, but just that we should start from the closest point to where we can get the most benefit. Even if TrollTech is going to help, I think we'd all prefer to spend the time just adding their features to something that already does the basics, rather than spending that time just getting to the basics and then working more from there.

Also, looking at the Gnash project, it doesn't seem to be very unstable, another release is slated for the end of this month, and its roadmap looks very straightforward to release. And it's still committed to running on embedded devices well, better than is Adobe's Flash, and uses OpenGL where available (or Cairo where not), which might mean it's "composite ready". Again, it looks like it's exactly in the same vein as the UI3 requirements, but already largely complete, and with a developer community (and large library) attached.

[danielk]

On the developer tools, I've talked with Aaron B (Pluto) and Aaron Seigo (TrollTech KDE) and the plan is very much to have a high end design tool to make theming easier.

As for the underlying toolkit, this would probably be based on libplasma. Plasma is a KDE library initially written to support the next generation desktop for KDE, but which has since been recognized as a portable library with a good design for writing sexy widgets. Depending on the software design chosen it may even be possible to place KDE desktop widgets in the new UI and have those widgets be aware of the 10' UI layout and font size restrictions. Theming would include not just be adjusting the look but also allow control over how the UI elements behave and even allow adding or subtracting applications within that layout without any need for a programmer to be involved.