All Developers, please read.

[tschak909]

note: this thread is intended to instigate discussion, and to try to generate a plan of attack.

I am noticing something _very_ disturbing here, that I feel must be dealt with especially given the relatively young age of this fork of the project.

LinuxMCE developers (both present and future) have inherited a very large code base from a developer who took a very straightforward (from their perspective) approach to putting together a device agnostic messaging system and a drawing-independent user interface system for the purpose of home control and the delivery of media.

Notice, I said, from their perspective. This is very important, because for a lot of us, we came here at the viewing of the LinuxMCE demo videos. They _are_ very intoxicating, and do serve to generate a ton of excitement.

And after those of us who do manage to get the system built, and running, a lot of us immediately see something, as developers, that we want to work on...

now what?

we start digging into the system, and find that a great big part of it is still very black-boxed, not in the sense that we can't see it and mess with it, but in the sense that there are a great number of moving parts, all working like a swiss watch, to ensure that this system DOES run like an appliance. And some of us do manage to understand bits of the system and realise just how clever parts of it really are.

Assuming we get past all of this, and we see things we want to improve, what are our options?

We have some reference material on both the LinuxMCE and Plutohome.com wiki sites, which relate to programmer material. However, this material is still very sparse once you actually read through it. It becomes very difficult to get a systems-level overview to figure out which direction you need to go in, to modify, or augment a particular feature. It seems that Plutohome never got around to writing stuff down that their programming staff obviously knew, and this is understandable, they make their money doing custom development, and I am not knocking anything done up to this point.

We need to figure out, as a community, what to do next.

* Can we coordinate with Pluto to get the missing pieces of programming information? if not, we need to do a full audit of the code-base.
* We need to get the missing tools built and accessible, they do exist in SVN, but they need to be brought out in a form that can easily be installed and used. This includes the Designer and the DCEGen tools.
* Skin development/refinement. We need lots of polishing work done on the user interface, although again, information on this process is limited, what are all our options herE?
* Device driver stuff. This one is pretty obvious, and it will get done, but we need to get a library of knowledge established to allow motivated developers to fill in more hardware support. again, obvious, duh, but it does need to be on this list.

I have a fairly complete media-centric set-up (complete with an XL1B jukebox), and will slowly be folding in home automation. But, there are various things I keep noticing, that I want desperately to work on, but I am finding more and more, there are missing tools and pieces which, while they don't stop development for me entirely, do impede it significantly.

Just my thoughts, what do you all think?
-Thom

[Zaerc]

Glad I'm not the only one to notice that.   Eventhough I'm no "LinuxMCE developer", that is the best plan I've heard so far.

From what I have seen so far the architecture is pretty much brilliant, incredibly flexible yet relatively simple, but unfortunately the implementation could use some cleanup here and there.  Apart from that, a few simple bugs still need to be squashed.

You can add user documentation to the list as well, I know nobody wants to do it, but it needs to be done nevertheless.  The way I understand it is that if we can provide proper guidelines and instructions on adding Devices, then anybody could make their own and even "upload" them to a central database for everyone else to use.

[tschak909]

there IS sqlCVS to deal with as well, webpaul1, can we use this? is there infrastructure to support this? if we need to set up a server or something for it, what needs to be done?

-Thom

[bulek]

Hi,

I've tried to transfer my contributions from Pluto system (support for Marantz Audio Receiver,etc...). I've posted to forum how to do it and received no response in quite some time now. I really have no clue what to do next other than waiting...

I'd also contribute a lot, but currently I don't know who will (if anyone at all) maintain and review changes, give some directions for contributions to be uniform and aligned with LMCE philosophy... We need some starting up effort to get more contributors on the line, but no sign of it. I understand that developers are filled with other, maybe more important stuff. But similar thing happened to Pluto also. Promising at the start with prompt help and then we all know what happened.

Regards,

Bulek.

[darrenmason]

Thom,

Excellent post and a good summary of what a lot of the lurking developers here are finding.
Just to add;
* The overall architecture/design of the system is good and very extensible. This includes the centralised messaging router concept and templated device heirarchy.
* The implentation of this design has, in my view, compromised the higher level design. This is not intended to be overly critical, I well understand the time pressures that were placed on the PlutoHome developers. But the device implementations, particularly involving media players/plugins and orbiters, has a high degree of coupling and undocumented dependancies.
* The current project (linuxMCE) has generated plenty of interest, which undoubtedly will include talented developers that have new ideas and are prepared to implement them. At the moment there seems to be some major barriers to this happening. These include lack of documentation to understand the mountains of code that exist and unclear mechanisms on how code can be submitted and absorbed into the project.

The sqlCVS part of the project seemed to be an effective way to have distributed development (including new devices/templates as well as the associated code) and I would like to see it used if possible.
I am keen to know what Paul and some of the others on the contact list for the project think about  these ideas.

I have quite a few ideas and potentially a technical  background that is well suited to helping, however each time I attempt to do something I get overwhelmed by the initial barriers that I mentioned and effectively lose interest. I assume others are probably in the same boat.

Anyway, I hope this thread generates some positive discussion and we can plan a path forward.

Regards
Darren

[dopey]

I think, currently, the main problem is that there isn't any kind of formal forum where developers can discuss things and then come to mutual decision with a plan of action. There has been numerous talks about this and so far the answer has been that the current active developers are too busy programming to deal with these community issues. I certainly understand, and appreciative,  that they are really busy, but this is something that just needs to be addressed, and sooner rather than later.

I too have some things in mind that I think should be done, but am overwhelmed by the fairly complex and sparsely documented code base. Although, given enough time, I'm sure I can figure it all out, but it's rather tedious work that's hard keep an interest in. Also, the lack of community structure is rather concerning to me and has already turned some developers away.

Although we can discuss the "plan of attack" here, really we need the developers that are currently active in the project to be in the discussion and be a big part of the decision making process.

[tschak909]

I have posted this to the mailing list, and am trying....to send mail to Paul directly to try and get him in on this.

Mostly, I'm just trying to get a response, ANY response, from the developer, as to "This is where we are, this is what we're going to do."

-Thom

[webpaul1]

Thom, thanks for emailing me...

I lived and breathed this code for many, many months, and went through just about every module and every class, stepped through all the calls, etc.  Yes, it was DAUNTING, and took many, many, many months to get a handle on what was going on.  The problem I have is that (a) I'm kind of lousy at documenting and don't really know how to document stuff well; kind of a self-taught programmer that lives for coding, but without any formal training on proper documentation techniques, etc.  (b) I'm not the most organized person at setting up teams, etc. (c) I'm *totally* swamped.  I spent such a huge amount of time getting the first release done, and then the second release that I neglected everything else (like my job), and, given the volume of forum posts, emails, etc., have trouble responding...

So, what I think would be in the best interest of the project is to find (1) a volunteer who wants to be the community coordinator, with the mission of recruiting developers, coordinating requests and activities, and kind of 'buffering'.  I could give this person my cell phone number so they can call me anytime.  Since I spent so much time with this code, if somebody wants to know "how do I do this", or "where is this code", or "how do these pieces fit together", I could probably answer those questions in a minute and point someone in the right direction.  Secondly, (2) we really need a volunteer who knows how to write proper documentation that can be the official 'standard bearer' for deciding how the architecture should be documented, coordinating with developers that are doing new code to be sure it's properly documented, and maybe, if anybody is up to the tedious task, coordinating other volunteers who want to go through some key pieces of the existing code and document them.  I think that if there was a somebody who knows how documentation should be done laying out the rules and providing examples, it would be easier for devs like me who don't document stuff to have examples to follow.

I think that have 2 people fill those roles would make a huge difference in the ability to get a thriving development community, and would establish some sort of order so other devs could join the team and not get frustrated.

The big, major task that needs to be undertaken in LinuxMCE is redoing the GUI (Orbiter/Designer/etc.).  You guys read the announcement from KDE that LinuxMCE is being merged into KDE, right?  Well the plan is to completely rewrite the GUI using QT 4, instead of SDL, leveraging a lot of the eye candy type effects that KDE has built in, and create a really powerful design tool that allows the creation of a GUI that is dynamic and highly skinnable.  Then KDE could get their guys who did the Oxygen and Plasma UI work to help with LinuxMCE, and the really good graphic artists/theme artists would have a rich toolset to create beautiful gui's.  The goal is that the GUI would be flexible enough to do neat effects like having the highlighted options in the main menu 'zoom' and 'explode', like the bottom icon bar in Mac OS X, but without hardcoding all that behavior.  Right now the behavior is very much hard-coded into Orbiter, so it would be difficult to create a new theme that does much more than change button images/placement, without re-writing a ton of code.  I've talked to the Pluto guys and they have an invested interest too in seeing this done because their commercial clients like Monster all want their own distinct GUI's that look nothing like what the other Pluto-based solutions are doing.  They can't currently do this with Orbiter.  It's my understanding that Pluto and perhaps even a few others are considering sponsoring developers to work on this new GUI, since it would be such a huge task and would go much faster with full time devs instead of purely part time volunteers.  I know there's discussions going on right now with KDE developers, Trolltech (the QT guys), and Pluto about how to get this going.  It's going to be a big task because afaik nobody else really has such a dynamic UI that is theme-able.  MythTV has a very theme-able UI that you don't need to write code for, but it doesn't have eye candy, 3d effects, etc.  Then you've got UI's like Windows MCE which have cool effects, but they're hardcoded and not theme-able.  So it seems like this is going to be pretty special if we get it done right.

If we get somebody to do the 'community coordinating' role, maybe that person could try to find a good dev who knows UI design well and could architect such a UI, and I can work on finding that person a sponsor so they can put all their efforts into it.

[Zaerc]

I would like to nominate Thom for community coordinator, as he already got the ball rolling on this, and he seems the most capable in my opinion. 

Developing a new UI would be awesome, I just hope it won't go at the expense of the already existing ones. 

[tschak909]

Building an eye-candy GUI is all well and good. Visual presentation _is_ important, and given the code-base and the current limitations of the Orbiter, this IS _a_ top priority.

With that said...

I do believe that a great amount of detail needs to be spent on making sure that the existing functionality (and I am talking from the user perspective), works, and works well, remember...this is an appliance, it needs to work as such. I'm not talking about INSTALLATION, i'm talking about AFTER the installation hurdle has been crossed and you're staring at a working orbiter, like making sure that:

* if we go into KDE, Arts doesn't clobber the soundcard, so the orbiter phone will still function.
* messaging breakdown when using a Sony VAIO jukebox (i've literally had the entire DCE router restart even when a rip was in progress, which TOTALLY causes even more bizarre behaviour with the jukebox as you continue using it.)
* button interaction, when I press a button, I want to know i pressed the button, even a simple color change, a blip sound, something.
* making sure all remote types can access all of the exposed UI elements. (Windows MCE remotes currently can't deal with the gesture rose for things like changing channel with the rotating guide,.. the MCE arrow keys are completely useless.)
* making sure auto-detected networked storage devices work

What about sqlCVS?

What about altering remote assistance for our needs?

What about the missing tools so that we can get working NOW?

The important thing to note here is that, yes, shiny interfaces good.. but it's also about the protocols and interfaces, making them cleanly understood.... I for one would like to see a Motorola RAZR (J2ME) orbiter, don't think we can port the Qt4 one to that...

Paul, you say there are discussions started with KDE, Pluto, and Trolltech, is there a public forum so that we can be in the loop? Bring them here, or can we go there?

(yes, I _am_ being a persistent, tenacious, and arrogant bastard. I've seen far too many good projects go up in flames because the developers didn't make themselves communicate on a regular basis.)

-Thom

[nite_man]

I came to LMCE from Pluto. I work with Pluto about one year. I do an installation for the big house. Unfortunately, this is not my main job (I work part-time). Actually, I dig just an SDL Orbiter functionality to bring it to Nokia770 (finally, it's done thanks to Chris Miron). So, I cannot say is the code is good or bad. As I see the system was designed by proper way. But from the point of view of system user I worry about many thing related with its functionality. For example, DVD playback on the wide screen. From the developers point of view it's not so important bug. But my customer asks me why he cannot watch DVD on his 50" TV. Why I spent about one week to install a new MD on his Pluto system? Because the SMP kernel was not tested well. As result it was not possible to activate SPDIF. And there are many small problem and bugs which do process of installation and configuration extremely  hard. Maybe it isn't so critical if you build the system for you or your friends. But I suspect that LMCE team is interested to see its system on the commercial area as well (but as I see it's impossible to use it for consumers).

Another thing which is scary for me is missing direction. I don't see the road map and plans (maybe I missed something).  I knew about a partnership between KDE 4 and LMCE from the KDE blog not from the LMCE site (or blog which can be very useful for the community and for developers also). I really don't know what I can expect from Pluto and from LMCE as well. And it's scared me more than bad code-style or not well organized project tree.
 

[Zaerc]

If I read Paul's words right (and considering the way things are going) then basicly we are on our own.  Now that may sound a bit harsh, but here is how I look at it:

He has given us a shrink-wrapped bitesize version of the Plutohome system complete with easy installer and wizard that is sufficient to get the basics going for most people, plus provided us with all sorts of tools like this forum and the wiki for instance.  It seems pretty obvious that he can't do everything by himself, and he shouldn't have to.  Instead of waiting for him to do it all for us, I think we should try leaning on him as little as possible, in the hope he will find the time to give us a little push here and there where it is needed most. 

Therefore we, as a community, should focus on picking up as much of the issues as we possibly can.  We are already fixing our own little corners of the puzzle anyway.  It would be nice if we could synchronize our puzzles so we can focus on the few missing and/or broken pieces.  SqlCVS and SVN could be used for that, and we will need Paul's assistence if we want to do it properly.

So what's in it for me, you might ask, well here is how it works.  When somebody figures something out and documents it, the next persons get a free ride which means that they can verify and improve the instructions and don't have to waste time googling around for answers.  In turn they can use that time to figure other stuff out, maybe not on the directly the stuff that the first person is interested in but in the long run everybody wins.  There is really no need for everybody to keep running into the same basic setup questions over and over again, please help yourself by helping others, it does pay off.  I suspect that the Pluto community never reached it's critical mass simply because everybody was left to fend for themselves, let's not repeat that same mistake.

The most important thing to do (in my opinion at least) is to get some sort of development cycle set up.  There are a lot of people (like Bulek pointed out) with small fixes and additions who currently have nowhere to go.  I'm sorry to say it but in an open-source community that is completely unacceptable.  We really must find a way to submit and test changes/updates or we are never going to be able to support everybody's hardware.  Nightly builds would be ideal but seem far off for now.

Another thing is that we are not going to resolve all these issues overnight.  We need to realize that we all have different priorities and goals (not to mention hardware).  I consider this a huge advantage as long as we work together and share our experience.  We have more then enough talent around to pull this off and get our systems working the way we want them to, not just for ourselves but for everybody that is willing and able to put in a bit of their own timeand effort.  In the end all we commonly want is the most versatile home system money can't buy.  On the downside it will be hard to formulate clear goals and roadmaps, since we all want to go in different directions, I agree with Michael that therein lies a big challenge that is important for the business side. 

I hope I didn't offend anyone as I don't intend to be disrespectful, please correct me if I'm wrong and sorry for the long rant. 

[webpaul1]

Zaerc,

That's not at all what I was saying.  For example, today, I got word from the DVD replicator that was supposed to be delivering the LinuxMCE mastered DVD's today (there are already almost 1,000 orders) that there's a problem with the dual layer dvd-9's layer 0 & 1 overlapping and they need the .iso split and all sorts of stuff like that which was all totally new to me, and I ended up blowing off a very important meeting because I had to spend the whole day buying new, high-end dvd-DL's, a new type of dvd burner, installing/fighting tons of software issues, etc., to fedex off a new master by 5pm since so many people are waiting.  And when the discs come in on Monday, I'm going to have address & mail 1,000 dvd's.  This in addition to fixing lots of mantis's (note how many I'm working on at any given time), etc.  And so on.  I'm still working on LinuxMCE all the time constantly, including every single weekend.  I didn't put this much effort into this for the past 9 months to 'abandon' it.

What I was saying is that I'm at 100% capacity right now, and I keep getting pressured that I need to be doing more (like writing docs, organizing development efforts, etc.), but I can only do so much and that's why things are moving so slowly.  Therefore, for the best interest of the project, there should be a team / community that works together.  And I will continue to devote 100% to it like I have since last year.

I get lots of emails each day related to LinuxMCE.  Many are asking for new features and making business offers and stuff like that, and I just don't have the time to read let alone respond to all of them, and if there are emails from developers offering to help or submit bug fixes (which is badly needed), I may miss them. 

That's why I think that there should be a 'community coordinator', who doesn't really need to be a coder and understand coding, but just someone who's organized and can take the lead in handling the incoming requests and filtering out what's important and what's not.  I'm not trying to 'abandon' LinuxMCE, just ask for help since this I realize that I can't keep up with everything.

[webpaul1]

RE: SqlCVS, SVN, remote assistance, submitting new code, daily builds, etc....

When I first did release 1 I tried to branch from Pluto completely and sever all ties.  Unfortunately, just managing the build server, sqlcvs server, svn, etc., was a huge (and expensive) task.  Pluto was very supportive of LinuxMCE and offered to help and let me use their resources.  So I sort of abandoned my own svn and sqlcvs and went back to using theirs.  There are build servers that make new builds every hour, and full releases each night.  There have been some users that have submitted patches by emailing me.  These *have* been merged into the svn head, and I am regularly downloading the nightly builds to be sure nothing broke.  The problem with making this all public is that the nightly releases are 4gb (it's a new quick start dvd each night).  And it's not my server so it wouldn't be my place to give out the URL and take down their servers with constant hits from people downloading the nightly releases.

For pluto it's not a problem because they all work in an office and they have testers that install the daily builds each day as far as I know.  And me being one guy who remotely downloads the latest dvd once every couple days isn't going to be a problem.  However we do need to find a way to get the latest versions out to the community, not just the official releases.  I'm open to suggestions, since I don't have a clear cut answer.  There are other projects, like Mono, that are also huge and have tons of binaries.  I'm not sure if there are ways to distribute patches or diff's to the binaries?  One possibility is to streamline the build process so it's easier for each dev to build his own.  While that's good, the fact that it takes 5 hours to compile/build all the code makes it unlikely a developer would want to do it on his own.  What I find useful myself is to take just the contents of the bin & lib directories from each nightly build and then update the database with sqlcvs.  But this doesn't really bring the system current...  Anybody have more ideas on this?

For now, it's still using pluto's server for sqlcvs.  I do want to get our own, it's just a matter of cost.  It's expensive to setup another server, so I'm just using theirs for now.  As far as I know, the way it's supposed to work (with Pluto at least) is that every user who learns new i/r codes or gsd codes, that stuff is automatically sent to their sqlcvs server each night and put into a 'quarantine' until it's approved.  i haven't had a chance yet to see if this mechanism works in LinuxMCE.  So it's possible the i/r & gsd stuff is getting sent to pluto and they have tons of records in quarantine waiting for approval.  It's high on my priority list to figure out if this mechanism still works with LinuxMCE, and if so, to get our own sqlCVS server, or work out some way to get linuxmce community guys permissions to use pluto's existing one to do approvals on quarantined records.

[dopey]

Paul,

The way I read Zaerc's post he was saying the same thing you were; you are already doing a whole lot and we are asking too much of you. I also agree with that, but it doesn't take away the frustration.

It seems pretty obvious that he can't do everything by himself, and he shouldn't have to.

I'm encouraged simply by the fact that we are having this discussion. There are a lot of issues where no one has a clear answer to and we just have to work through them together.

On the note of distribution, we could continue to use bittorrent. The only change would be that we would need nightly tar balls of the code. Then someone else could build the DVD and release it through bittorrent or the devs can just build their own. Of course this is dependant on streamlining the build process (which I think should be done anyhow)... A 5 hour build time actually wouldn't bother me too much. But then again I'm a Gentoo user so I'm used to it and have a few of distcc slaves that'll make it build faster. Also, is that 5 hour build time including a large ccache?

Bittorrent certainly isn't the ideal solution. I think everyone here would prefer SVN, but we should not expect you to pay for it or even maintain it. I don't have any experience with that so I really wouldn't know how much it costs in time and money, so I'll leave this part of the discussion to someone more knowledgeable in this area.

I think using Pluto's SqlCVS is a realistic and mutually beneficial approach, assuming they agree to it. This would allow both Pluto and LinuxMCE to benefit from contributions from both projects. I think the main problem would lie in the approving of the quarantined item, like you said. It's doubtful that Pluto would want people from LinuxMCE to approve things for their system. We would most likely need a mechanism so you can approve a quarantined item for only one project. Perhaps we can read the quarantined items and then "approve" it for us by copying it to a local database? Just a thought...

I also would like to nominate Thom to be the community coordinator. Thom, would you even be interested doing that?