Pluto -> Java interface

[archived]

Dan asked about a Java DCE interface, we've use email... Copying to a forum thread for other ideas:

>> Can you guys send me a few different binary files containing sample DCE messages for analysis and testing of the Java integration side?   I'm unfamiliar with the C++ project stuff so I was hoping to get a jump on it without having to try and build/run Pluto locally from source code.

DCE is pretty simple.  First, the basic wrappers are in plain text so it is very easy to follow in the logs.  There are a few special text messages, such as RELOAD, etc., but the only one that really matters for sending messages is MESSAGE.  Basically you just send MESSAGE xxx

Where xxx is the size in bytes of the binary data that will follow.  The binary data it contains the actual message.  All messages contain a from device and a to device, type and an ID.  The most common type is 1, which means command.  In that case the ID is the command's number.  The second most common type is 2, which is the event, and the ID the events number.  Each message also has an optional number of parameters.

 

What the commands and events are is defined in the database.  In your Pluto admin web site, if you click advanced, DCE, you can see a list of all the commands and events and the parameters they take.  Our utility DCEGen uses this database to build a pregenerated C++ program that encapsulates everything.  So for example the command “MH Move Media” is ID 241.  It expects 2 parameters: “PK_EntertainArea” is parameter ID 45, and “StreamID” is parameter ID 41.

 

Media_Plugin is a DCEDevice that implements that command.  So DCEGen builds a base class for media plug-in that listens for incoming messages on a socket, automatically parses them and deserializes them, and then passes them to a switch block which will include a case 241, that will then extract the parameters, and call a virtual function called: “MH_Move_Media(string sPK_EntertainArea,int iStreamID)”.

 

So for C++ it's really simple, the programmer has nothing to do.  It he wants to send the command he just types in DCE:: and then auto complete gives them a list of all the commands, and when he chooses the move media command, auto complete gives all the parameters, and when he calls SendCommand(), the command goes through and in media plug-in the MH_Move_Media() function gets called automatically.

 

To implement the entire functionality within java including the whole framework, the class generator and everything would be a pretty big task.  however if all you want to be able to do is send a command or an event, you simply can open a sockets to DCE router and send MESSAGE xxx followed by the message in the same binary format.  I attached a sample file containing the move media command going from device 123 to device for 83, with the stream ID of 382 and an entertainment area of “abc”.  If you cat that to a socket the message goes through.  Now the question is how to do this in Java.  I included the message header and CPP file.  Do you know C++?  

 

There is another way of doing this. It is to make a C++ to Java gateway where all the guts of the code is still in C++ and uses the exact same libraries, but the user can do the implementations in Java.  This is how we encapsulated Ruby.  In this case there is very little code in Ruby, Ruby does not form messages, it does not open sockets, it does essentially nothing.  All of that is done in a normal C++ DCE device.  The C++ device parses the messages and then calls a Ruby function with the same name.  This is how our C++ GSD device works.  It would create a Ruby function MH_Move_Media that was an empty stub for the user to fill in.  Virtually the only code in Ruby is what the user puts in to implement that function.  The advantage is that it works faster, because C++ usually runs faster than the high-level languages, and more importantly, all our existing C++ libraries are still used so we do not have two sets of code to maintain.  If we wanted to change the message format for example, we would just change the C++ message function, and would not need to change any Ruby code.

 

Does Java have the same type of C++ <-> Java interaction?

[archived]

Dan replied:

Thanks for the very detailed response, Aaron.  I greatly appreciate this information.  In response to your questions:
Do you know C++?

Yes I do.  It's been a while since I've done C++ development on a PC, though.  I've installed MS Visual Studio 2005 Beta 2.  Since I primarily do Java these days, I cringe every time I start that app up :-)
Does Java have the same type of C++ <-> Java interaction?

Yes, we could certainly wrap a JNI (Java Native Interface) class around it.  However, I dislike this option as it is tied to native code and does not permit optimizations which could be made for Java folks.

One huge question on my mind is the BINARY protocol.  I notice that you guys are using sizeof() to determine jump indexes.  I would assume -- and I may be wrong -- that this would cause a binary protocol failure when sending messages between two machines that have different processors or had their code compiled with different flags.

The text protocol is easy.  It's the binary one I need to look at because of the sizeof() issue (Java does not have sizeof()).  Strings are also different, but that's not a difficult thing to map.

[archived]

>> I notice that you guys are using sizeof() to determine jump indexes

That is correct, however our code does already run on windows, Windows CE, Linux, Symbian and Microsoft Smart phone.  So it is pretty cross-platform, although perhaps less cross-language.  Up to this point nobody has made their own extra devices that speak DCE.  So that one C++ class is presently the only place where the code exists.  Therefore it is possible to make a change that will not break anything because our upgrade system ensures that the same version of the devices and router are running and everything uses the same class.

The reason for using native C++ (ie sizeof()) is that although it lacks portability, it is much more efficient.  Since the router is sometimes processing hundreds of messages per second, in addition to streaming 10 different movies throughout the house, and handling simultaneously phone calls and videoconferencing, we wanted the leanest fastest solution.  When we get our performance tests this was orders of magnitude more efficient than using a portable protocol like XML.  However the other possibility is that we add another type of message: MESSAGES_XML…  This way you could use that structure for Java, and we have some Perl programmers wanting to do the same thing.  It would be very trivial on our part to permit this-- probably two days worth of work or something to incorporate an XML parser that would convert between XML and our internal C++ message class.  Then when a device registers it would report whether it wants messages in XML or binary.  The massive messages, such as all the media players reporting their time code every second, doorbells, motion sensors, and all that stuff would still use the existing binary, and only the occasional message to high-level devices written in java or Perl would use this.

Our goal with Pluto is not so much to offer or promote our protocol-- lots of protocols exist, such as xAP, UPnP, etc.  Pluto's goal is to be a solution.  Customers buy solutions and benefits, not protocols.  So we came up with DCE only because it was very efficient and easily maintainable.  However we have always anticipated and expanding to support the existing protocols.  So another possibility, is that rather than DCE_XML, we integrate something like xAP.  I am not very familiar with it, but I assume it has the same basic concept of encapsulating commands and events, and therefore perhaps we could write a xAP module.

What do you think?

[archived]

Actually, I wasn't thinking of XML when I asked about sizeof().  I think the efficient way to do it is to specify the byte length for the various types you use and stick to it.

So you could say, for example, that instead of using sizeof(long) for a variable, you could use 4.

Then you could write utilities to ensure that an int, long, etc., are written out as 4 bytes for whatever variable you're trying to send.

This explicitly specifies the protocol in terms of byte sizes and allows each platform to make its own decisions how to best store the domain.  sizeof() can be eliminated, making it truly portable.

Sure, XML is another way, but it's not the best choice whatsoeever for something like DCERouter.

Your thoughts?

[archived]

BTW, for anyone that is interested I have started the JPluto project at SourceForge.net.

You can check it out (not much to see yet) here:

http://sourceforge.net/projects/jpluto

Dan

[archived]

Our goal with Pluto is not so much to offer or promote our protocol-- lots of protocols exist, such as xAP, UPnP, etc.  Pluto's goal is to be a solution.  Customers buy solutions and benefits, not protocols.  So we came up with DCE only because it was very efficient and easily maintainable.  However we have always anticipated and expanding to support the existing protocols.  So another possibility, is that rather than DCE_XML, we integrate something like xAP.  I am not very familiar with it, but I assume it has the same basic concept of encapsulating commands and events, and therefore perhaps we could write a xAP module.

What do you think?

While I am not familiar with the inner workings of xAP, I know that it is well-regarded as a general IP network protocol for this kind of usage.  I believe xAP as a replacement to DCE would be A Very Good Idea to look into.

As good as DCE probably is, it is always better to have a standards-based protocol which the community at large helps develop and prove out.

I like the idea.

Dan

[archived]

You might also look at xPL, which is very similar.  Both xPL and xAP have a good following.  I'm not clear on what the difference is yet.

Looking on the net I see many instances of people saying they are supporting both xAP and xPL.

Dan

[archived]

I just made a quick change...  Now you send/receive messages in plain text.  When you register your command handler, send "PLAINTEXT" and all your messages will come in plain text.  With the .30 release I'll include some documentation with a sample you can use with just telnet and some messages to copy/paste showing how to send/receive messages.

I should have done this from the beginning...  There was already a Message constructor to build a message from plain text, the only part missing was a method to convert the message back into text.  It took half an hour, and now the Perl/Java programmers can write a DCE Device without any problems.

I also sent Paul from xAP an email and will look into adding a xAP gateway to Pluto as well.

[archived]

I uploaded a tutorial explaining how the plain text messages for perl/java works, and tested it today against our daily builds.  If anybody has comments please let me know so I can make changes before the .30 release and before people start using it:

http://plutohome.com/support/index.php?section=document&docID=286

[archived]

Thanks, Aaron.

However, there is really no reason why Java, Perl, or any other general I/O-capable language should not be able to create your binary stream.

In fact, JPluto currently has the capability to write a binary DCE message.  Perl should be able to easily do this as well.

The "C++ structures" you refer to are nothing more than bytes.  A c-string is a 0-byte terminated character string, your longs are 4 bytes, etc.

I'm not concerned.  I would prefer, obviously, an xPL or xAP protocol instead, but not because it's text-based but rather because it's standardized and more widely supported.

Dan

[archived]

Thanks, Aaron.
However, there is really no reason why Java, Perl, or any other general I/O-capable language should not be able to create your binary stream.

In fact, JPluto currently has the capability to write a binary DCE message.  Perl should be able to easily do this as well.

Awesome wrt JPluto, A while ago I'd spent some time looking at a port to Java also, with considerations for J2ME and something like SuperWaba.  Then work went nuts.

The "C++ structures" you refer to are nothing more than bytes.  A c-string is a 0-byte terminated character string, your longs are 4 bytes, etc.

I was concerned that when Pluto was moved between compiler/platform then the DCE form might change.  

My thought was that maybe something like YaML http://www.yaml.org would be better than a language-dependant encoding method.   Just thinking about that now, since YAML has JavaScript support, that might suggest a DHTML or  AJAX solution would be an option. (I.e. an in-the-browser-orbiter).

I'm not concerned.  I would prefer, obviously, an xPL or xAP protocol instead, but not because it's text-based but rather because it's standardized and more widely supported.

Dan

I'll have to look at those protocols, I'm not familiar with them.