LinuxMCE Forums

General => Developers => Topic started by: jspeckman on January 18, 2008, 03:13:24 pm

Title: GSD network connection help
Post by: jspeckman on January 18, 2008, 03:13:24 pm
Hi all,
  I'm in the process of rewriting a weather plugin that I had written a while back and need a little help.  The plugin goes out to weather.gov to retrieve the current weather conditions.  Originally I did this by having ruby shell out to a perl script to do the actual work.  The perl script in turn used lynx and grep to get the info, parse it with regex's and the insert the data directly into the database.  I'm now working on redoing the entire thing in ruby.  I wrote a test version as a ruby script (stand alone with no linuxmce functions)  that uses net/http.  This works fine.  Now I'm putting the code into linuxmce and have run into a small snag.  I'm trying to use conn_ to connect to the website, but don't really know how to use it.  I've looked at the panasonic ip camera code, as well as other examples and all of the connect to user defined ip addresses.  Is this the only way?  Can I use additional ruby libraries with the embedded ruby?
  Also, what is the best way to retrieve data from my device.  Here's an example of with I want to do: Weather device retrieves weather conditions once an hour.  If there is any change since last time, an event is fired off.  A respond to event is setup to shut off my outdoor fountain if an event is received from the weather device that it is now raining or there is lightning.  This is working in the old version no problem.  What I want to add is when the timed event to turn the fountain on happens, I would like to check the weather device for if it is currently raining or whatever and not turn the fountain on if the weather is already bad.  Is this possible?

Thanks,
Jason
Title: Re: GSD network connection help
Post by: bulek on January 18, 2008, 04:20:09 pm
Hi all,
  I'm in the process of rewriting a weather plugin that I had written a while back and need a little help.  The plugin goes out to weather.gov to retrieve the current weather conditions.  Originally I did this by having ruby shell out to a perl script to do the actual work.  The perl script in turn used lynx and grep to get the info, parse it with regex's and the insert the data directly into the database.  I'm now working on redoing the entire thing in ruby.  I wrote a test version as a ruby script (stand alone with no linuxmce functions)  that uses net/http.  This works fine.  Now I'm putting the code into linuxmce and have run into a small snag.  I'm trying to use conn_ to connect to the website, but don't really know how to use it.  I've looked at the panasonic ip camera code, as well as other examples and all of the connect to user defined ip addresses.  Is this the only way?  Can I use additional ruby libraries with the embedded ruby?
  Also, what is the best way to retrieve data from my device.  Here's an example of with I want to do: Weather device retrieves weather conditions once an hour.  If there is any change since last time, an event is fired off.  A respond to event is setup to shut off my outdoor fountain if an event is received from the weather device that it is now raining or there is lightning.  This is working in the old version no problem.  What I want to add is when the timed event to turn the fountain on happens, I would like to check the weather device for if it is currently raining or whatever and not turn the fountain on if the weather is already bad.  Is this possible?

Thanks,
Jason

Hi,

maybe not of much help, but :
1. is there any more info about your work posted somewhere ? It's interesting concept also for all kinds of other "data retrieveing " plugins...

2. retrieving data from device: I also run into this problem - for instance, House Security Mode. We have events to change and notify changes, but AFAIK we don't have convenient way to ask for current state, beside looking directly into sql database...

3. To use state info in Event handler, maybe you could create several devices indication state - for instance one generic I/O device for raining, lightning, etc.. and then include it in event handler... Maybe you could have somekind of parent object with all those children - similar as home automation system is handled. For instance I have home automation controller (interface) as parent device and then all devices as child devices (temperature sensors are already there)... So similar could be done for Weather, with children of temperature, humidity, brightness, etc...

But I'm not sure if this approach is best suited with LMCE's intended architecture, so some opinions from more experienced developers would be also helpful here... It would be nice, if you can make Wiki page for it, and we can discuss this further...

HTH,

Bulek.
 
Title: Re: GSD network connection help
Post by: jspeckman on January 18, 2008, 05:15:35 pm
Thanks for the input.  I had given some thought to child devices, but hadn't gone any further with it.  I wanted to ask first.  All my previous work is over in the plutohome wiki.  I'd be happy to start a page over here.  Retrieving data would also be useful for sunrise/sunset as well.  I have need for knowing when say a half hour before sunrise is.  I know currently those times are calculated in one of the plugins (forget which one) and an event fires off at that time, versus being stored for retrieval at any time.

Jason
Title: Re: GSD network connection help
Post by: ddamron on January 18, 2008, 06:22:47 pm
jspeckman,

Sure, bring that over here and start a new thread with it.

I'll be happy to help.

Dan
Title: Re: GSD network connection help
Post by: jspeckman on January 19, 2008, 12:46:44 am
Hi again,
  I answered one of my questions and solved one problem.  You can include libraries and this fixed my connecting to website problem.
Now on to passing data.  Should I post what I have so far here, in a new thread or start a new wiki page for it?  I'm asking 'cause I don't know which is best and where it will benefit the most people.

Thanks,
Jason
Title: Re: GSD network connection help
Post by: ddamron on January 19, 2008, 06:39:56 am
I'd recommend you attach it here first...
I'd rather not put it up on the wiki until it's at least in beta..
Don't embed the code into the message, you'll run out of space..
attach the code as an attachment..

Looking forward to seeing your code!

Regards,

Dan
Title: Re: GSD network connection help
Post by: jspeckman on January 19, 2008, 06:23:06 pm
OK, here's what I've got so far.

Jason
Title: Re: GSD network connection help
Post by: ddamron on January 19, 2008, 10:31:43 pm
  75:#cmd = Command.new(devIDfrom, devIDto, priority, type, id)
  76:#cmd.params_
  77:#SendCommand(cmd)
  78:
  79:ms = "/usr/pluto/bin/MessageSend dcerouter " + device_.devid_.to_s + " -1000 8 0 5 2 4 10"
  80:system(ms)

Is there a reason you're doing it this way?

Title: Re: GSD network connection help
Post by: jspeckman on January 20, 2008, 02:30:10 am
For the shell command to messagesend?  It's the only way I know how at the moment.  I haven't quite figured out how to use the command function yet.  I'm also not entirely sure if the command function supports all the same options as messagesend.  So for now I stuck with what I know to have it working faster.  I would like to do away with as many system calls as possible.

Jason
Title: Re: GSD network connection help
Post by: ddamron on January 20, 2008, 07:38:21 am
Code: [Select]
59:log("Starting NOAA Weather Check")

60:log("Setup message intercept for DCERouter")

61:

62:###

63:# devIDfrom: device_.devid_ = my device id

64:# devIDto: -1000 = dcerouter

65:# priority:

66:# type: 1 = command, 2 = event

67:# id:

68:

69:$devIDfrom = device_.devid_

70:devIDto = -1000

71:priority = ""

72:type = 2

73:id = ""

74:

75:#cmd = Command.new(devIDfrom, devIDto, priority, type, id)

76:#cmd.params_[x] = y

77:#SendCommand(cmd)

78:

79:ms = "/usr/pluto/bin/MessageSend dcerouter " + device_.devid_.to_s + " -1000 8 0 5 2 4 10"

80:system(ms)


Starting on line 63, I assume you're trying to set the params to send the command..
devIDfrom = device_.devid_ This will send a command from YOU (NOT a child device)
(I assume this is correct)

devIDto = -1000 (This is correct for a COMMAND.. if you want to send an event, set it to -1001)

line 66 type, simply set it to 1 for a command, or 2 for an event.

67 id.  this is the ID of the command/event you want to send...

This confuses me a bit...

79:ms = "/usr/pluto/bin/MessageSend dcerouter " + device_.devid_.to_s + " -1000 8 0 5 2 4 10"

MessageSend has this syntax:
MessageSend server [-targetType [device|category|template]] [-r | -o] [-p path] [-bl BroadcastLevel] [-retry value] DeviceFrom DeviceTo MsgType(1=Command, 2=Event) MsgID [parm1id param1value] [parm2id parm2value] ...

Removing all the 'flags' we don't use here, we get:
MessageSend server DeviceFrom DeviceTo MsgType(1=Command, 2=Event) MsgID [parm1id param1value] [parm2id parm2value] ...
Seems to me, the command your sending is missing the MsgType?  8 != 1|2...
is 8 the cmdid you want to send?
If you can explain each variable your sending, that would help.

HTH,

Regards,

Dan
Title: Re: GSD network connection help
Post by: bulek on January 20, 2008, 11:31:15 am
...

This confuses me a bit...

79:ms = "/usr/pluto/bin/MessageSend dcerouter " + device_.devid_.to_s + " -1000 8 0 5 2 4 10"

MessageSend has this syntax:
MessageSend server [-targetType [device|category|template]] [-r | -o] [-p path] [-bl BroadcastLevel] [-retry value] DeviceFrom DeviceTo MsgType(1=Command, 2=Event) MsgID [parm1id param1value] [parm2id parm2value] ...

Removing all the 'flags' we don't use here, we get:
MessageSend server DeviceFrom DeviceTo MsgType(1=Command, 2=Event) MsgID [parm1id param1value] [parm2id parm2value] ...
Seems to me, the command your sending is missing the MsgType?  8 != 1|2...
is 8 the cmdid you want to send?
If you can explain each variable your sending, that would help.

HTH,

Regards,

Dan


Hi,

I guess he's trying to register message interceptor. You can register either by being device (or parent for its child devices) or as a message interceptor.
But I'm not sure if this can be done on same socket (I wanted device and interceptor, but couldn't get it to work on same socket)....

http://wiki.linuxmce.org/index.php/Message_Interceptors (http://wiki.linuxmce.org/index.php/Message_Interceptors)

HTH,

regards,

Bulek.
Title: Re: GSD network connection help
Post by: ddamron on January 20, 2008, 12:33:42 pm
Even then, the message is still malformatted..

weird..

definately need a bit more explanation on the message is attempting to do.

Title: Re: GSD network connection help
Post by: bulek on January 20, 2008, 02:21:34 pm
Even then, the message is still malformatted..

weird..

definately need a bit more explanation on the message is attempting to do.


Hi,

I think you're right... it needs 6 parameters, although formatting of such message is different - there are no parameters ID before parameters, but rather parameters by themselves....

I think that anyone that tries to deal with DCE Router and messages should go through this tutorial :

http://wiki.linuxmce.org/index.php/Plain_Text_DCE_Messages (http://wiki.linuxmce.org/index.php/Plain_Text_DCE_Messages)

It's particularly important cause at step 17. you can see proper DCE message for registering message interceptor...

Sorry, for including this a bit offtopic...

Quote
Matthew, are you reading this (regarding Perl integration) ?
One easy way to start is to register message interceptor - it receives all needed messages and with addition of sending events, commands you can get pretty descent event handler in Perl. I have some dirty mockups of LMCE.pm to help in this area. I agree, it's not proper event handler, cause it's totally "external", but it's a good start. I think we need some skeleton object, that anyone should be able to start contributing on, later we can enhance Perl to be properly embedded... if we organize code properly, also MH users will be able to use it...

Regards,

Bulek.
Title: Re: GSD network connection help
Post by: jspeckman on January 20, 2008, 10:21:18 pm
You guys are correct, it's supposed to set up a message interceptor for climate messages.  Honestly I don't really understand it.  I copied the formating from step 14 in the plain text dce messages tutorial  replacing security event (84) with climate event (10).  However after reading your replies and looking in the database again, it appears that security events should be (9).  Like I said I don't really understand what is going on, but I try what I think works and hopefully you guys who've been writing the more complex devices can help me out.

Thanks,
Jason

PS I did find I had incorrect event numbers in the other 2 messagesend commands.  Weather changed  and temperature changed.  I have corrected those to so far.
Title: Re: GSD network connection help
Post by: jspeckman on January 23, 2008, 11:40:03 pm
Made a couple of changes.  I'm currently testing the code with the register as message interceptor removed to see how things work.  I also added some code to the idle loop to automaticly get weather updates every 15 past the hour.  I'll post the new code once it's been tested a bit.  I've also run into an odd problem that maybe someone can elighten me with an answer or pointer.  The ruby editor in the admin site is deleting parts of my code when I hit the update button.  Half of the xml tags were missing, I added them back in, clicked update and they were gone again???  They're part of the regex and I really need them to stay.  Does the editor know something I don't?  It seems to think it's smarter than me (maybe it is).  Ideas?

Thanks,
Jason
Title: Re: GSD network connection help
Post by: ddamron on January 24, 2008, 02:40:34 pm
Jason,

I ran into the same thing.  I don't know what it is.. but the work around is:
 var = "<" + "tag>"

Be careful, As I found, when trying to LOG stuff... XML get's wiped in the log too..
I found out the hard way while trying to log var.inspect.
it didn't work.. but the code was FINE in IRB.
With regards to the log, putc '<'

HTH

Dan

Title: Re: GSD network connection help
Post by: jspeckman on February 03, 2008, 04:41:39 pm
Hi all,
  I've got a new version of the weather code.  I've had this code running for a while now, just haven't had time to post it.  I was going to post it now, but everytime I try I get a error message saying 'Cannot access attachments upload path'.  I did a search, but no solutions to this were found.  Anyway the auto retrieval code is working, and I fixed the problem I was having with the editor removing my xml tags.  Since it was part of a regex I simply used the tag I am searching for without the first '<'.  The editor likes it and the regex code finds only the data I'm looking for.  Also added some more logging.  I've had it running for a couple days now and it seems well behaved so far.  I'd like to start adding data retrieval now so that other devices (and hopefully events) can get weather data whenever they want it, rather than waiting an hour to see if something changed.

Thanks,
Jason
Title: Re: GSD network connection help
Post by: ddamron on February 04, 2008, 01:37:19 am
Glad to hear that!
Adding data retrieval is almost trivial.. simply create (or use existing) commands and pass data via the command..

Great Work!

Title: Re: GSD network connection help
Post by: bulek on February 05, 2008, 10:44:51 am
Hi all,
  I've got a new version of the weather code.  I've had this code running for a while now, just haven't had time to post it.  I was going to post it now, but everytime I try I get a error message saying 'Cannot access attachments upload path'.  I did a search, but no solutions to this were found.  Anyway the auto retrieval code is working, and I fixed the problem I was having with the editor removing my xml tags.  Since it was part of a regex I simply used the tag I am searching for without the first '<'.  The editor likes it and the regex code finds only the data I'm looking for.  Also added some more logging.  I've had it running for a couple days now and it seems well behaved so far.  I'd like to start adding data retrieval now so that other devices (and hopefully events) can get weather data whenever they want it, rather than waiting an hour to see if something changed.

Thanks,
Jason
Please post it...

Regards,

Bulek.
Title: Re: GSD network connection help
Post by: jspeckman on February 06, 2008, 06:34:40 am
I tried again, but got the same error message 'Cannot access attachments upload path!' when trying to post with an attachment.  Any ideas?

Jason
Title: Re: GSD network connection help
Post by: ddamron on February 06, 2008, 09:17:39 am
Any spaces in the path?
Try copying the code to root, then attach that..
HTH,

Dan
Title: Re: GSD network connection help
Post by: jspeckman on February 07, 2008, 12:25:33 am
No luck.  The forums are giving me the error, so I'm guessing the problem is not on my end.  I did as you suggested though.  I removed all spaces from the file name and moved it to the root of my local drive.  I'm at a loss.  Being over 250 line long I don't think pasting it into a message is the best of ideas.  I'm open to suggestions though as to how to get my code up here.

Thanks,
Jason
Title: Re: GSD network connection help
Post by: tschak909 on February 07, 2008, 03:06:50 am
yup, I had tried to upload a picture attachment, with the same problem.. there is a permissions problem on the attachments path.

-Thom
Title: Re: GSD network connection help
Post by: jspeckman on February 23, 2008, 03:41:56 pm
Ok, Here's the code.  Sorry to post it inline, but uploading attachments is still broke and the original I posted is no longer available.  Have a look and let me know what you all think.

Thanks
Jason


NOAA Weather ruby code - current
Code: [Select]
   0:require 'Ruby_Generic_Serial_Device'
   1:class Command < Ruby_Generic_Serial_Device::RubyCommandWrapper
   2:end
   3:class Device_35 < Ruby_Generic_Serial_Device::RubySerialWrapper
   4:#### 351 ####################################################################
   5:def cmd_351(cmd=nil)
   6:@returnParamArray.clear
   7:### Auto check weather every quarter past the hour per NOAA
   8:t = Time.now
   9:
  10:if t.min == 15
  11:  t -= (60 * 60)
  12:
  13:  if $obs_time =~/(\d\d):\d\d:\d\d/
  14:    obs_hour = $1
  15:  end
  16:
  17:  if t.hour != obs_hour.to_i
  18:    log("Quarter past the hour and weather update needed")
  19:    check_weather()
  20:  end
  21:end
  22:return @returnParamArray
  23:end
  24:#### 355 ####################################################################
  25:def cmd_355(cmd=nil)
  26:@returnParamArray.clear
  27:###
  28:# NOAA Weather check
  29:# 2008 Jason Speckman
  30:###
  31:require 'net/http'
  32:
  33:# Init Vars
  34:
  35:$logFile = File.new("/var/log/pluto/NOAA_Weather.log", "w")
  36:
  37:$DEVICEDATA_ICAO_CONST = 249
  38:$DEVICEDATA_UNITS_CONST = 250
  39:
  40:$units = ""
  41:$identifier = ""
  42:$location = ""
  43:$obs_time = ""
  44:$weather = ""
  45:$temp = ""
  46:$pressure = ""
  47:$dew_point = ""
  48:$heat_index = ""
  49:$windchill = ""
  50:$humidity = ""
  51:$wind = ""
  52:$visibility = ""
  53:
  54:### Check for identifier and units
  55:if device_.devdata_ != nil
  56:  if device_.devdata_[$DEVICEDATA_ICAO_CONST] != nil
  57:    $identifier = device_.devdata_[$DEVICEDATA_ICAO_CONST]
  58:    log("ICAO identifier: "+$identifier)
  59:  else
  60:    log("No ICAO identifier specified")
  61:  end
  62:  if device_.devdata_[$DEVICEDATA_UNITS_CONST] != nil
  63:    $units = device_.devdata_[$DEVICEDATA_UNITS_CONST]
  64:  else
  65:    $units = "F"
  66:  end
  67:end
  68:
  69:log("Units in degrees "+$units)
  70:
  71:if $units == "F"
  72:  $press_units = "in. Hg"
  73:else
  74:  $press_units = "hPa"
  75:end
  76:
  77:
  78:### Tell DCE we are a climate device
  79:log("Starting NOAA Weather Check")
  80:log("Setup message intercept for DCERouter")
  81:
  82:###
  83:# devIDfrom: device_.devid_ = my device id
  84:# devIDto: -1000 = dcerouter
  85:# priority:
  86:# type: 1 = command, 2 = event
  87:# id:
  88:
  89:$devIDfrom = device_.devid_
  90:devIDto = -1000
  91:priority = ""
  92:type = 2
  93:id = ""
  94:
  95:#cmd = Command.new(devIDfrom, devIDto, priority, type, id)
  96:#cmd.params_[x] = y
  97:#SendCommand(cmd)
  98:
  99:#ms = "/usr/pluto/bin/MessageSend dcerouter " + device_.devid_.to_s + " -1000 8 0 5 2 4 10"
 100:#system(ms)
 101:
 102:log("Init complete")
 103:
 104:### Initial weather check
 105:check_weather()
 106:return @returnParamArray
 107:end
 108:#### 356 ####################################################################
 109:def cmd_356(cmd=nil)
 110:@returnParamArray.clear
 111:if ($logFile != nil) then
 112: $logFile.close
 113:end
 114:return @returnParamArray
 115:end
 116:#### PRIVATE METHODS ####################################################################
 117:### Logging function
 118:def log(word)
 119:  $logFile.print( word + "\n" )
 120:  $logFile.flush()
 121:end
 122:
 123:### Weather check function
 124:def check_weather()
 125:  log("Backing up previous data")
 126:  oldweather = $weather
 127:  oldtemp = $temp
 128:  oldpressure = $pressure
 129:  olddew_point = $dew_point
 130:  oldheat_index = $heat_index
 131:  oldwindchill = $windchill
 132:  oldhumidity = $humidity
 133:  oldwind = $wind
 134:  oldvisibility = $visibility
 135:
 136:  log("Checking for weather updates")
 137:
 138:### Get xml file from NOAA
 139:#  xmlPath = "www.weather.gov/data/current_obs/"+$identifier+".xml"
 140:#  h = "GET "+xmlPath+" HTTP/1.0\r\n"
 141:#  log("HTTP Request: #{h}")
 142:#  conn_.Send(h)
 143:#  resv = ""
 144:#  while(true)
 145:#    buff = conn_.Recv(16384, 5000)
 146:#    if buff.length() == 0
 147:#      break
 148:#    end
 149:#    recv = recv + buff
 150:#  end
 151:  h = Net::HTTP.new('www.weather.gov', 80)
 152:  data_loc = "/data/current_obs/"+$identifier+".xml"
 153:  resp, data = h.get(data_loc, nil)
 154:
 155:  if resp.message == "OK"
 156:
 157:#  if recv =~ /^HTTP[^\r\n]+200\sOK.+?\r\n\r\n(.+)$/m
 158:#    data = $1
 159:#    log("Recieved:\n#{data}")
 160: 
 161:    data.each_line {|line|
 162:      if line =~/location>(.*)<\/location>/
 163:        $location = $1
 164:      end
 165:      if line =~/observation_time_rfc822>(.*)<\/observation_time_rfc822>/
 166:        $obs_time = $1
 167:      end
 168:      if line =~/weather>(.*)<\/weather>/
 169:        $weather = $1
 170:      end
 171:      if $units == "F"
 172:        if line =~/temp_f>(.*)<\/temp_f>/
 173:          $temp = $1
 174:        end
 175:        if line =~/pressure_in>(.*)<\/pressure_in>/
 176:          $pressure = $1
 177:        end
 178:        if line =~/dewpoint_f>(.*)<\/dewpoint_f>/
 179:          $dew_point = $1
 180:        end
 181:        if line =~/heat_index_f>(.*)<\/heat_index_f>/
 182:          $heat_index = $1
 183:        end
 184:        if line =~/windchill_f>(.*)<\/windchill_f>/
 185:          $windchill = $1
 186:        end
 187:      else
 188:        if line =~/temp_c>(.*)<\/temp_c>/
 189:          $temp = $1
 190:        end
 191:        if line =~/pressure_mb>(.*)<\/pressure_mb>/
 192:          $pressure = $1
 193:        end
 194:        if line =~/dewpoint_c>(.*)<\/dewpoint_c>/
 195:          $dew_point = $1
 196:        end
 197:        if line =~/heat_index_c>(.*)<\/heat_index_c>/
 198:          $heat_index = $1
 199:        end
 200:        if line =~/windchill_c>(.*)<\/windchill_c>/
 201:          $windchill = $1
 202:        end
 203:      end
 204:      if line =~/relative_humidity>(.*)<\/relative_humidity>/
 205:        $humidity = $1
 206:      end
 207:      if line =~/wind_string>(.*)<\/wind_string>/
 208:        $wind = $1
 209:      end
 210:      if line =~/visibility_mi>(.*)<\/visibility_mi>/
 211:        $visibility = $1
 212:      end
 213:    }
 214:  end
 215:
 216:### Send out events for weather changes
 217:  if $weather != oldweather
 218:    ms = "/usr/pluto/bin/MessageSend dcerouter "+device_.devid_.to_s+" -1001 2 75 13 \""+$weather+"\""
 219:    system(ms)
 220:    log("Message sent: #{ms}")
 221:  end
 222:  if $temp != oldtemp
 223:    ms = "/usr/pluto/bin/MessageSend dcerouter "+device_.devid_.to_s+" -1001 2 25 13 \""+$temp+"\""
 224:    system(ms)
 225:    log("Message sent: #{ms}")
 226:  end
 227:
 228:  check_weather_done()
 229:end
 230:
 231:### Post weather check
 232:def check_weather_done()
 233:  log("Weather data from: "+$obs_time)
 234:  log("Weather check complete")
 235:end
 236:#### 395 ####################################################################
 237:def cmd_395(cmd=nil)
 238:@returnParamArray.clear
 239:check_weather()
 240:return @returnParamArray
 241:end
 242:#### 396 ####################################################################
 243:def cmd_396(failed, cmd=nil)
 244:@returnParamArray.clear
 245:check_weather_done()
 246:return @returnParamArray
 247:end
 248:#### START SETTERS ####################################################################
 249:def initialize()
 250:super
 251:@returnParamArray=Array.new
 252:end
 253:####  END  SETTERS ####################################################################
 254:end

Title: Re: GSD network connection help
Post by: ddamron on February 23, 2008, 09:10:17 pm
jspeckman

Nice!

That is a *great* piece of code.  If I were you, I'd document that in the wiki!
the more 'sample' code we have in the wiki, the more people can see how easy Ruby really is..

Great Work!

Dan