LinuxMCE Forums

General => Feature requests & roadmap => Topic started by: brononius on August 29, 2011, 08:19:46 am

Title: Monitoring
Post by: brononius on August 29, 2011, 08:19:46 am
Maybe a new feature in linuxmce is a monitoring tool?

I don't know how (if?) you guys check today the bandwidth, disk space, services...?
Personally i use cacti a lot, i'm a network engineer, therefor i use it very much. It's easy to install (through apt-get install cacti), easy to maintain (webinterface / snmp based), nice to see (cool graphs, up/down status).

This way, we could easily manage/troubleshoot/alert some extra stuff:

And of course, when you've got other devices in your network that are snmp capable (aren't they all these days?), you can add them as well.
I'm thinking about a switch, a modem/router, normal pc's, ip phones, home automation links, power meters, water meters...


One of these days, i'll install cacti on my linuxmce server (I first need to take of backup of it  ::) ).
And after a couple of day, i'll see to extract some graphs as an example.
Title: Re: Monitoring
Post by: tschak909 on August 29, 2011, 05:21:00 pm
We would love more integration like this. As always, we're accepting patches for new features, etc. And we can help you in integrating this if you're willing and able to put in the elbow grease. :)

-Thom
Title: Re: Monitoring
Post by: brononius on August 29, 2011, 05:52:44 pm
No problem to help, but to be honest, i'm a network guy with some linux experience and web knowledge.
So don't expect real programming of me. I would like to able to do it. But you can't do everything in your life. ;)

If you can point me on the way how/what/where i should do, i can have a look.
Title: Re: Monitoring
Post by: cfernandes on August 29, 2011, 07:32:48 pm
Hi brononi,

  i have some Knowledge and free time   to help .


 
Title: Re: Monitoring
Post by: brononius on August 30, 2011, 08:32:56 am
Great.
What can/must we do in order to get this up and running?

I see 4 major parts:

Title: Re: Monitoring
Post by: dextaslab on September 03, 2011, 04:20:07 am
Rather than running cacti, I put forward using plain old mrtg as impementing scripts will be easier and lighter, as well as making changes/updates;

example;
-- create core mrtg.cfg --
do for each hdd on core (or mount points) from database - insert basic template filling details.
do for each NIC on core in database - insert basic template filling details.
Insert standard free memory template
Insert standard process template
etc..

--Create MD mrtg.cfg's --
do for each MD in database
   do for each hdd in current MD - insert basic template filling details.
   do for each NIC in current MD - insert basic template filling details.
   Insert standard free memory template
   Insert standard process template
   etc..
loop
 
*Advantages:
 As long as mrtg is installed on all MD's the script just has to create and place/replace the mrtg.cfg on each MD (/usr/pluto/diskless/<md#>/etc/mrtg.cfg
 You just have to create an appropriate place on the core that is mounted on all MD's to store the image and html files; /home/public/data/mrtg ?
Title: Re: Monitoring
Post by: tschak909 on September 03, 2011, 04:22:38 am
I'm of the opinion that talk only goes so far.

Build a prototype, then figure out what you need to make that prototype into production.

Working models are currency. ;)

-Thom
Title: Re: Monitoring
Post by: brononius on September 03, 2011, 07:56:04 am
Rather than running cacti, I put forward using plain old mrtg as impementing scripts will be easier and lighter, as well as making changes/updates;

I don't agree, cacti is installed in about 2 minutes. Without the need of knowledge for any scripts, and cacti does a lot more then mrtg (services, tresholds, syslogging, weathermaps...). The user interface is so great, that you don't want anything else. When i explain my colleagues/customer cacti in 15 minutes, they're off with it, without any linux knowledge! And resources? If you start with 'own script's, at the end, your resources will be must higher then with cacti.

For me? When you want that people use 'your' system, you need to give them something that's simple.
Once they need to start changing config files, they are gone/lost. That's the only reason why windows gains above linux for the normal computer user.  :-X


But of course,everybody his choice. ;)
Title: Re: Monitoring
Post by: brononius on September 03, 2011, 08:00:08 am
I'm of the opinion that talk only goes so far.

Build a prototype, then figure out what you need to make that prototype into production.

Working models are currency. ;)

-Thom


My cacti is already running, as a standalone version next to linuxmce. To start, i only monitor some things of the server (processes, diskspace, load & memory), and my switch (bandwidth of all interfaces, cpu)
So not only talking!  ;)

My major question (=problem) is how must i present it? Some printscreens?
Like i wrote before, i'm not a developper/programmer, just a network engineer. So i don't have any clue how i can help you guys to put this into the global project.
I can start trying to change the files like i want (fe change the webmin pages), but with the next release, all my changes can/will be lost. No? And i'm not sure how the dhcp stuff can be added in here. Since with any monitoring tool, you'll need to know the exact ip. Therefor i always use mac-based dhcp...

The installation procedure can already be found on my blog:
http://www.oniria.be/pri/index.php/linux/233-linuxmce#mon

I need to add the plugin module, that something for next week.
Then i can start adding all my devices in my network (power meter, workstations, router, NAS, wireless access point, phones...).

Let me know what i can else do to help...
Title: Re: Monitoring
Post by: dextaslab on September 03, 2011, 06:42:18 pm
I don't agree, cacti is installed in about 2 minutes. Without the need of knowledge for any scripts, and cacti does a lot more then mrtg (services, tresholds, syslogging, weathermaps...). The user interface is so great, that you don't want anything else. When i explain my colleagues/customer cacti in 15 minutes, they're off with it, without any linux knowledge! And resources? If you start with 'own script's, at the end, your resources will be must higher then with cacti.

For me? When you want that people use 'your' system, you need to give them something that's simple.
Once they need to start changing config files, they are gone/lost. That's the only reason why windows gains above linux for the normal computer user.  :-X


But of course,everybody his choice. ;)


I think maybe we're not quite talking about the same thing... The scripts are written to be generic as a '1 shoe fits all' once deployed they don't even have to know they are there.

This is a proof of concept to auto create a MRTG config file http://www.modlog.net/temp/MRTG_config.sh (http://www.modlog.net/temp/MRTG_config.sh)
example output:

# /usr/pluto/bin/MRTG_config.sh
WorkDir: /var/www/mrtg
LoadMIBs: /usr/share/snmp/mibs/UCD-SNMP-MIB.txt,/usr/share/snmp/mibs/TCP-MIB.txt
EnableIPv6: no

#-----------NEW DEVICE--------------
Target[HDD.174]: `/etc/mrtg/disk_usage.sh` /dev/sdb
MaxBytes[HDD.174]: 100
Options[HDD.174]: gauge,nopercent,pngdate
Title[HDD.174]: Disk usage for /dev/sdb
PageTop[HDD.174]: <h1>Disk usage for /dev/sdb</h1>
YLegend[HDD.174]: Percent
ShortLegend[HDD.174]: %
Legend1[HDD.174]: Total Space
Legend2[HDD.174]: Used Percentage
LegendI[HDD.174]: Total
LegendO[HDD.174]: Free

#-----------NEW DEVICE--------------
Target[HDD.173]: `/etc/mrtg/disk_usage.sh` /dev/md0
MaxBytes[HDD.173]: 100
Options[HDD.173]: gauge,nopercent,pngdate
Title[HDD.173]: Disk usage for Software Raid 5
PageTop[HDD.173]: <h1>Disk usage for Software Raid 5</h1>
YLegend[HDD.173]: Percent
ShortLegend[HDD.173]: %
Legend1[HDD.173]: Total Space
Legend2[HDD.173]: Used Percentage
LegendI[HDD.173]: Total
LegendO[HDD.173]: Free

Title: Re: Monitoring
Post by: dextaslab on September 04, 2011, 10:44:34 am
I have spent some more time on it, again, it's dirty bash coding, but makes it show it's very possible (and half done), it can also integrate into the webadmin to show for example disk usage graphs while selecting a raid drive, or CPU usage graph when an MD is selected, etc..

Will put more time into it later next week, and put it forward to see if anyone thinks it's any good or worth continuing.
Title: Re: Monitoring
Post by: dextaslab on September 04, 2011, 02:39:02 pm
After editing the bash script the only thing left to do is either manually install mrtg:
# apt-get install mrtg

edited " /var/www/lmce-admin/operations/myDevices/editDeviceParams.php"

...
  </script>

        <fieldset>
+            <a href="./mrtg/'.$deviceID.'/index.html">Graphs If Available</a>   
        <legend>'.$TEXT_DEVICE_INFO_CONST.' #'.$deviceID.'</legend>
        <table border="0">
...

(I couldnt get the test to see if the file exists so it'll show on all devices not just MD's)
http://modlog.net/temp/monitoring.png (http://modlog.net/temp/monitoring.png)

Only thing left is to integrate it into the core scripts that are used when creating MD's and clean it up, maybe add more graphs, etc.. Any one got any interest in it?
Title: Re: Monitoring
Post by: cfernandes on September 04, 2011, 04:44:00 pm
I think just make a graphic would not be enough, it is necessary to send alerts when something is wrong.
Title: Re: Monitoring
Post by: dextaslab on September 05, 2011, 12:45:04 am
You can set thresholds with mrtg; example:

ThreshDir: /path/to/mrtg/thresh
ThreshMaxI[performance]: 1
ThreshMaxO[performance]: 1
ThreshProgI[performance]: /path/to/mrtg/scripts/alert.php
ThreshProgO[performance]: /path/to/mrtg/scripts/alert.php

For the alert scripts , I'd be looking at something like this, which sends orbiter notifications: http://www.modlog.net/?p=109 (http://www.modlog.net/?p=109)
Title: Re: Monitoring
Post by: brononius on September 05, 2011, 08:08:19 am
I think we need to see this larger then only the server. I agree that if you just want a cpu graph, you can do it manually with hand-written scripts/files/...
But since linuxmce provides a lot more (dhpc, dns, proxy, firewalling...), we can take this up a higher level. If you put in tools like cacti, you can start monitoring your whole network/house. Think about all computers, switches, routers, power-meters... The key-element here is the user-friendly stuff. People need to be able to do it themselves, simple. With this, you gain that more people will use it, less people will ask you questions...
And it must be flexible...

I'll see to create some extra templates. This way, we can monitor a lot more of the server. Think about the statistics of the firewall, dns-request, dhcp-assigment, syslogging of the server/ the network...


Title: Re: Monitoring
Post by: dextaslab on September 05, 2011, 10:57:39 am
I think we need to see this larger then only the server. I agree that if you just want a cpu graph, you can do it manually with hand-written scripts/files/...
But since linuxmce provides a lot more (dhpc, dns, proxy, firewalling...), we can take this up a higher level. If you put in tools like cacti, you can start monitoring your whole network/house. Think about all computers, switches, routers, power-meters... The key-element here is the user-friendly stuff. People need to be able to do it themselves, simple. With this, you gain that more people will use it, less people will ask you questions...
And it must be flexible...

LOL I see now, your looking for a magical do everything gui that records EVERYTHING you want with virtually no scripting/command prompt. Bwahahahahahah!!

The solution I provided (if you read the bash script) is generic, it'll work with ALL linuxmce systems (testing is needed) and if accepted by the Linuxmce gods, you do virutally nothing just install the package, and click on links in webadmin... It also does as you wanted; Intergrates into Linuxmce backend, looks through your database to finds how many MD's you have, and if they have HDD's it adds them to the graphing list.

"larger then only the server" - I wrote the script in about a day, and it is.
"And it must be flexible" - It is, I have a graph that shows downloaded weather reports and compares them with HDD temperatures.
"Think about all computers, switches, routers, power-meters" - it works with all snmp enabled devices, and those that you can generate scripted statistics. Again... written in about a day.

You've obivously just discovered cacti, and haven't had much experience using other related software... I have used most, and when your talking about flexibility and low overheads MRTG is better. I thought as a network engineer, you'd have used MRTG, as most larger networks do.

And with that I say "Your welcome", "Keep dreaming", and by all means prove me wrong with your working demo.
Title: Re: Monitoring
Post by: brononius on September 05, 2011, 11:27:05 am
Positive reaction... 
Sorry if I'm dreaming for a better world of linuxmce.

I'll see how others react on this topic. And decide from their on if i'll keep on trying to promote this...


Little remark though: RRDtool is extracted from MRTG, and Cacti is based on RRDtool.
1+1=?





Title: Re: Monitoring
Post by: hari on September 05, 2011, 09:35:29 pm
also have a look at the existing data logger plugin..
Title: Re: Monitoring
Post by: mkbrown69 on September 06, 2011, 04:04:35 am
Not to throw gas on a fire, but here's an example of Cacti power graphs and temperature graphs feeding into Misterhouse home automation .  Marc's been around MH and Linux for a while...

http://marc.merlins.org/perso/linuxha/post_2010-08-13_Fine-grained-house-wide-power-monitoring-with-Brultech-ECM1240_-ecmread_py-_with-net-metering-support_-and-graphing-with-cacti.html

http://marc.merlins.org/perso/linuxha/post_2010-08-06_Temperature_-moisture_-humidity_-and-UV-monitoring-and-graphing-with-1wire-devices_-owfs_-and-cacti.html

As Hari suggests, you might want to check out the data logger functionality that's presently built-in to LMCE, and compare / contrast that against MRTG/Cacti.  It might be beneficial to have a requirements and capability analysis of the three, since logging and graphing can form a crucial part of the system and will likely have a data warehousing function for many people who want historical data for a reason.  I'm speaking as someone who's been using Misterhouse myself since 2004 to record internal house temperatures, and I have the RRD's to prove it!

I've used SAR, CollectD, nmon/topaz/ganglia, RRDtool with Perl, Tivoli, and so on...  There are lots of great tools out there; in this case, I think what's most important is that the logging, data warehousing, and graphing are all like Lego blocks.  Parts that can be re-used as common infrastructure consistently throughout the LMCE architecture.

My $0.02 CDN before HST...

/Mike
Title: Re: Monitoring
Post by: dextaslab on September 06, 2011, 03:32:02 pm
also have a look at the existing data logger plugin..
LOL, yeah that could work :-P

I had a quick look through the source to try and work out how to use it, it there any online documentation on the arguments it takes, and how many variables can you pass to it? I could only work out it takes very specific events? would it take much to convert it (if it needs it) to accept:
<device_ID>,<data description>,<max value>,<alert issued at value><number of values>,<value>,<value>, etc...
Example:
<35>,<"Disk Usage">,<100%>,<95%>,<2>,<100%>,<43%>... Just an idea.

Basically I want to know if you can fake sensor output?


Title: Re: Monitoring
Post by: hari on September 07, 2011, 12:58:38 pm
we can extend it to log additional events. We can add new events like "System health report"/whatever..
An event can have more than one parameter.

br Hari
Title: Re: Monitoring
Post by: brononius on January 09, 2012, 02:52:59 pm
I finally had the time to create/install/configure a bit my monitoring.You can check it out on realtime: http://cacti.oniria.be. Login with linuxmce / lnxmc. For the moment, this is a limited user. Check out on the left the server part. Corius is the name of the server...
I'll see if I keep it like this or not (abuse, spam, hack...). I hope I can keep it open like this. At least for a while.
If you want more privilege to see it working, send me a pm...

For the moment, I just monitor some basic stuff of the server (bandwidth, disk space, load, database performance...).
I mainly use the cacti for network monitoring, power usage  of my home... And some alerting (tresholds) for some stuff (like to much power usage and so).

The plan is to add:

Let me know if you think of interesting things...
You never know what the result can be...



ps of course this is running on the same machine as linuxmce...  ;D
Title: Re: Monitoring
Post by: bongowongo on January 09, 2012, 05:00:22 pm
I finally had the time to create/install/configure a bit my monitoring.You can check it out on realtime: http://cacti.oniria.be. Login with linuxmce / lnxmc. For the moment, this is a limited user. Check out on the left the server part. Corius is the name of the server...
I'll see if I keep it like this or not (abuse, spam, hack...). I hope I can keep it open like this. At least for a while.
If you want more privilege to see it working, send me a pm...

For the moment, I just monitor some basic stuff of the server (bandwidth, disk space, load, database performance...).
I mainly use the cacti for network monitoring, power usage  of my home... And some alerting (tresholds) for some stuff (like to much power usage and so).

The plan is to add:
  • templates for the services (graphs)
  • templates for the asterik (graphs)
    once i started to configure my phones in there
  • up/down status for services (manage)
    dns/proxy/dhcp/samba/web...
  • logging module (syslog)
  • Create a weathermap
    Graphical scheme with status
  • ...

Let me know if you think of interesting things...
You never know what the result can be...



ps of course this is running on the same machine as linuxmce...  ;D

Very cool, do you have a wiki how to implement this?
Title: Re: Monitoring
Post by: brononius on January 09, 2012, 05:12:51 pm
Very cool, do you have a wiki how to implement this?

Not directly. But once i see what the rest thinks about it, I'll write one. I first want to see what we need/want. And then I want to see if it's possible to create a kind of template of it all. This way, people don't have to 'understand' this module, they just can use/view it...
Add it, log it, check it... And be alerted if something is wrong.


ps but in the meantime, it's really easy to install/use cacti. Just try a 'sudo apt-get install cacti' and you should be on your way.
Title: Re: Monitoring
Post by: bongowongo on January 09, 2012, 07:18:07 pm
Not directly. But once i see what the rest thinks about it, I'll write one. I first want to see what we need/want. And then I want to see if it's possible to create a kind of template of it all. This way, people don't have to 'understand' this module, they just can use/view it...
Add it, log it, check it... And be alerted if something is wrong.


ps but in the meantime, it's really easy to install/use cacti. Just try a 'sudo apt-get install cacti' and you should be on your way.

Yes that is me, I am your target user :)
I have very little time to test now, if I find time I will do, but I want to put all my time in the wiki and some snapshots (ahum possy ;) )
Title: Re: Monitoring
Post by: brononius on January 13, 2012, 02:15:52 pm
Small example of the syslog module: http://forum.linuxmce.org/index.php/topic,12292.0.html
A pitty i can't give you limit access to it like for the graphs.
Title: Re: Monitoring
Post by: tschak909 on January 19, 2012, 04:22:44 am
guys...guys...guys...

Please actually take a look at the system we have.

Let it soak in, WE HAVE A DISTRIBUTED MESSAGING SYSTEM, WHICH CAN NOT ONLY SEND COMMANDS, BUT EMIT AND CONSUME __EVENTS__ ... THIS IS HOW THE DATA LOGGER WORKS!

The data logger can be extended to intercept any events defined in the database via the web admin, along with their parameters. This isn't complicated, it just takes a little of ACTUALLY UNDERSTANDING THE SYSTEM, INSTEAD OF WRITING AROUND IT!

But if you're too impatient:

(1) create the events you want
(2) use MessageSend to synthesize those events from outside if you wish, or write a DCE device to emit them.
(3) extend the Data Logger plugin to intercept and consume those events.

Come on guys, seriously. ;)
-Thom
Title: Re: Monitoring
Post by: brononius on January 19, 2012, 07:35:00 am
I'm not sure that i understand what you're saying here.
You mean that you've got a syslog server already in place?
Is there somewhere a wiki about this? The most part i'm interested in, is the ease to view it. The GUI to say it like that...


ps I'm not trying to implement "other" things. I'm just a big fan of the cacti architecture because it's really fast, complete monitoring/alerting tool, and above all: it's awonderful and  'idiot prove' GUI... And this last part has been proven several times in my professional live... :P
Monitoring presence, graph bandwidth-load-..., alerting, nicely view all your logs, weathermaps, inventory your network... With all this,  it's just your imagination that blocks you... So it's not only the log module, it's the combination of it all...
Fe i receive a mail once my power usuage is going above 5000W, or a mail is send out once my HD is about 75% full. And all this without scripting (or at least without the knowledge of scripting)...
For me, it's just a big + to have something like this added to 'my' central server that already controls everything...
Send me a PM if you want to log in to my server with a bit more user credentials. So you can see what else we (=linuxmce users) could do with this tool...
Title: Re: Monitoring
Post by: tschak909 on January 19, 2012, 07:44:02 am
you can take, create events that you wish to monitor, alter monitoring software to emit those events either via MessageSend, OR via a custom DCE device, then alter the Data Logger plugin to consume those events and put them in the database, so that the UI in the web admin can use them?

-Thom
Title: Re: Monitoring
Post by: mkbrown69 on January 23, 2012, 02:07:40 am
Brononi,

What I think Thom is getting at is that if you're going to champion Cacti integration, you'll have to extend Cacti a bit to send DCE events, so that DataLogger can record them, and the rest of LMCE can act on them.  So, instead of Cacti sending an e-mail or SMS when a threshold is exceeded, it would use MessageSend to emit an LMCE event which Datalogger would record, and that the system could act on if a scenario was defined.  Like so...

Power usage goes over 5kw, Cacti generates a MessageSend to LMCE, Datalogger records the threshold exceeded event, and an "excessive power use" scenario is triggered, which broadcasts a message to all orbitors, and a "chi-Ching" sound is played over all audio devices.

So, it means some work extending Cacti, some work extending Datalogger, and some work figuring out events.  Plus, doing all that in such a way that someone else can extend your work to graph other things (soil moisture readings, humidity, Squid proxy results for domains, etc).  It would also probably involve integrating Cacti's web GUI into the LMCE web GUI, and use LMCE defined users.

I agree with you that Cacti and rrd's are great for recording some types of data, like multiple temperature sensors.  Assuming OWFS as a sensing infrastructure, it samples by default every 10 seconds per sensor.  So, that works out to 8640 samples per sensor per day.  Over time, those samples can be averaged because the further away they are from the present, the less need there is for precision (which is what RRD's do).   Cacti is also great at dealing with all the noise that syslog can generate.  Same with other types of repetitively sampled data, where the need for precision becomes less important over time.

Where I think Datalogger integration could really shine is for correlating events from multiple subsystems into a "timeline of events".  It's the concept of federated data; one overseer of other more specialized reporting/monitoring systems. It could be useful for debugging complex scenerios (this motion event triggered that lighting event when this other condition existed) as well as for security auditing (think alarm system event log), telephony logging, MythTV events, etc.  It would be especially nice if the Datalogger events could be viewed from an orbitor.  What would be really cool is heuristics to mine the Datalogger for patterns.  Think a "vacation mode" which operates lights based on the inhabitants past behaviours.

Just my $0.02 CDN as someone who's presently dealing with Federating configuration and compliance management systems at work (in addition to other things).

/Mike