Author Topic: I made a monitoring tool for debugging.  (Read 1371 times)

l3mce

  • NEEDS to work for LinuxMCE
  • ***
  • Posts: 1081
    • View Profile
I made a monitoring tool for debugging.
« on: March 18, 2011, 10:30:31 pm »
For the install process (or really any changes you want to monitor that an executable produces), I have made a tool that monitors definable human readable files in a specific directory. You supply:
1 The folder to monitor, it will wait for it to exist to start if need be.
2 The pattern of files (eg *.conf) to read diffs from, if left blank will search all human readable files.
3 Where you want the diff files to live, default is /home/diffs.

You will have to run this, then execute your script in another terminal. I tried to launch scripts from it, and everything got gummed up.

It then spits out diff files whenever files change or are added, what script is running at that time, and what the difference is in which human readable file. I will tweak it more when I get some other things done. It needs to be recursive on demand. Need to be able to specify more folders.

diffmon1.6
Requires dialog and symlinks (local ubu repos)
Code: [Select]
#!/bin/bash
### This script is free.

#Needs to be the only instance running. Makes sure that happens but is name specific.
trap 'rm -r /tmp/difftmp' EXIT
procs=$(echo | ps -eo pid,user,args | grep 'diffmon.sh' | wc -l)
if (($procs > 3)); then echo "Multiple instances running. Terminating... please reload"; sleep 1; rm -r /tmp/difftmp; killall -v 'diffmon.sh'; fi

### Sets suffix numberer, installs dialog and cleans up after old operations
if [ -z $runonce ]; then
   suffix=1
   type -P dialog &>/dev/null || apt-get install dialog -y
   type -P symlinks &>/dev/null || apt-get install symlinks -y
   if [ -d /tmp/difftmp ]; then rm -r /tmp/difftmp; fi
   if [ -e /tmp/ast1.txt ]; then rm /tmp/ast1.txt; fi
   if [ -e /tmp/ast2.txt ]; then rm /tmp/ast2.txt; fi
   runonce=something
   getout=live
fi

### Sets the directory to monitor
thedir () {
   searchdir1=$(dialog --title "Directory to Monitor" --stdout --inputbox "Please choose a folder to monitor
   (eg /etc/asterisk ) cannot be blank" 14 48)
   if [ $? = 1 ]; then exit 0; fi
}
if [ -z $searchdir1 ]; then thedir; fi

### Figures out if the directory to monitor exists already and makes sure you didn't typo
if [ ! -d $searchdir1 ]; then dialog --title "Confirm" --yesno "The directory $searchdir1 does not exist. Are you expecting it to be created by the process?" 10 48; fi

### Sets the file pattern to monitor
#filemon=$(dialog --title "File Pattern" --stdout --inputbox "Please indicate a file pattern to monitor (eg *.conf, or for all readable files leave blank)" 14 48)

### Sets the output folder for diff files/deleted files
difhome=$(dialog --title "Output" --stdout --inputbox "Please specify a folder to put diff files in. If it does not exist it will be created. If left blank will default to /home/diffs" 14 48)
if [ -z $difhome ];then difhome=/home/diffs; fi

### Main loop
rundiff () {
   getin=die
   symlinks -dr $searchdir1
   ### Waits for directory to exist and creates the temporary environment.
   while [ ! -d $searchdir1 ]; do sleep 1; done
   if [ ! -d $difhome ]; then mkdir $difhome; chmod 777 $difhome; fi
   if [ ! -d /tmp/difftmp ]; then mkdir /tmp/difftmp; chmod 777 /tmp/difftmp; rsync -rtv --links $searchdir1/ /tmp/difftmp; fi
   while true; do
      if [ ! "$getout"="$getin" ]; then echo "-1"; break; fi
     
      ### Figures out if files are trying to be deleted by something, and moves them for review. If they come into existence then that is noted, but not read in diff.
      home=$(ls -R $searchdir1 | wc -l)
      target=$(ls -R /tmp/difftmp | wc -l)
      if [ "$home" -lt "$target" ]; then
         if [ ! -d $difhome/deleted ]; then mkdir -p $difhome/deleted; fi
         mover=$(diff -r $searchdir1 /tmp/difftmp | grep Only | sed -e 's/Only in /rsync -avb --remove-source-files /g;s/\: /\/*/g')
         movdir=$(echo "$mover $difhome/deleted")
         #   remover=$(diff -r $searchdir1 /tmp/difftmp | grep Only| sed -e 's/Only in /rm -r /g;s/\: /\/*/g')
         $movdir
         #   cleanrm=true;
      fi
     
      ### Finds diffs and spits them into format
      if (diff -r $searchdir1 /tmp/difftmp); then sleep 1; else
         if [ -z $filemon ]; then filemon=$(find $searchdir1 -name '*' -prune -readable -type f | sed 's_.*/__'); fi
         if [ ! -e /tmp/ast1.txt ]; then
            echo | ls -l $searchdir1 > /tmp/ast1.txt;
            input_variable=$"diffmon"${suffix}
            iv2=$input_variable;
            echo "" > $difhome/$iv2
            echo "" >> $difhome/$iv2
            echo "                ************** $iv2 ***************" >> $difhome/$iv2
            echo "" >> $difhome/$iv2
            echo "Files added > or removed <" >> $difhome/$iv2
            (echo | diff /tmp/ast1.txt /tmp/ast2.txt >> $difhome/$iv2);
            iv1=$(ls $searchdir1/$filemon);
            echo "" >> $difhome/$iv2
            echo  "****** Scripts running during change " >> $difhome/$iv2
            (echo | ps -eo pid,user,args | grep 'bin/bash' >> $difhome/$iv2);
            for iv1diff in $iv1; do
               if ! diff -r $searchdir1/$iv1diff /tmp/difftmp/$iv1diff; then
                  echo "" >> $difhome/$iv2
                  echo  "********* Changes in $searchdir1/$iv1diff"  >> $difhome/$iv2
                  (diff -r $searchdir1/$iv1diff /tmp/difftmp/$iv1diff >> $difhome/$iv2)
               fi
            done
            echo "Diff $difhome/$iv2 written";
            rsync -azdr $searchdir1/ /tmp/difftmp
            if [ -e /tmp/ast2.txt ]; then rm /tmp/ast2.txt; fi
            let suffix=$suffix+1
         else
           
            ### Redundant function allowing two files to always be contrasted and never confused
            echo | ls -l $searchdir1 > /tmp/ast2.txt;
            input_variable=$"diffmon"${suffix}
            iv2=$input_variable;
            echo "" > $difhome/$iv2
            echo "" >> $difhome/$iv2
            echo "                ************** $iv2 ***************" >> $difhome/$iv2
            echo "" >> $difhome/$iv2
            echo "Files added > or removed <" >> $difhome/$iv2
            (echo | diff /tmp/ast1.txt /tmp/ast2.txt >> $difhome/$iv2);
            iv1=$(ls $searchdir1/$filemon);
            echo "" >> $difhome/$iv2
            echo  "****** Scripts running during change " >> $difhome/$iv2
            (echo | ps -eo pid,user,args | grep 'bin/bash' >> $difhome/$iv2);
            for iv1diff in $iv1; do
               if ! diff -r $searchdir1/$iv1diff /tmp/difftmp/$iv1diff; then
                  echo "" >> $difhome/$iv2
                  echo  "********* Changes in $searchdir1/$iv1diff"  >> $difhome/$iv2
                  (diff -r $searchdir1/$iv1diff /tmp/difftmp/$iv1diff >> $difhome/$iv2)
               fi
            done
            echo "Diff $difhome/$iv2 written";
            rsync -azdr $searchdir1/ /tmp/difftmp
            if [ -e /tmp/ast1.txt ]; then rm /tmp/ast1.txt; fi
            let suffix=$suffix+1
         fi
         
         ### Backup insist one contrast file exist at all times.
         if [ ! -e /tmp/ast1.txt ] && [ ! -e /tmp/ast2.txt ]; then echo | ls -l $searchdir1 > /tmp/ast1.txt; fi
      fi
      sleep 1
   done
}

### Confirms input information is correct before launching
dialog --title "Confirm" --yesno "Choose ok to begin, or cancel to exit and start over" 10 48;
if  [ $? = 0 ]; then clear
   rundiff
fi
else
exit 0


I would appreciate feedback
« Last Edit: March 31, 2011, 12:21:47 pm by l3mce »
I never quit... I just ping out.

l3mce

  • NEEDS to work for LinuxMCE
  • ***
  • Posts: 1081
    • View Profile
Re: I made a monitoring tool for debugging.
« Reply #1 on: March 19, 2011, 12:24:17 am »
* l3mce didn't realize that dialog was not standard on k810. Fixed.

*** Now handles subdirectories, and moves files that are being deleted by something. Echoes a mv error while moving things that something is trying to delete, but it moves them... and whatever else deletes them. Am going to do an install and monitor all changes to /etc to see what breaks.

Sample output from watching folder /etc while installing vlc.
Code: [Select]
cat diffmon
1c1                                       
< total 1828                             
---                                       
> total 1824                             
139c139                                   
< -rw-r--r--  1 root     root      84382 2011-03-20 17:16 ld.so.cache                                           
---                                                     
> -rw-r--r--  1 root     root      83182 2011-03-13 20:53 ld.so.cache                                           
163c163                                                 
< -rw-r--r--  1 root     root      22564 2011-03-20 17:16 mailcap                                               
---                                                     
> -rw-r--r--  1 root     root      19358 2011-03-12 21:56 mailcap                                               
179c179                                                 
< -rw-r--r--  1 root     root       1251 2011-03-20 17:15 mtab                                                 
---                                                     
> -rw-r--r--  1 root     root       1251 2011-03-20 16:51 mtab                                                 
213c213                                                 
< -rwxrwxrwx  1 root     root        683 2011-03-20 17:11 pluto.conf                                           
---                                                     
> -rwxrwxrwx  1 root     root        683 2011-03-20 16:51 pluto.conf                                           

****** Scripts running during change
 3548 root     /bin/bash /usr/pluto/bin/Spawn_Device.sh 28 localhost LIRC_DCE                                   
 4831 root     /bin/bash ./diffmon.sh                   
10437 root     /bin/bash /usr/pluto/bin/Start_NewMD_interactor.sh                                               
10438 root     /bin/bash /usr/pluto/bin/Dhcpd-Plugin.sh
10439 root     /bin/bash /usr/pluto/bin/VoiceMailMonitor.sh                                                     
10441 root     /bin/bash /usr/pluto/bin/VoiceMailMonitor.sh                                                     
10610 root     /bin/bash /usr/pluto/bin/StorageDevices_StatusRadar.sh background                               
10654 root     /bin/bash /usr/pluto/bin/StorageDevices_SambaRadar.sh background                                 
10671 root     /bin/bash /usr/pluto/bin/StorageDevices_NFSRadar.sh background                                   
11454 root     /bin/bash /usr/pluto/bin/LaunchOrbiter.sh -d 21 -r localhost -l /var/log/pluto/21_LaunchOrbiter.sh.log                                                   
13762 root     /bin/bash /usr/pluto/bin/Start_X_Wrapper.sh --parms start --client /usr/bin/xfwm4 --server :0 -ignoreABI -ac -allowMouseOpenFail vt7 -logverbose 9 -br --flags -config /etc/X11/xorg.conf                       
13821 root     /bin/bash /usr/pluto/bin/LaunchManagerDaemon.sh                                                 
14056 mcc      /bin/bash                               
15896 root     /bin/bash /usr/pluto/bin/Spawn_DCERouter.sh                                                     
16462 root     /bin/bash /usr/pluto/bin/Spawn_Device.sh 15 localhost App_Server                                 
16542 root     /bin/bash /usr/pluto/bin/Restart_MythBackend.sh                                                 
16642 root     /bin/bash /usr/pluto/bin/Spawn_Device.sh 16 localhost HAL                                       
16721 root     /bin/bash /usr/pluto/bin/Spawn_Device.sh 18 localhost Asterisk                                   
17125 root     /bin/bash /usr/pluto/bin/Spawn_Device.sh 19 localhost Text_To_Speech                             
17643 root     /bin/bash /usr/pluto/bin/Spawn_Device.sh 31 localhost Disk_Drive                                 
19648 root     /bin/bash /usr/pluto/bin/Spawn_Device.sh 32 localhost External_Media_Identifier                 
19974 root     /bin/bash /usr/pluto/bin/Spawn_Device.sh 39 localhost ZWave                                     
20496 root     /bin/bash /usr/pluto/bin/Spawn_Device.sh 21 localhost LaunchOrbiter.sh                           
20768 root     SCREEN -d -m -h 3000 -S SimplePhone_22 /bin/bash -x Spawn_Device.sh 22 127.0.0.1 SimplePhone     
20770 root     /bin/bash -x Spawn_Device.sh 22 127.0.0.1 SimplePhone                                           
20772 root     SCREEN -d -m -h 3000 -S Xine_Player_23 /bin/bash -x Spawn_Device.sh 23 127.0.0.1 Xine_Player     
20773 root     /bin/bash -x Spawn_Device.sh 23 127.0.0.1 Xine_Player                                           
20776 root     SCREEN -d -m -h 3000 -S MPlayer_Player_26 /bin/bash -x Spawn_Device.sh 26 127.0.0.1 MPlayer_Player                                                       
20778 root     /bin/bash -x Spawn_Device.sh 26 127.0.0.1 MPlayer_Player                                         
20782 root     SCREEN -d -m -h 3000 -S MythTV_Player_37 /bin/bash -x Spawn_Device.sh 37 127.0.0.1 MythTV_Player
20783 root     /bin/bash -x Spawn_Device.sh 37 127.0.0.1 MythTV_Player                                         
20787 root     SCREEN -d -m -h 3000 -S Game_Player_50 /bin/bash -x Spawn_Device.sh 50 127.0.0.1 Game_Player     
20788 root     /bin/bash -x Spawn_Device.sh 50 127.0.0.1 Game_Player                                           
20793 root     SCREEN -d -m -h 3000 -S Hulu_Player_52 /bin/bash -x Spawn_Device.sh 52 127.0.0.1 Hulu_Player     
20798 root     /bin/bash -x Spawn_Device.sh 52 127.0.0.1 Hulu_Player                                           
21292 root     /bin/bash ./diffmon.sh                   
21393 root     /bin/bash ./diffmon.sh                   
22782 mcc      /bin/bash                               
31589 mcc      /bin/bash                               

********* Changes in file ld.so.cache
Binary files /etc/ld.so.cache and /tmp/difftmp/ld.so.cache differ     
                                         
********* Changes in file mailcap                       
51,62d50                                               
< video/mpeg; vlc '%s'; description="MPEG Video"; test=test -n "$DISPLAY"                                       
< video/x-mpeg; vlc '%s'; description="MPEG Video"; test=test -n "$DISPLAY"                                     
< video/mpeg-system; vlc '%s'; description="MPEG Video"; test=test -n "$DISPLAY"                               
< video/x-mpeg-system; vlc '%s'; description="MPEG Video"; test=test -n "$DISPLAY"                             
< audio/x-wav; vlc '%s'; description="WAV Audio"; nametemplate=%s.wav; test=test -n "$DISPLAY"                 
< video/mpeg4; vlc '%s'; description="MPEG-4 Video"; test=test -n "$DISPLAY"                                   
< audio/mpeg; vlc '%s'; description="MPEG Audio"; nametemplate=%s.mpg; test=test -n "$DISPLAY"                 
< audio/mpegurl; vlc '%s'; description="MPEG Audio URL"; nametemplate=%s.m3u; test=test -n "$DISPLAY"           
< audio/x-mp3; vlc '%s'; nametemplate=%s.mp3; description="MPEG Audio"; test=test -n "$DISPLAY"                 
< audio/mpeg4; vlc '%s'; description="MPEG-4 Audio"; test=test -n "$DISPLAY"                                   
< application/mpeg4-iod; vlc '%s'; description="MPEG-4 Video"; test=test -n "$DISPLAY"                         
< application/mpeg4-muxcodetable; vlc '%s'; description="MPEG-4 Video"; test=test -n "$DISPLAY"                 
152,169d139                                             
< video/mpeg; vlc -I rc -V caca '%s'; needsterminal; description="MPEG Video"                                   
< video/x-mpeg; vlc -I rc -V caca '%s'; needsterminal; description="MPEG Video"                                 
< video/mpeg-system; vlc -I rc -V caca '%s'; needsterminal; description="MPEG Video"                           
< video/x-mpeg-system; vlc -I rc -V caca '%s'; needsterminal; description="MPEG Video"                         
< audio/x-wav; vlc -I rc -V caca '%s'; nametemplate=%s.wav; needsterminal; description="WAV Audio"             
< video/mpeg4; vlc -I rc -V caca '%s'; needsterminal; description="MPEG-4 Video"                               
< audio/mpeg; vlc -I rc -V caca '%s'; nametemplate=%s.mpg; needsterminal; description="MPEG Audio"             
< audio/mpegurl; vlc -I rc -V caca '%s'; nametemplate=%s.m3u; needsterminal; description="MPEG Audio URL"       
< audio/x-mp3; vlc -I rc -V caca '%s'; nametemplate=%s.mp3; needsterminal; description="MPEG Audio"             
< audio/mpeg4; vlc -I rc -V caca '%s'; needsterminal; description="MPEG-4 Audio"                               
< application/mpeg4-iod; vlc -I rc -V caca '%s'; needsterminal; description="MPEG-4 Video"                     
< application/mpeg4-muxcodetable; vlc -I rc -V caca '%s'; needsterminal; description="MPEG-4 Video"             
< video/x-msvideo; vlc '%s'; description="MS Video (AVI)"; test=test -n "$DISPLAY"                             
< video/quicktime; vlc '%s'; description="Apple Quicktime Video"; test=test -n "$DISPLAY"                       
< application/ogg; vlc '%s'; nametemplate=%s.ogg; description="Ogg stream"; test=test -n "$DISPLAY"             
< application/x-ogg; vlc '%s'; nametemplate=%s.ogg; description="Ogg stream"; test=test -n "$DISPLAY"           
< application/x-ms-asf-plugin; vlc '%s'; description="Windows Media Video"; test=test -n "$DISPLAY"             
< application/x-mplayer2; vlc '%s'; description="Windows Media"; test=test -n "$DISPLAY"                       
211,216d180
< video/x-msvideo; vlc -I rc -V caca '%s'; needsterminal; description="MS Video (AVI)"
< video/quicktime; vlc -I rc -V caca '%s'; needsterminal; description="Apple Quicktime Video"
< application/ogg; vlc -I rc -V caca '%s'; nametemplate=%s.ogg; needsterminal; description="Ogg stream"
< application/x-ogg; vlc -I rc -V caca '%s'; nametemplate=%s.ogg; needsterminal; description="Ogg stream"
< application/x-ms-asf-plugin; vlc -I rc -V caca '%s'; needsterminal; description="Windows Media Video"
< application/x-mplayer2; vlc -I rc -V caca '%s'; needsterminal; description="Windows Media"

Ok. This appears to be working and stable. At least in my dev enviro. Can ctrl+c out of it and everything.
* l3mce Important note. I am installing, if it is not already, a package called symlinks, which I use to delete broken symbolic links in the specified directory. This is necessary because you cannot ignore symbolic links with a recursive diff, and if they are broken, it loops... spitting diff files out.
« Last Edit: March 28, 2011, 06:50:00 am by l3mce »
I never quit... I just ping out.

l3mce

  • NEEDS to work for LinuxMCE
  • ***
  • Posts: 1081
    • View Profile
Re: I made a monitoring tool for debugging.
« Reply #2 on: March 28, 2011, 06:14:07 am »
Update:

This tool now works in Ubuntu/Kubuntu 0810 and 1004, and is not dangerous. You cannot specify a file to monitor however. It simply monitors all human readable files in a specified directory, recursively.

It is a never ending loop. You have to ctrl+c to exit. Therefore it does not clean up after itself. After monitor you will want to:
Code: [Select]
rm -r /tmp/difftmpOr run it again. It WILL clean up that way. You can just exit at the first screen.

    This script MUST BE RUN as sudo.

    As a failsafe, it makes sure no other instances are running when it starts, and for this MUST be named "diffmon.sh", though I have not had a disgraceful exit since revision.

    I do not know what it will do if you do not have enough space to mirror the directory into the temp environment. I recommend NOT monitoring  /  or /var . There will be recursive loops... and probably be hard to achieve whatever you want. You could monitor /var/cache etc... just not the root dir /var. There are probably other directories for which this is true, but I know only of that one.

    The diffs, if you do not specify a directory to spit to, end up in /home/diffs. If you want to monitor /home you will need to specify another directory to put the diffs.

    If files come into existence, they are noted, but not read, in the diff file. This only monitors CHANGES to files and directories. So if they are deleted, but never changed, they end up in /the-directory-specified/deleted so you can review them.
« Last Edit: March 28, 2011, 08:42:45 am by l3mce »
I never quit... I just ping out.

posde

  • Administrator
  • LinuxMCE God
  • *****
  • Posts: 3019
  • Wastes Life On LinuxMCE Since 2007
    • View Profile
    • My Home
Re: I made a monitoring tool for debugging.
« Reply #3 on: March 28, 2011, 08:07:11 am »
l3mce,

maybe
Code: [Select]
trap 'rm -r /tmp/difftmp' EXITat the beginning of the script, will help cleaning up

l3mce

  • NEEDS to work for LinuxMCE
  • ***
  • Posts: 1081
    • View Profile
Re: I made a monitoring tool for debugging.
« Reply #4 on: March 28, 2011, 08:32:46 am »
Perfect. TYVM. Works on my end.

*fixed the numerous ls entries.
« Last Edit: March 29, 2011, 05:21:08 am by l3mce »
I never quit... I just ping out.

l3mce

  • NEEDS to work for LinuxMCE
  • ***
  • Posts: 1081
    • View Profile
Re: I made a monitoring tool for debugging.
« Reply #5 on: March 29, 2011, 04:13:21 pm »
This was my attempt to watch asterisk during a 1004 install. (Was watching /etc, while this was an accident, it appears that asterisk makes changes in several subdirectories, so it proved useful)

I noticed asterisk was choking while looking for the "modules.conf" file which had been deleted. The tool backs up anything deleted while monitoring, so I moved it back, and asterisk installed without error. I am not suggesting it works, but did install w/o error. I do not have hardware to test.

This is one big file with all of the diffs in it. They are labeled.

http://pastebin.com/W6idTP7V

It is treating directory changes like file changes too... misnomering them "Changes in file..." sorry for the confusion. That should now be fixed... with some other minor bugs. Working on not overwriting existing diffs if same dir is chosen twice to write to.
« Last Edit: March 31, 2011, 12:23:27 pm by l3mce »
I never quit... I just ping out.