stevepedwards.com/DebianAdmin linux mint IT admin tips info

Using Sendemail/Curl For a Motion Alert and Hacking Its 2+ year old Perl TLS Bug

After studying the conf file for Motion looking for an Email Alert function of some kind, I found:

# Command to be executed when an event starts. (default: none)

# An event starts at first motion detected after a period of no motion defined by gap

; on_event_start value

I thought about using sendemail as the value that could send a message and attach a jpg of a motion event.

This opened up a real can of worms because of an unfixed, 2+ year long issue with Perl modules, and research to get the command structure, jpg file name to be useable and an insight into the Perl programmers world that I never even knew existed.

SendEmail

Years ago, when security was less stringent, you could use these command line programs to send a quick email to an SMTP server on port 25. No encryption required, just an account, uname and password – about as simple as things could be.

Now, for various reasons and advantages for us all for privacy, spam blocking etc. SSL (Secure Sockets Layer) and TLS (Transport Layer Security) are used by most providers like gmail, on ports 465 or 587, so an example line to use for sendemail may be:

sendEmail -t xxx@gmail.com -f picam@spoofed.com -xu uname -xp password -a /attach.jpg -m "msg" -u "subject"  -o tls=auto -s smtp.gmail.com:587

For more switches, use the –h help, or man page.

So far, so good? No.

Test this line works before using it in Motion – because it probably won't work, depending on what OS, updates, perl modules, and wot-not you have!

I got fault lines similar to:

 

Aug 13 14:49:15 piblanc sendemail[4810]: NOTICE => Authentication not supported by the remote SMTP server!

Aug 13 14:49:16 piblanc sendemail[4810]: ERROR => Received: 530 5.7.0 Must issue a STARTTLS command first. h6sm3520159wiy.3 - gsmtp

then:

invalid SSL_version specified at /usr/local/share/perl/5.14.2/IO/Socket/SSL.pm line xxx:

This seems to be an old bug, and it took a while to research and sent me down an educational but incorrect path regarding CSPAN (eh?), and doing updates for Perl modules that was intriguing, but did not solve the issue completely.

How can this be, that such an old problem has not been fixed in some?/most? linux distribs, including Apple?

I found info and forums about this, including the programmer's homepage which really pissed me off that he hasn't got it fixed, with the pathetically vague help text shown below:

http://caspian.dotconf.net/menu/Software/SendEmail/

TLS Support
Starting with sendEmail v1.54, TLS support is included! To enable TLS support
simply install the Net::SSLeay andIO::Socket::SSL perl modules. The following new command line parameters are now available:
    
-o tls=auto This is the default, TLS will be used if possible.
    
-o tls=yes Use this to require TLS for message delivery.
    
-o tls=no Use this to disable TLS support.
If TLS is giving strange errors, try upgrading the Net::SSLeay and IO::Socket::SSL perl modules. Please do NOT report TLS bugs unless you have already done this! If you're running up-to-date versions of these modules and you are getting TLS errors, your detailed bug report will be appreciated.
Yes, you can finally use SendEmail to send messages to your GMail account :

Really, Mr Programmer...? Have you tested it? We've been here before recently with Pmagic eh?

What does that even mean unless you are a Perl geek? How do you install/upgrade those modules?

So, doing his suggested research for ages on google, I wasted more time upgrading unnecessary SSL modules, and failed with SSLeay anyway, when a one character ($) script hack fixes it thanks to some other clever dude that I forget what page I found it.

Anyway, what is debian/raspbian/others doing having a known buggy program in the repositories?

http://stackoverflow.com/questions/12750662/install-perl-module-netssleay-through-cpan

http://raspberrypi.stackexchange.com/questions/2118/sendemail-failure

For interest, I'll show you what the Perl upgrades involve, as I want to keep both Pis as similar as possible, so the other Pi now needs the same treatment.

I did a fresh install on PiBlanc, which for interest, took about 30 mins for dd to complete:

DellMint stevee # dd if=/Pi.img of=/dev/sdg

15523840+0 records in

15523840+0 records out

7948206080 bytes (7.9 GB) copied, 1833.87 s, 4.3 MB/s

30 mins to re-image the default 8GB NOOBS Pi image I saved, to a new 16GB card to re-install from scratch for the 2nd Pi.

Perl Updates command:

perl -MCPAN -e shell

CPAN.pm requires configuration, but most of it can be done automatically.

If you answer 'no' below, you will enter an interactive dialog for each

configuration option instead.

Would you like to configure as much as possible automatically? [yes]

Now, this Q+A goes on for a long while, and as I have no idea what its on about, I just default entered through all of it until:

Looking for CPAN mirrors near you (please be patient)

....... done!

New urllist

http://CPAN.mirror.rafal.ca/

http://httpupdate25.cpanel.net/CPAN/

http://mirror.csclub.uwaterloo.ca/CPAN/

Autoconfiguration complete.

commit: wrote '/root/.cpan/CPAN/MyConfig.pm'

You can re-run configuration any time with 'o conf init' in the CPAN shell

Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v1.960001)

Enter 'h' for help.

cpan[1]>

then do:

cpan[1]> install IO::Socket::SSL

cpan[1]> install done!

New urllist

http://CPAN.mirror.rafal.ca/

http://httpupdate25.cpanel.net/CPAN/

http://mirror.csclub.uwaterloo.ca/CPAN/

Fetching with HTTP::Tiny:

http://CPAN.mirror.rafal.ca/authors/01mailrc.txt.gz

Autoconfiguration complete.

commit: wrote '/root/.cpan/CPAN/MyConfig.pm'

You can re-run configuration any time with 'o conf init' in the CPAN shell

Terminal does not support AddHistory.

After all that, the SSLEAY failure goes along the lines of:

cpan[13]> install Net::SSLeay

** Found OpenSSL-1.0.1e installed in /usr

*** Be sure to use the same compiler and options to compile your OpenSSL, perl,

and Net::SSLeay. Mixing and matching compilers is not supported.

Do you want to run external tests?

These tests *will* *fail* if you do not have network connectivity. [n] y

SSLeay.xs:163:25: fatal error: openssl/err.h: No such file or directory

compilation terminated.

Makefile:353: recipe for target 'SSLeay.o' failed

make: *** [SSLeay.o] Error 1

MIKEM/Net-SSLeay-1.70.tar.gz

/usr/bin/make -- NOT OK

'YAML' not installed, will not store persistent state

Running make test

Can't test without successful make

Running make install

Make had returned bad status, install seems impossible

Failed during this command:

MIKEM/Net-SSLeay-1.70.tar.gz : make NO

cpan[14]> look Net::SSLeay

perl Makefile.PL

*** Found OpenSSL-1.0.1e installed in /usr

*** Be sure to use the same compiler and options to compile your OpenSSL, perl,

and Net::SSLeay. Mixing and matching compilers is not supported.

Do you want to run external tests?

These tests *will* *fail* if you do not have network connectivity. [n] Y

Checking if your kit is complete...

Looks good

Writing Makefile for Net::SSLeay

Writing MYMETA.yml

make

SSLeay.xs:163:25: fatal error: openssl/err.h: No such file or directory

compilation terminated.

Makefile:353: recipe for target 'SSLeay.o' failed

make: *** [SSLeay.o] Error 1

So, a waste of time but interesting that all these servers exist for Perl and its modules...

Does sendemail work after the successful SSL install section above?

Aug 13 15:10:21 piblanc sendemail[4981]: NOTICE => Authentication not supported by the remote SMTP server!

Aug 13 15:10:21 piblanc sendemail[4981]: ERROR => Received: 530 5.7.0 Must issue a STARTTLS command first. pf4sm3579898wjb.23 - gsmtp

No. So...

Googling around, I found:

apt-get install libnet-ssleay-perl

apt-get install libio-socket-ssl-perl

apt-get install libssl-dev

So what does sendemail say now?

invalid SSL_version specified at /usr/share/perl5/IO/Socket/SSL.pm line 332

The Hack:

vi /usr/share/perl5/IO/Socket/SSL.pm

search in vim:

(/) then paste the term below, and ENTER the term to find it just above where the search line takes you:

m{^(!?)(?:(SSL(?:v2|v3|v23|v2/3))|(TLSv1[12]?))$}i

delete the $ sign, so it changes to:

m{^(!?)(?:(SSL(?:v2|v3|v23|v2/3))|(TLSv1[12]?))}i

Save in vim (wq!)

Test your sendemail command:

sendemail -t user@gmail.com -f user@gmail.com -xu user -xp pword -a /Share/motion/*35*00.jpg -u "msg" -m "picam test file" -o tls=auto -s smtp.gmail.com:587

Aug 13 12:10:28 pinoir sendemail[4749]: Email was sent successfully!

The default Motion output jpg naming format is not user friendly, or very amenable to have just one chosen for the sendemail attachment, so you'll have to work out a way to rename them, or whatever to suit yourself, or you may want a video attachment instead if you have fast connections etc:

The Summary

Once you have installed the libs, amended the script $ hack, named the Motion output jpgs suitably, or captured a small size video from the type you set in motion.conf, probably not using asterisk wildcards due to the motion naming format to not send yourself tons of unwanted files in the mail, worked out how to not overload the Pi when Motion is capturing jgs and sendemail is trying to access them from the output folder, and send them out, you should get a mail alert containing the jpg of name you chose:

What Else?

Now all that's left is to get sendemail to run from inside the motion.conf file - either in motion start or end  sections?

You have to open ports in your home router, and set a DynDNS service if you have DHCP ADSL to get your home IP address, should you also want to view your camera WAN remotely using Firefox etc. on a port of your choice to watch your live web stream should you get an alert.

Let's hope all that doesn't crash the Pi just as the criminal/desired event was about to be filmed, or some such...

Good luck with all that.

Later...

I have motion sending .avi files to me by email now, as I had missed a msg text for the -m switch initially, so sendemail was waiting for StdIn.

One camera is producing 10 times more jpgs than the other, with the identical conf file?

Also, Piblanc has a brightness/darkness "swell" issue in the live image stream, when Pinoir cam has a stable image right from the start? Odd.

I'm going to diff the conf files to check they make sense...Wow, I missed a few things here!

< v4l2_palette 8
---
> v4l2_palette 6
122c122
< auto_brightness off
---
> auto_brightness on
257c257
< ffmpeg_cap_new on
---
> ffmpeg_cap_new off
265c265
< ffmpeg_timelapse 1
---
> ffmpeg_timelapse 0
298a299

I guess the auto brightness on/off is the cause of pinoir's cameras image "swell". Fixed that...

Now, one camera is producing twice as many jpgs as the other, though the capture settings are the same, but no mail is being sent due to a naming error, so the file did not exist.

The easiest and most informative file type and initial name to use as a mail file at first, is a timelapse as *.mpg, as only one is created periodically, and no other, until the next motion start and end period, if that 's what you set.

About 4 mins of timelapse is only about 800k in size, so good as alert info and not too big for an attachment.

The final motion.conf sendemail line that does what I want (so far!) is:

on_event_end sendEmail -t user@gmail.com -f noir@cam.com -a /Share/motion/*.mpg -xu user -xp pword -s smtp.gmail.com:587  -u "noir cam motion detected !!!" -m "noir cam motion detected !!!"

MotionTests.jpg

MotionTestsCamConfs.jpg

Oh! I nearly forgot! Motion complains re PID access for the daemon with a manual start. The problem is this happens each start time after a reboot as the PID directory does not exist:

ls -als /var/run/motion/
ls: cannot access /var/run/motion/: No such file or directory

From earlier Posts, remember there is another file to enable motion:

vi /etc/default/motion

# set to 'yes' to enable the motion daemon
start_motion_daemon=yes

I have a problem with the daemon not reading the conf file at boot. The service is starting but the LEDs are flashing, not constant, and there is no action or streaming.

As the manual testing so far was for the conf file in /etc/motion using:

motion -c /etc/motion/motion.conf

which tells motion specifically where the conf file is, it may need to be moved to the users default directory called .motion, so as root will be booting it, make it in roots directory:

~
$ mkdir .motion

cp -vr /etc/motion/motion.conf        .motion/

'/etc/motion/motion.conf' -> `.motion/motion.conf'

reboot to check it comes up with the camera LEDs on.

I did this also, checking perms are the same etc. and it still does not function from boot, as it does manually, like motion is running with an unconfigured conf file...?

Would only happen to me again! I messed up the update-rc command somehow, and removed the links to the /etc/init.d/rc2 dir, so removed and re-installed them (they were visible again) but they did not run.

You have to copy your conf file somewhere safe, then use remove with --purge and auto-remove to completely remove and re-install motion back to defaults (and the conf file so save yours!!) for the links to be replaced properly, then update-rc enable for motion to run at boot.

update-rc motion enable

One error like that and your chasing dir/file permissions and god knows what else for nothing.

Phew, what a nightmare again..! I'm obviously not cut out for this Linux Admin stuff...

reboot

All working now though, and sending me emails.

Also, in gmail, you can send the vids to Gdrive, attach an mpg viewer to Gdrive, and view them directly on the web with no need to DL them to the PC if you don't want.

GDrive.jpg

When you have set up your router to port forward, you can check the WAN IP in Kmeleon/Firefox to see your camera stream - here my back gate viewed from work:

homemotion.jpg

Fortunately, the default settings for pixel motion are sensible enough to not alert for the moving plant near the window, visible in the live stream, else I'd be bombed with email all day!

# Threshold for number of changed pixels in an image that
# triggers motion detection (default: 1500)
threshold 1500

Pretty cool stuff, even though it all took days to get to grips with.

Each camera and Pi2 set cost about £65 each, and I know there are now much cheaper 4 cam, DVR co-ax systems on Ebay for that, but...this wouldn't be a Linux site then would it?

All that is left now is to sign up with a DynDNS site (the vigor router tells of a vigor offering) to get any changes of DNS sent to me, or accessible, so I don't have to check manually each day before leaving home. Hmm - maybe not - it's in Chinese only...

I got setup for free at www.noip.com which was instant in giving my IP once the account was input into my router below.

You may also want to setup remote SSH access to the Pi (on a personal port other than 22) so you can stop it, or reboot etc.

If you have an Android phone, put Firefox on from play.google.com and watch the streams.

vigorddns.png

16/8/15

Other problems with this...

So far this has been testing different options, but one problem I haven't resolved is the timelapse film created gets APPENDED to for successive motion events, so grows to a size that can't be sent!

That's is no good! It needs to create a new video for each new event, to keep you informed, and not be a massive attachment with everything in it.

I've lost interest for now, so stopped the service, but on doing so I also discovered by accident, running tail -f, that sendemail has 47 old messages in its queue! No doubt from mail/vids it could not send.

$ tail -f /var/log/syslog
Aug 16 13:54:32 pinoir nullmailer[2195]: Starting delivery: protocol: smtp host: mail file: 1439080867.3457
Aug 16 13:54:32 pinoir nullmailer[2195]: Starting delivery, 47 message(s) in queue.
Aug 16 13:54:34 pinoir nullmailer[3743]: smtp: Failed: Connect failed
Aug 16 13:54:34 pinoir nullmailer[2195]: Sending failed: Host not found

I don't know how to remove these either, except try to uninstall sendemail, which did not stop the nullmailer queue.

apt-get remove sendemail

They may be in the mail queue?

mailq | wc -l
94

Hmm...

man mailq

INTRODUCTION
nullmailer is a simple and secure relay-only mail transport agent.
Documentation on how messages are reformatted and injected into the
queue can be found in nullmailer-inject(1). Documentation on how mes-
sages are actually inserted into the queue can be found in nullmailer-
queue(8). The process of sending queued messages is described in null-
mailer-send(8).

man nullmailer-queue

OTHER FILES
/var/spool/nullmailer/queue

vi /var/spool/nullmailer/tmp/45
4549 4551 4552

Let's remove these /tmp files:

$ rm -vr --i=no /var/spool/nullmailer/tmp/45*
removed `/var/spool/nullmailer/tmp/4549'
removed `/var/spool/nullmailer/tmp/4551'
removed `/var/spool/nullmailer/tmp/4552'

No - still there:

Aug 16 14:05:56 pinoir nullmailer[2195]: Starting delivery, 47 message(s) in que

$ rm -vr --i=no /var/spool/nullmailer/queue/1439*

Duh! It's a service, so just stop it:

Aug 16 14:06:58 pinoir nullmailer[2195]: Delivery complete, 47 message(s) remain.
Aug 16 14:07:59 pinoir nullmailer[2195]: Rescanning queue.
^C
$ service nullmailer stop
[ ok ] Stopping mail-transfer-agent: nullmailer.

Tail now shows:

Aug 16 14:09:01 pinoir /USR/SBIN/CRON[4768]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))

Hopefully that's stopped it. Remove motion from update-rc for now until I think of something else...or not...

I was looking at curl to send a txt msg instead of sendemail, but it did not work internationally for my phone even though Vodafone UK is listed, via:

DellMint stevee # curl -X POST http://textbelt.com/intl -d number=xxxxxxxxxx
d "message=Dellmint too?"
{
"success": true

17/8/15

More info on rollover:

http://www.lavrsen.dk/foswiki/bin/view/Motion/MotionGuideBasicFeatures

ffmpeg_timelapse_mode

  • Type: Discrete Strings
  • Range / Valid values: hourly, daily, weekly-sunday, weekly-monday, monthly, manual
  • Default: daily
  • Option Topic

The file rollover mode of the timelapse video.Note that it is important that you use the conversion specifiers in ffmpeg_filename that ensure that the new timelapse file indeed is a new file.

If the filename does not change Motion will simply append the timelapse pictures to the existing file.

The value 'Manual' means that Motion does not automatically rollover to a new filename. You can do it manually using the http control interface by setting the option 'ffmpeg_timelapse' to 0 and then back to your chosen value. Value 'hourly' rolls over on the full hour. Value 'daily' which is the default rolls over at midnight. There are two weekly options because depending on where you come from a week may either start on Sunday or Monday. And 'monthly' naturally rolls over on the 1st of the month.

gap

  • Type: Integer
  • Range / Valid values: 0 - 2147483647
  • Default: 60
  • Option Topic

Gap is the seconds of no motion detection that triggers the end of an event. An event is defined as a series of motion images taken within a short timeframe.Recommended value is 60 seconds (Default). The value 0 is allowed (but not recommended) and disables events causing all Motion to be written to one single mpeg file and no pre_capture. You can force an event to end and a new to begin using the http control 'http://host:port/thread_number/action/makemovie'. Disabling events has bad side effects on noise_tune and smartmask. Both features can only work properly outside an event. When gap is set to 0, both features don't work properly anymore.An event is defined as a series of motion images taken within a short timeframe. E.g. a person walking through the room is an event that may have caused 10 single jpg images to be stored. This option defines how long a pause between detected motions that is needed to be defined as a new event.The gap timer starts after the last motion is detected and post_capture images have been saved and appended to open movie mpeg files.Any motion detected before the gap timer times out resets the gap timer so it starts counting over again.

Detailed Description

The option 'gap' is important. It defines how long a period of no motion detected it takes before we say an event is over. An event is defined as a series of motion images taken within a short timeframe. E.g. a person walking through the room is an event that may have caused 10 single jpg images to be stored. Motion detected includes post_captured frames set by the 'post_capture' option. The 'gap' option defines how long a pause between detected motions that is needed to be defined as a new event. A good starting value is 60 seconds.The way 'gap' works in more technical terms is:

  • Gap is a timer that timeout 'gap' seconds after the last video frame with motion is detected.
  • If 'post_capture' is activated then the gap timer starts counting after the last image of the post_capture buffer has been saved.
  • The gap timer is reset and starts all over each time new motion is detected, so you will not miss any action by having a short 'gap' value. It will just create more events (e.g. more mpegs files)

The gap value impacts many functions in Motion.

  • When the gap timer runs out the event number is increased by one next time motion is detected. When you use the %v conversion specifier in filenames or text features this means that the number in filename or text increased by one.
  • The pre_capture feature only works at the beginning of an event. So if you have a very large 'gap' value pre_capture is not working very often.
  • When you make mpegs using the ffmpeg features, a new mpeg file is started at the beginning of an event when the first motion is detected. When 'gap' seconds has passed without motion (and post_captured frames saved) the mpeg files is completed and closed.
  • Do not use large gap values to generate one large mpeg4 file. If Motion stops working this mpeg4 file never gets properly completed and closed and you will not be able to view it.
  • Some of the tracking features sets the camera back to the center position when an event is over.

Note that 'gap' and 'minimum_gap' have nothing to do with each other.

I am just trying a shorter gap time between events, and a max movie size of 10 minutes - this will probably generate a lot more emails, but at the mo its sending me 1 x 15MB file!:

# Recommended value is 60 seconds (Default). The value 0 is allowed and disables
# events causing all Motion to be written to one single mpeg file and no pre_capture.
gap 30

# Maximum length in seconds of an mpeg movie
# When value is exceeded a new mpeg file is created. (Default: 0 = infinite)
max_mpeg_time 600

Just to try to prevent 1 frame bird shadows being a trigger, let's not get ALL motion:

# Picture frames must contain motion at least the specified number of frames
# in a row before they are detected as true motion. At the default of 1, all
# motion is detected. Valid range: 1 to thousands, recommended 1-5
minimum_motion_frames 3

You may want to reboot for a "lost" camera:

# Command to be executed when a camera can't be opened or if it is lost
# NOTE: There is situations when motion doesn't detect a lost camera!
# It depends on the driver, some drivers don't detect a lost camera at all
# Some hang the motion thread. Some even hang the PC! (default: none)
on_camera_lost reboot

Later...

Those changes did nothing to the appended timelapse, but it looks like the name is missing the event trigger, %v, in the conf file, so adding that may cause a new film for each consecutive event?  I'll try and retest, with the Hour added also.

# File path for timelapse mpegs relative to target_dir
# Default: %Y%m%d-timelapse
# Default value is near equivalent to legacy oldlayout option
# For Motion 3.0 compatible mode choose: %Y/%m/%d-timelapse
# File extension .mpg is automatically added so do not include this
timelapse_filename %v-%Y%m%d%H-timelapse

Also, I want the date in reverse order - easier to read:

# External Commands, Warnings and Logging:
# You can use conversion specifiers for the on_xxxx commands
# %Y = year, %m = month, %d = date,
# %H = hour, %M = minute, %S = second,
# %v = event, %q = frame number, %t = thread (camera) number,
# %D = changed pixels, %N = noise level,
# %i and %J = width and height of motion area,
# %K and %L = X and Y coordinates of motion center
# %C = value defined by text_event
# %f = filename with full path
# %n = number indicating filetype
# Both %f and %n are only defined for on_picture_save,
# on_movie_start and on_movie_end
# Quotation marks round string are allowed.

So, %v-%Y%m%d%H-timelapse, becomes:

%v-%H%d%m%Y-timelapse

or this sort of format for easy reading:

timelapse_filename %v-%Hhr%Mmin_%d_%m_%Y-timelapse

which gives the name:

01-18hr08mins_17_08_2015-timelapse.mpg

or - (but not a wise filename for Linux):

# You can put quotation marks around the text to allow
# leading spaces

OK - you cannot have multi - timelapse movies as separate event summaries by its nature! it's not for that. You always get an appended file.

ffmpeg_timelapse

  • Type: Boolean
  • Range / Valid values: 0 - 2147483647
  • Default: 0 (disabled)
  • Option Topic

[2147483647 is (2^32/2) -1, to account for the 0 between +ve and -ve 32 bit numbers]

http://www.lavrsen.dk/foswiki/bin/view/Motion/ConfigOptionFfmpegTimelapse

Create a timelapse movie saving a picture frame at the interval in seconds set by this parameter. Set it to 0 if not used.This feature uses ffmpegs libavcodec to encode a timelapse movie saving a picture frame at the interval in seconds set by this parameter. Setting this option to 0 disables it.The feature gives your viewer the chance to watch the day pass by. It makes a nice effect to film flowers etc closeup during the day. Options like frame_rate, snapshot, gap etc have no impact on the ffmpeg timelapse function.Note that the timelapse format is always mpeg1 independent of ffmpeg_video_codec. This is because mpeg1 allows the timelapse to stop and the file to be reopened and more film appended.

Comments are closed.

Post Navigation