Thursday, March 14, 2013

Monitoring a web page for changes using bash

There's this conference that I'd like to attend and I've heard that it's a hard-to-get-into type conference.  When I go to their site it doesn't have any new info.

Rather than checking the site every day, I'd like to have it monitored and be alerted when something new DOES appear on it.

Now I know there are services like ChangeDetection.com that can monitor it for me, but I was wanting to cobble something together with the tools I already have.  I'd also like to have the ability to customize what it consider "a change" at my disposal when/if I need it.

To that end, I threw together the following bash script.  It monitors a URL and if it detects a change, it sends an email to my gmail account letting me know.

Hope you find it useful.  BTW, I'm using a program called sendEmail to send the email notification.  It's in apt if you're using a debian/ubuntu-like distribution.

#!/bin/bash

# monitor.sh - Monitors a web page for changes
# sends an email notification if the file change

USERNAME="me@gmail.com"
PASSWORD="itzasecret"
URL="http://thepage.com/that/I/want/to/monitor"

for (( ; ; )); do
    mv new.html old.html 2> /dev/null
    curl $URL -L --compressed -s > new.html
    DIFF_OUTPUT="$(diff new.html old.html)"
    if [ "0" != "${#DIFF_OUTPUT}" ]; then
        sendEmail -f $USERNAME -s smtp.gmail.com:587 \
            -xu $USERNAME -xp $PASSWORD -t $USERNAME \
            -o tls=yes -u "Web page changed" \
            -m "Visit it at $URL"
        sleep 10
    fi
done

Then from a bash prompt I run it with the following command:

nohup ./monitor.sh &

Using nohup and throwing it in the background allows me to log out and have the script continue to run.

68 comments:

  1. It's exactly the script I was looking for. Thank you, Steve, for making it!

    ReplyDelete
  2. Cheers Steve, very handy script

    ReplyDelete
  3. Is it possible to send tweet (private message) instead email?

    ReplyDelete
  4. I find this tool for posting tweets from CLi
    A command-line power tool for Twitter.
    http://sferik.github.com/t

    ReplyDelete
  5. I edited this to use PushBullet to alert instead of email

    #!/bin/bash

    # monitor.sh - Monitors a web page for changes
    # sends a PushBullet push alert on change every 10 min

    PBTOKEN="PB ACCESS TOKEN"
    URL="PUT URL HERE"

    for (( ; ; )); do
    mv new.html old.html 2> /dev/null
    curl $URL -L --compressed -s > new.html
    DIFF_OUTPUT="$(diff new.html old.html)"
    if [ "0" != "${#DIFF_OUTPUT}" ]; then
    curl -u $PBTOKEN: https://api.pushbullet.com/v2/pushes -d type=note -d title="Site Changed" -d body="Visit it here $URL"
    sleep 600
    fi
    done

    ReplyDelete
  6. Hi,
    Thanks a lot for your script, very helpful
    Is it possible to check exact part (text, image, ...) in the page and not the entire as we can put it dynamical things ?
    Regards

    AyGitci

    ReplyDelete
    Replies
    1. You could... possibly using grep and it's extended regex, but I suspect you'd be opening a can of worms by trying to parse the HTML DOM that way.

      Delete
  7. you need a sleep 10 after fi too , otherwise you will cause loads of problems

    sleep 10
    fi
    sleep 10
    done

    ReplyDelete
    Replies
    1. +1
      If there is no changes, the script is spinning in a loop without any delays, causing significant performance issues.

      Delete
    2. It did look like it was spamming the request non-stop, this extra sleep 10 fixed it after the if loop.

      Delete
  8. Since you're pulling the diff.. you may as well add the diff to the body of the email.

    ReplyDelete
  9. Very useful article about Absolute great. Thanks.
    Business Directory India

    ReplyDelete
  10. im getting this error in ubuntu 14.04 LTS

    check.sh: 10: check.sh: Syntax error: Bad for loop variable

    help me

    ReplyDelete
    Replies
    1. Perhaps 3 years to late : ) Maybe it will help anyone other... Check if you start in the first line (#!/bin/bash) with the # or if before this a space. Delete this blank space and it will be fine : )

      Delete
  11. Good post....thanks for sharing.. very useful for me i will bookmark this for my future needs. Thanks.
    New Branded Laptops and Desktops In Delhi

    ReplyDelete
  12. hie steve, i am looking for a script simple has your but to actually check a page where you need to log in first, any idea ?

    thx

    ReplyDelete
    Replies
    1. Have a look at curl's '-u' argument in the man page.

      Delete
  13. Thank you so much! I've been looking for a script like this. The only problem I'm running into is that it keeps saying that sendEmail, new.html, and old.html do not exist... I do have these but for some reason I keep getting these messages.

    ReplyDelete
  14. Hi Steve,

    Thanks for the script.

    Is it possible to monitor multiple websites? What triggers the on/off button for this script? How would I have the script continually run every 1 second?

    ReplyDelete
  15. Hi Steve,

    Thanks for the script.

    Is it possible to monitor multiple websites? What triggers the on/off button for this script? How would I have the script continually run every 1 second?

    ReplyDelete
    Replies
    1. Assuming you're running on a Linux system, you'd probably want to use cron (type 'man crontab' from a bash prompt). Cron tab run the script for whenever and however often you'd like. As for monitoring multiple sites, just run multiple versions of the script.

      Delete
  16. Hi there! I found your script very very useful. I modified to send DM on twitter using oysttyer and it works great.
    Thank you very much.

    ReplyDelete
  17. PLEASE fix the initial script to move the "sleep 10" outside of the email loop. This is obviously not the right place for it. It belongs *after* the fi, or it will HAMMER whatever URL you are pointing it at repeatedly, as fast as it can, and only pausing for 10 seconds after it detects a change and sends an email. Obviously not the behavior you intended.

    ReplyDelete
  18. This was very helpful, thanks! Echoing the above, MAKE SURE you move the sleep to outside of the if...fi loop. Also, I found it helpful to strip the HTML tags out of the page before doing the diff. This allows me to ignore structural changes and things like ad network identifiers and GUIDs:

    mv new.html old.html 2> /dev/null
    curl $URL -L --compressed -s | sed 's/<[^>]\+>//g' > new.html
    DIFF_OUTPUT="$(diff new.html old.html)"

    ReplyDelete
  19. Hello, Please tell me where to run this script. I am using windows.

    ReplyDelete
  20. Hello Steve, Is it possible to run this in windows?

    ReplyDelete
  21. A similar service for monitoring web-pages when you are too busy to do it yourself. Just kick-start it and tell which page to monitor and it will silently keep doing it and send you an email! You can set up-to 5 monitors per users for free!

    https://lexybit.com/webpage-monitor/

    ReplyDelete
  22. You haven't heard from your guy in a few days and he doesn't respond to 800. You're frustrated and aggravated at the way he's treating you. You feel insignificant and angry with him.

    ReplyDelete
  23. What I'm looking for is an [ideally free] way to place a box on a page on my own website so that visitors can enter their email (rather than me enter mine) to be informed when that site changes. I used to use a site called changealarm.com, but it seems to have closed shop. Any suggestions?

    ReplyDelete
  24. I'm getting "Permission denied" in the nohup.out file. Any idea why? Username and pw are correct.

    ReplyDelete
  25. Hello,

    Can I have the same script with windows.

    ReplyDelete
  26. For non-ascii characters, like [é, æ, ø, å] in an email message body, add the command-line parameter '-o message-charset=utf-8'.

    ReplyDelete
  27. Hey! Do you use Twitter? I’d like to follow you if that
    would be ok. I’m undoubtedly enjoying your blog and look forward
    to new posts.
    espn.com/activate

    ReplyDelete
  28. Thanks very much Steve, this was really helpful. For anyone who wants something a bit simpler to implement although not as flexible, the script can be modified to sound an audible alert when a change is made (no sendmail app is required). This is for a Mac. I'm not sure how portable it is for other platforms but maybe someone else can modify it if needed.

    #!/bin/bash

    # monitor.sh - Monitors a web page for changes and sounds an alert on a Mac

    # If you prefer a system alert, change "say changed"
    # to: afplay /System/Library/Sounds/Funk.aiff

    URL="URL="http://thepage.com/that/I/want/to/monitor""

    for (( ; ; )); do
    mv new.html old.html 2> /dev/null
    curl $URL -L --compressed -s > new.html
    DIFF_OUTPUT="$(diff new.html old.html)"
    if [ "0" != "${#DIFF_OUTPUT}" ]; then
    say changed
    sleep 10
    fi
    sleep 10
    done


    ReplyDelete
    Replies
    1. Hi there - I was wondering if there's a way to check if this is working? I'm new to bash scripts and want to know if I implemented it correctly.

      Delete
  29. Thanks for sharing this informative content , Great work
    To crack scrum master interview : Scrum Master Interview Questions

    ReplyDelete
  30. Thanks for sharing this informative content , Great work
    Leanpitch provides online training in Scrum Master during this lockdown period everyone can use it wisely.
    CSM Certification online

    ReplyDelete
  31. Thanks for sharing this informative content , Great work
    Leanpitch provides online training in Advanced Scrum Master during this lockdown period everyone can use it wisely.
    Advanced Scrum Master Training Online

    ReplyDelete
  32. Android Application Development is popular nowadays. People download free Android Apps in hordes and it is easy to develop apps in it as it s open source and extremely flexible for making apps tailored to different requirements and needs. spy phone

    ReplyDelete
  33. Thanks for sharing this informative content , Great work
    Leanpitch provides online training in Scrum Master during this lockdown period everyone can use it wisely.
    ICP-CAT certification

    ReplyDelete
  34. The practice of technology transfer can greatly benefit an organization. What is technology transfer? Technology transfer is the sharing of technology between two or more organizations. mobile tracker

    ReplyDelete
  35. Thanks for sharing this informative content , Great work
    To crack scrum master interview : Scrum Master Interview Questions

    ReplyDelete
  36. Thanks for sharing this informative content , Great work
    To crack Agile coach interview : Agile coach Interview Questions

    ReplyDelete
  37. A thoughtful insight and ideas I will use on my website. Youve obviously spent some time on this. Congratulations! Mega888 apk download

    ReplyDelete
  38. Thanks for sharing this informative content , Great work
    Leanpitch provides online training in Enterprise Agile coaching during this lockdown period everyone can use it wisely.
    Enterprise Agile Coaching

    ReplyDelete
  39. The practice of technology transfer can greatly benefit an organization. What is technology transfer? Technology transfer is the sharing of technology between two or more organizations. camera spy

    ReplyDelete
  40. Thanks for sharing this informative content , Great work
    Read this blog to know about : Definition of done

    ReplyDelete
  41. Dude.. My group is not considerably into looking at, but somehow I acquired to read several articles on your blog. Its fantastic how interesting it’s for me to visit you fairly often. 스포츠토토

    ReplyDelete
  42. 918Kiss Casino (Kiss918) is an international online casino company by a registered SOLEI in Island Casino Ltd. wholly owned and operated. All of our games are ...

    Best Regards:
    918kissvault.com

    ReplyDelete
  43. Group Buy SEO Tools | 875 followers on LinkedIn. We are the best Group Buy SEO Tools Provider in the world and most advanced All-In-One SEO Tools.
    Best Regards:
    buy seo tools group

    ReplyDelete
  44. Proven SEO Services From Malaysia's Leading SEO Company. No Long Term Contracts. Lots of #1 Results. For Fast & Long Term Rankings. Go Now! Consistently. Profitable Traffic.

    Best Regards:
    Seo Company Malaysia

    ReplyDelete
  45. SEO group buy tools allow access to many SEO tools for a low cost. If you're wondering whether they are safe,

    ReplyDelete
  46. Thanks for sharing this.,
    Leanpitch provides online training in Scrum Master during this lockdown period everyone can use it wisely.
    Join Leanpitch 2 Days CSM Certification Workshop in different cities.


    CSM online certification

    ReplyDelete
  47. Thanks for sharing this informative content.SEO tools will help you to make sure that you optimize your website and increase your ranking on search engines such as Google. SEO tools can give you different tips and information on how you should improve your site so that you can get more traffic and better ranking. You can choose from different software and software available online, some of which are free while others cost you money. Group Buy SEO Tools

    ReplyDelete
  48. Music Videos That Will Make You Fall In Love Again here is Top 5 Romantic Music Videos. These videos I present to you are my favourite romantic music videos and maybe even my favourite videos ever. I hope they will make you feel something, just like they did to me. May these videos be an evidence that true love is not a lie. Top 5 Romantic Music Videos

    ReplyDelete
  49. I respect your work, regards for all the interesting blog posts.

    ReplyDelete
  50. I respect your work, regards for all the interesting blog posts.

    ReplyDelete
  51. That's really a good idea to make money online with John Crestani Course

    ReplyDelete
  52. It’s really a great and useful piece of info. I am satisfied that you just shared this helpful info with us. satta matka

    ReplyDelete
  53. I have learned huge knowledge from here. I am waiting for your updates
    Regards: "satta matka"

    ReplyDelete
  54. Thanks for sharing this information.
    Yes Done is the best online site for on demand home services, professional service providers in Jaipur, Rajasthan. We are providing top services like carpenter, cleaning, TV, and many more services at your doorstep in Jaipur, Rajasthan. For more services visit our website.

    ReplyDelete
  55. Thanks for sharing this information.
    Pheeta is the best online shopping sites for Salwar suits for women and woman Kurti online in Jaipur, Rajasthan. To shop more latest fancy and attractive dresses and Kurtas visit us at our website.

    ReplyDelete