websitary 0.2.1

DESCRIPTION: websitary (formerly known as websitiary with an extra "i") monitors webpages, rss feeds, podcasts etc. It reuses other programs (w3m, diff, webdiff etc.) to do most of the actual work. By default, it works on an ASCII basis, i.e. with the output of text-based webbrowsers like w3m (or lynx, links etc.) as the output can easily be post-processed. With the help of some friends (see the section below on requirements), it can also work with HTML. E.g., if you have websec installed, you can also use its webdiff program to show colored diffs. This script was originally planned as a ruby-based websec replacement. For HTML diffs, it stills relies on the webdiff perl script that comes with websec. By default, this script will use w3m to dump HTML pages and then run diff over the current page and the previous backup. Some pages are better viewed with lynx or links. Downloaded documents (HTML or ASCII) can be post-processed (e.g., filtered through some ruby block that extracts elements via hpricot and the like). Please see the configuration options below to find out how to change this globally or for a single source. == FEATURES/PROBLEMS: * Handle webpages, rss feeds (optionally save attachments in podcasts etc.) * Compare webpages with previous backups * Display differences between the current version and the backup * Provide hooks to post-process the downloaded documents and the diff * Display a one-page report summarizing all news * Automatically open the report in your favourite web-browser * Experimental: Download webpages on defined intervalls and generate incremental diffs. ISSUES, TODO: * With HTML output, changes are presented on one single page, which means that pages with different encodings cause problems. * Improved support for robots.txt (test it) * The use of :website_below and :website is hardly tested (please report errors). * download => :body_html tries to rewrite references (a, img) which may fail on certain kind of urls (please report errors). * When using :body_html for download, it may happen that some JavaScript code is stripped, which breaks some JavaScript-generated links. * The –log command line will create a new instance of the logger and thus reset any previous options related to the logging level.