
Web Spider
----------
The project was originally designed to send HTTP DELETE commands to as many pages as possible, in the hope that at least
one would respond - nihilistic but fun.  Obviously this doesn't really work nowadays, so it was converted to do anything
with links.

This project is thus a very aggressive and irresponsible spider for links - In its current form the python outputs all
links it finds across pages.  The accompanying shell script then downloads all of those links, making it suitable to
download full sites or even big swathes of the internet.

Use
---
First, edit deletinator to add a custom URL.  Then either run it from bash with ./deletinator or inerpret it with python
using 'python deletinator'.  This will output its list to the terminal.

If you want to download the full site then simply run ./sitegrabber.sh, which will run deletinator and then wget.  This
is amazingly network intensive, and can DOS me off my own my connection...

License
-------
All of this is my work, no complications...
