The Vitriol Generator

The Vitriol Generator was a small web app that let you see what the Daily Mail thinks of you — or of any topic you cared to name.

Retired

This one used to run live: it scraped the Daily Mail on demand, so it stopped working once the site changed underneath it. What follows is a write-up of what it did, rather than a working tool.

What it did

You typed a topic into a box — immigrants, europe, health and safety, that sort of thing — and the app went off, searched the Daily Mail, found articles on that subject, and pulled back a random selection of reader comments so you could learn from the wisdom of the commentariat. By default it served up the worst-rated comments, though you could ask for the best instead (it made surprisingly little difference).

How it worked

Under the hood it was a small Ruby CGI script that doubled as a JSON web service. Given a topic it would:

  1. Hit the Daily Mail’s search to find recent article IDs on that topic.
  2. Pick articles at random and fetch their comment threads via the site’s own comment API, sorted by rating.
  3. Return a random handful of comments — the highest- or lowest-rated, depending on your mood.

As well as the web front-end, the same endpoint spoke JSON if you asked it to, taking these parameters:

  • topic — the topic to search for (between 3 and 100 characters).
  • format — web (the front-end) or json.
  • offset — how far from the best/worst end to start. Default 0.
  • count — how many comments to sample from each article. Default 10.
  • number — how many articles to sample. Default 5.
  • best — true to return the best-rated comments, false (the default) for the worst.

Because it made quite a few HTTP requests in the background per query, it was never fast, and it politely asked callers not to hammer it.