Blat

About

Blat is a ruby library that wraps cURL and offers high-speed, sustained downloads of unlimited numbers of web resources in a highly parallel fashion. It was originally designed as part of LWAC, but is entirely general and may be used to make any curl request.

Blat works by providing an easy interface to Curl::Multi, offering an easy way to download lists or maintain a worker that consumes links from a producer class. This allows it to make requests indefinitely, and sustain massive throughputs to external servers.

Download

Blat is distributed as a gem. It is thus available to install by running the gem install blat command in your shell.

The source code can be downloaded from the git summary page.

Use

Blat offers two modes of use. The most powerful is the Blat::Pool interface, which presents a way of controlling the pool of workers directly. The easiest to use is the Blat::Batch class, which wraps the pool to provide simple batch operation.

Pool

A workflow for the pool would be to construct a pool (telling Blat where to output in the process), provide some work, and then call perform with some code to execute whilst it runs:

require 'blat/queue' 

p = Blat::Queue.new(100, false)
1000.times{ |n|
  puts "Added #{n}"
  p.add( get_url ){ |res|
    puts "#{res.url} complete."
  }
}

p.perform{
  puts "#{p.request_count} requests active..."
}

Batch

Batch operation is altogether simpler, and maintains its dispatch system for jobs:

require 'blat/batch'

results = Blat::Batch::run(10, list) do |c| 
  # Configure curl in here
  c.follow_location = true 
  c.on_complete{ |res|
    puts "#{c.url} complete!"
  }
}

puts "Results: #{results}"