LWAC
About
LWAC is a tool for constructing corpora from web data. Simply, it is a system for retrieving many URLs in as short a time as possible, and storing the results in a useful way.
The main focus is on longitudinal sampling (though single samples are possible), thus LWAC is capable of running uninterrupted for many months or years. It supports theoretically infinite corpus sizes (with static memory usage), and is designed to be particularly aggressive at link retrieval in order to minimise time differences in samples. The main features run to:
- A highly parallel downloading system—LWAC can sustain tens of megabytes a second throughput, given a sufficiently colossal internet connection.
- Static memory usage—Clients and servers respect configurable limits on their memory usage, and never load even index lists in full.
- Longitudinal corpus structure—Corpora are stored with a ‘repeated samples’ structure and full information on time, duration etc.
- Flexible endpoint configuration—LWAC can be configured in such a way that remote servers are never aware they are being hit by automated tools.
- A scalable deployment size—LWAC can be deployed across multiple machines in order to maximise the utilisation of network resources or monitor differences in hosts’ access.
The suite consists of four tools—A client/server pair for downloading data in a distributed manner, and an import/export system for entering URLs and extracting data from the resultant corpus.
Slides
LWAC was presented at the WaC8 workshop using these slides.
Dependencies
LWAC is written in Ruby >= 1.9, and requires some things to run:
- SQLite3 or MySQL/MariaDB
- libcurl
- Some ruby gems (installed automagically)
Download
LWAC is distributed as a gem. It is thus available to install by running the
gem install lwac
command in your shell.
The source code can be downloaded from the git summary page.
Bugs
Please report bugs and feature requests by getting in touch, such as email.
Use
At a high level, use of LWAC can be broken down into four stages:
- Selection of links for a given study
- Importing into a new corpus
- Running the client/server tools for the study period
- Exporting data in an operationalised format
Each of these stages is covered in more detail in the bundled documentation, which is currently available in the doc/user directory, the gem’s built-in documentation, or included in the download.
Technical Documentation
The project is split into libraries to manage storage and data formats, atomic commits and consistency, and user inteface functions. All code is fully commented and annotated in detail, and integrating new functions into the codebade should be relatively straight-forward. If you wish for any features or edits, or have made some yourself, please don’t hesitate to get in touch