hawler's Introduction
# $Id: README 16 2008-03-23 06:02:07Z warchild $ Hawler, the ruby HTTP crawler. Written to ease satisfying curiousities about the way the web is woven. Install by: gem install --source http://spoofed.org/files/hawler/ hawler Rough usage is... define a method that will take a URI, a referer, and an Net::HTTPResponse as arguments, and then does whatever it wants. Create a Hawler object, passing the start URI and the above method as arguments. Set Hawler options, then start it. Basic usage is an implementation of htgrep: #!/usr/bin/ruby require 'rubygems' require 'hawler' require 'hawleroptions' def grep (uri, referer, response) line = 0 unless (response.nil?) response.body.each do |l| line += 1 if (l =~ /#{@match}/) puts "#{uri}:#{line} #{l}" end end end end options = HawlerOptions.parse(ARGV, "Usage: #{File.basename $0} [pattern] [uri] [options]") if (ARGV.empty?) puts @options.help else @match = ARGV.shift ARGV.each do |site| crawler = Hawler.new(site, method(:grep)) options.each_pair do |o,v| crawler.send("#{o}=",v) end crawler.start end end
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.