Hydra Web Crawler entry for TwilioCon

Welcome to the Hydra Web Crawler entry for the Twilio Hackathon Conference

Watch my Video Entry:


 

Details:

This is a C language distributed web crawler.  I am using the Redis NoSql database to coordinate multiple instances of this web crawler on different internet connections.  I am using the new Dropbox API to synchronize the downloaded files on each computer.

Code:

def main(prog_name, args):
    term = DropboxTerm(APP_KEY, APP_SECRET)

    if not term.sess.is_linked():
        try:
            term.sess.link()
        except rest.ErrorResponse, e:
            term.stdout.write('Error: %s\n' % str(e))
            sys.exit('could not link')

    from_file = open(os.path.expanduser(sys.argv[1]))
    term.api_client.put_file(term.current_path + "/" + sys.argv[1], from_file)
    print "Uploading to Dropbox API"

 

Screenshot:

 

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>