Hydra Web Crawler entry for TwilioCon

Welcome to the Hydra Web Crawler entry for the Twilio Hackathon Conference

Watch my Video Entry:


 

Details:

This is a C language distributed web crawler.  I am using the Redis NoSql database to coordinate multiple instances of this web crawler on different internet connections.  I am using the new Dropbox API to synchronize the downloaded files on each computer.

Code:

def main(prog_name, args):
    term = DropboxTerm(APP_KEY, APP_SECRET)

    if not term.sess.is_linked():
        try:
            term.sess.link()
        except rest.ErrorResponse, e:
            term.stdout.write('Error: %s\n' % str(e))
            sys.exit('could not link')

    from_file = open(os.path.expanduser(sys.argv[1]))
    term.api_client.put_file(term.current_path + "/" + sys.argv[1], from_file)
    print "Uploading to Dropbox API"

 

Screenshot:

 

Posted in Uncategorized | Leave a comment

Perfect Time Machine Server (.iso)

Question?

Do you have a Mac or MacBook that doesn’t have backups?  What would you do if you lost all the data on your harddrive?

Answer!

Enter the Perfect Time Machine Server! This is a light weight Linux distribution will act as a network Time Machine Server and backup your Mac.  This linux in meant to run on it’s own computer with no maintenance or hassle.  You can even run it inside a Virtual Machine if that suits you.  It’s also FREE!

 

Some Background…

Here I am releasing the first version of my Perfect Time Machine Server!  This linux distribution comes with all the software needed to act as a server for your Mac.  It doesn’t have a GUI interface, and only runs on the command line.  All the steps are easily explained on the introduction screen, so don’t worry if your not good at using Linux.  You should have a large harddrive to store backups.  About 500 gigs will be fine.  If your mac has a large harddrive, you will need to account for that with your backups disk.  By default the Perfect Time Machine will use this entire drive for backups.  Below you will find the .ISO or virtual machine download.

Download

ISO Version: Time_Machine_Server.i686-1.0.5.iso

Vmware Version: Time_Machine_Server.i686-1.0.5.vmx.tar.gz

 

Instructions

If your not sure which version to use, go with the ISO version.  Burn it to a CD using a slower burning speed.

  1. Insert CD into your computer and boot
  2. Login using the username “root”, and password “timemachine”.  Type “./help” for instructions or keep reading.
  3. Type “yast” and choose “Miscellanous”.  Press tab to highlight “Live Installer” and press enter.  Follow the instructions.  Reboot.
  4. From your Mac, open a terminal.  (Press Command+Space, then type Terminal)  Inside the terminal type: “defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1″
  5. From your Mac, click on the desktop.  Press Command+K.  Type in the following, but using the ip of your server: “afp://192.168.1.100″   If you don’t know the ip of your Time Machine Server, type “./help” from the command line.  Use the username “tux” and password “linux”
  6. Now that you have connected to the Server, you need to configure Time Machine.  Go to System Preferences, and choose Time Machine.  Click “Select Disk” and choose the time machine volume.
  7. All done! Your Mac will start backing up automatically

 

More Configuration

  • You will probably want to keep the Time Machine at a fixed IP.  To do this, type “Yast” and select “Network Devices” and then choose “Network Settings”.  When the page loads, press the right arrow key twice.  This takes you to the “routing tab”  from here you can put in your routers address.  Now tab back to the top and press left twice to get back to Overview.  Tab to Edit, and choose “Statically assigned IP Address”.  Fill in an fixed ip.  You can probably use “255.255.255.0″ for the subnet mask.  You can also choose a hostname here.
  • To configure where backups are stored, edit the symlink in the home directory by typing “nano AppleVolumes.default”.  The very last line is an entry for the default share, which is /macbackups  Change this line to backup to a different location.
  • If you have two harddrives, one large and one small, you can still install the Time Machine server.  In the installer, mount the larger drive at the moutnpoint /macbackups.  You can also use a different mount point if you want, just be sure to change the share as

 

Posted in Uncategorized | Leave a comment

Suse Studio…. to the cloud!

http://susestudio.com

I have nothing but good things to say about Suse Studio.  Wow.  But before we get started with this amazing product, lets talk names.

Suse“.  How do you say that?  I know that Bryan Lunduke of http://lunduke.com/ pronounces the name, “su-sa“.  I have also heard people pronounce it “soose“, which makes me think of the word “sues“;  For instance, have you seen these headlines in the news!??  Sony suse GeoHot for hacking the PS3.   But I digress…

So whats the big deal bout Suse Studio anyways?  Well basically it’s a cloud service that lets you build and test your own Suse Linux distribution.  You can configure your own packages, choose a background and startup logo, and then download an .ISO of your very own personalized linux.  So lets get started.

Account / Login

To get started, head over to http://susestudio.com/login.  They support a surprising number of login methods, with google being my favorite.  After logging in, click on the studio link in the upper right-hand corner.  Now click “New Appliance”

Base System

You should be at the “Choose a base template” screen.  For now I will just be sticking to the topmost group, “openSUSE 11.4″ in my case.  I recommend that you choose either “GNOME desktop”, or “KDE 4 desktop” at this point.  If you are hardcore (like me) choose “Server”.  Scroll to the bottom of the page and pick either 32/64 bit mode.  Click “Create appliance”.

Config

After this you will be taken to the welcome screen.  This screen is pointless, and I personally think they should just link you to the next place we are going: the “Software” tab.  Click the Software tab now!

What you see before your eyes is a basic package manager.  Search for package names under the “Search for Software” heading.  Anything you add here will be present on the final live cd.  Some packages you should consider adding:

  • iputils  (this includes the ping command)
  • nmap    (look for all active hosts on ur network with) nmap -sP 192.168.1.2-255
  • iptraf    ( run this to see all inbound/outbound network connections and their speeds)
  • screen  ( lets you detach and re-attach a console for overnite or otherwise longterm commands)
  • mdadm ( add for raid support )

Select and add all the packages that you think your distro will need.  Go to the next tab, “Configuration”.  Here you can configure user accounts, the bootup look, and the startup mode (Normal Console or Graphical Login).  I won’t cover this tab too much because it’s pretty simple.

Build

For now, skip the next tab and go straight to “Build”.  After the build finishes, click “Testdrive“!!  Now here is where the magic happens.  No really, they have a wizard who literally waves his wand when you click the testdrive button.  His beard powered magic will create a clone of your new baby distro, and shock it to life with 1.21 Gigawatts! A new tab will open and you can watch as the machine boots live in the cloud.  The new tab has an integrated VNC viewer which directly displays your machine.  You can control it just like you would a normal computer with your keyboard and mouse.  This is all made possible because Suse Studio spins up a new Virtual Machine in the cloud, running an instance of the distribution that you just configured.  It’s quite impressive.

On the build page there are multiple build types listed under “Default Format”.  My favorite type to download is “Live CD/DVD (.iso)”, however all the types listed can run the testdrive.  I like the live cd because is my favorite because it seems to be the best for importing into XenServer (or other unlisted distributions).  It does list Xen, however this is actually different than XenServer.  The images tend to not be compatible.  If you know an easy way to convert from a Sude Studio build to a .XVA file, please let me know.

Conclusion

My final review? Awesome! I couldn’t be happier with this service.  I can invision many different uses for this website.  There are also tons of options that I won’t cover in this post, but I should mention just a few.  You can add users and set their passwords.  You can upload your own files, including .TAR.GZ.  You can set build scripts to extract, make, install your tar during the “Build” state (different than run time).  And So. Much. More.

 

Posted in Uncategorized | 1 Comment

Fridays…

Sometimes you just want friday to be over.  Well actually I always want that.  Today was one of those days.

Today I had two projects planned for work.  Together they would have taken just enough time to fill my day.  But that didn’t happen.  About an hour after getting into work, the power went off, and the boss was really upset that he couldn’t use his computer.  So what did I do?  Inverter baby!  I pulled my car outside his window, slipped a 150W inverter into the cigarette lighter, and ran a cord through his window!  That’s right, Awesome.

Oh and what else did I do?  Well, after the power came back on I boiled some water….for science!  I guess it all comes with the job :)

Posted in Uncategorized | Leave a comment

First Post

Welcome everybody! I will be using my first post to introduce myself and talk about the turbo-spring tech blog.

My name is Ben.  I am the creator, owner, and director of turbo-spring.  I am a technology enthusiast with a background in Linux, embedded C, user interfaces, virtualization, PHP, perl and more!  You will probably see posts in all these areas and more!

Well that’s all for now, stay tuned for more tech blogging!

Posted in Uncategorized | Leave a comment