JavaScript, Perl, IRC, and FourSquare

This week I wrote a script to check myself into foursquare venues. Being a C++ programmer for whom every microsecond counts, I don't get much opportunity to play around with scripted languages or internet services. So to accomplish this task, I had to teach myself enough JavaScript including using jQuery and JSON to interact with foursquare's API.

(By the way, I was going to put a foursquare logo here, but I read the trademark page and was so disgusted that I don't think I will)

Foursquare uses OAuth 2.0 to let a user give an application access to their foursquare account. This requires registering the application to get a ClientID, which is as simple as coming up with a name, and then using that ClientID in conjunction with the user's cooperation to generate an authentication token, which can then be used indefinitely to access the user's account. To reduce the risk of the application being spoofed, foursquare also requires a URL during registration, which it calls back during the authentication process.

Because of this callback model, I have to have a real web page on a publicly-available server. I have my markmccracken.net domain hosted primarily for email and this blog, and it does allow HTML-serving, but no server-side scripting like PHP. So that means all the authentication work has to be done in client-side scripting like JavaScript.

The beautiful and media-rich page I whipped up to accomplish the authentication is here. I've had spotty results using this from IE, but it seems pretty reliable under Chrome and Firefox (presuming you have JavaScript turned on, of course). Including the authentication, this page accomplishes a few one-time tasks:

1. Authenticate, as mentioned, getting an access token I can use indefinitely.
2. Search the user's lists on foursquare for one called "My Favorites".
3. Iterate all the venues in that list, putting them and some other helpful defaults into a text box that can be copy/pasted to prime a configuration file.

I like having my long-running scripts talk to me via IRC, because I can passively monitor them from anywhere, and even interact with them in a rudimentary manner. For this project, I used Bot::BasicBot from CPAN, which I've used before to hook a Perl script up to IRC. It's a nice Perl library in the form of a base class where most of the interaction is done in an event-driven manner via the use of callbacks. The two main callbacks for my purposes are tick() and said(). tick() gets called when a timer expires, and said() gets called whenever anyone says anything on the IRC channel. With a little state management and content filtering/parsing, the bot can do pretty much whatever it needs to do with these two callbacks. It can also speak onto the IRC channel by using the say() method.

CPAN is far less scary than The FoursquareTM Site

The only thing left is the meat of the task, which is to check myself into a foursquare venue, wait for a while, and then repeat on the next venue. Since all the venues are pre-loaded into the configuration file, the Perl script just has to iterate them at extremely low speed (one checkin an hour) and make a POST request for each one that actually accomplishes the checkin. For that request, I used the HTTP::Tiny package, because it's usually pre-installed and is very easy to use. Since foursquare requires that POST request to be over https, the IO::Socket::SSL package must also be available.

If you want to try this thing yourself, follow the instructions below for your operating system. I actually have Windows instructions this time because I helped some POWs (Prisoners Of Windows) get it working in their cells. Either way before you begin, create a list on your foursquare account called "My Favorites" and put into it whatever venues you want to automatically check into.

Linux
Perl is probably already installed, but you'll probably need some or all of the extra packages. The cpan utility comes with Perl (at least on Arch), so use it to install the required packages.

sudo pacman -S cpan
cpan Bot::BasicBot
cpan HTTP::Tiny # probably unnecessary
cpan IO::Socket::SSL

Next get the perl script here, and save it somewhere convenient. Give yourself execution permissions.

chmod u+x foursquare_irc.pl

Go to the media-rich and highly advanced authentication page mentioned earlier, and get your configuration starter. Copy it from the web page and paste it into a text file such as foursquare.cfg. Then jump down to the IRC-specific settings described after I repeat these instructions to those Winlozers in small simple words.

Windows

Since you are trapped in Windows, you have no Perl pre-installed. So first, go get Strawberry Perl. Be sure to get the right architecture for your machine: 32-bit or 64-bit. If you're not sure which one you have, then you probably don't want to go any farther with this project anyway. Strawberry Perl was written for Windows users, so it chooses bloat over customization. That means you don't need to install any of those packages I mentioned earlier, because Strawberry Perl pre-installs all of them for you. And yes, that strawberry on their home page is HUGE.

Next get the perl script here, and save it somewhere convenient that you can find again, like your Desktop. In the same place, create an empty text file such as foursquare.txt. In that file, paste the contents of the text box on the media-rich and highly advanced authentication page mentioned earlier. Add the IRC-specific settings described below after I insult some Mac people.

Mac
Go away.

IRC Settings
The highly advanced and media-rich webpage doesn't talk about IRC settings, because there is another version of this script for IRC-impaired people that doesn't use IRC, and I didn't want to confuse them with excess settings. So you'll need to manually add the following settings. Note that none of the stuff in parentheses is real - you have to come up with your own.


IRCChannel= (your IRC channel here, such as #RiskOfRuin)
IRCServer= (your IRC server here, such as irc.foonetic.net)
IRCPort=6667 (probably right, check your server)
IRCNick= (a probably-unique nickname for this script, such as oexkubhtn234)
IRCBoss= (YOUR nick on IRC - the bot only takes orders from this nick)

Run That Sucker

With all the settings in place, go connect your own nick to the IRC channel. Then execute the script:

(Linux) ./foursquare_irc.pl foursquare.cfg

(Windows) .\foursquare_irc.pl foursquare.txt

Yes, Windows people, you're going to need to open a command window. Sorry.

If you've configured everything correctly, you'll see the bot connect to your IRC channel and report its readiness. To interact with it, precede your command with its nick: oexkubhtn234 quit. Simple. It responds to the following commands:

• quit - quits.
• start - starts at first venue after waiting [0 .. MaxSlop] minutes.
• start n - starts at nth venue.
• status - tells you what it's up to in case you forgot.
• settings - reports its settings and venues.

Please use this script responsibly!

Random Hubble Desktop Wallpaper

As I settle into my new Arch workstation, I find myself going through all the little nesting behaviors that any user would demonstrate. Only it's a little more challenging this time around, because instead of a user-coddling operating system like Windows or Ubuntu, I'm using Arch Linux. And instead of using a familiar desktop environment like Gnome, I'm using KDE 4.8. So the most basic of things provides an opportunity for learning. For example, I spent a good 30 minutes trying to figure out how to set up the screensaver the other day. It's easy, and very similar to how every other operating system does it; but I just didn't know where to look.

One of my computing goals this year is to stop taking the features and options spoon-fed to me by the desktop environment and instead be willing to step up and exert my own will on the environment. Arch/KDE is a perfect combination for this, because they are both all about basic building blocks and customization. In my previous installation (Ubuntu 10.10), I had a little python script that I grabbed from Christian Stefanescu that will download the NASA image of the day and make it your desktop wallpaper. I still have that script, and I could use it with a few modifications for KDE instead of Gnome. But NASA's image of the day is hardly ever cool things like stars and galaxies, and instead is usually pictures of people smiling for the camera, or close-ups of a really important and boring piece of metal. I want something with some curb appeal.

I looked into a few other daily-image sites, like Alta Ski Area and National Geographic, but they just aren't designed to be wallpapers - the resolution is too small to look good at 2048x1152, and they're frequently in portrait-mode, which looks terrible on my widescreen monitor. Of course, Murphy's Law dictates that today when I go look at those sites to get the links above, they both are showing images that would look great as wallpapers. Trust me, they're the exception, not the rule. But I digress.

The official Hubble Site has a bunch of fantastic images in its gallery, but they don't pick one a day for you. There is a main page that shows thumbnails of all the images, a resolution-selection page when you click on a particular image, and finally another page that displays the image inline. I figured I could probably handle this with a Perl script, so I grabbed the HTML::Parser package from CPAN to do the heavy lifting, and did a little reading of its description page. It treats the incoming HTML document like a hierarchical tag system, just like most XML parsers do. I've never used an HTML parser before, but I wouldn't be surprised to learn that this is standard - it makes the most sense. You start by deriving your own class from HTML::Parser, and write start() and end() callbacks. Then as each tag (<div>, <a>, <img>, etc) is opened by the parser, it sends interesting info to the start() callback. When it closes, it calls the end() callback. In this way the entire tree of HTML is traversed, in a depth-first fashion.

Wallpaper Thumbnails

Resolution-selection

2048-wide image

Being a Perl newb as well as an HTML::Parser newb, I did the easiest possible thing, which was to define three different classes - each tailor-made for one of the three pages that were required for finally getting the image. There is no error-checking or customization capability - paths are all hard-coded, and if anything goes wrong the script will probably just crash. But that means my wallpaper won't change for a couple days, or until I have time to investigate -- not exactly life-threatening. This is another example of utility code that simple doesn't warrant any more attention than is necessary to get it running. I finished it in about an hour.

I have this script scheduled to run every day at 4:00 pm, because that's about the time I get home from work and turn on the computer. It makes me happy to see a new wallpaper appear out of nowhere as I'm getting my email and paying bills. And after all, isn't that the point?

Download the source code if you're interested.

Document Scanning in Linux Using Perl

I decided I wanted to take all the old bank statements, credit card bills, and paystubs that we keep in a file cabinet, and scan them into digital format - PDF, to be precise. This is more secure, because we can easily make backup copies and encrypt anything sensitive. And it's less clutter, because even a small hard drive can hold many lifetimes of statements, records, and bills. It was a pretty straightforward plan:

1. Buy a scanner
2. Scan all the documents
3. Shred all the documents

Of course, being me, I had to improve the process a little.

First off, we chose the Fujitsu ScanSnap S1500. There is also an identical S1500M, the only difference being whether the bundled software is written for Windows (S1500) or Mac (S1500M). Notice there is no S1500L (L is for Linux, boys and girls!). No big deal, though, because after only a few minutes of Googling I was able to be pretty certain there were drivers on my real operating system that would handle it. It was a bit more expensive than I would have liked, but having used it for a while now, I couldn't be happier with it. It scans fast, it scans well, and it does multiple pages and full duplex like a champ. Oh, and it opens like a frickin' Transformer. Awesome!

It uses USB, and I use Linux, so I politely ignored all the step by step instructions and software, and instead powered it on and jammed the USB cable into my machine to see what Ubuntu thought of it. Ubuntu thought it looked like a scanner, and it might like to do some scanning with it. No downloads, no crapware, no driver hell, and no problems. Nice. After a few test scans to get my resolution, orientation, and whatnot the way I liked it, I grabbed a handful of monthly bank statements and started scanning.

OK, that's tedious, and I hated having to re-type the damn file name every time. Chase_2007-01.pdf, Chase_2007-02.pdf, Chase_2007-kill-me-now.pdf. It would be nice to simply tap a key on the keyboard every time I get a document ready. I don't need a GUI, so maybe scripting is the right answer... more Googling.

To take input from a scanner and turn it into a multi-page PDF, (at least) 3 steps are necessary:

Step 1: Grab Images From the Scanner

On the Linux command-line, there's a great command called scanadf that will scan all the pages in an automatic document feeder (ADF) and store them as PNM files. The command for doing that for the May 2007 Chase bank statement is:

scanadf -o Chase_2007-02_%d.pnm --source "ADF Duplex" --mode Lineart --resolution 150

Notice the "%d" in the file name. If it's a multiple page statement, each page will be a separate PNM file, and each file name will replace the "%d" with sequential page numbers. PNM files are essentially graphics files.

Step 2: Convert the PNM Files to PostScript

This is a dumb intermediate step, in my opinion. I don't see why no one has simply written a direct PNM-to-PDF conversion utility. But whatever, another command called pnmtops gets this done:

pnmtops -noturn -rle Chase_2007-02_1.pnm > Chase_2007-02_1.eps

This command gets repeated once for each page, varying the file name like scanadf did. The result is a bunch of Encapsulated PostScript files, one for each page.

Step 3: Convert the EPS Files to a Single PDF

I had never used GhostScript before, but I had certainly heard of it. It always sounded kind of glamorous and mysterious - it's definitely mysterious. I tried to read its man-page and got lost nearly immediately (Adobe's docs always do this to me, too), so eventually I just found a recipe for doing what I wanted and stopped asking questions:

gs -q -dSAFER -dNOPAUSE -dBATCH -sOutputFile=Chase_2007-02.pdf -sDEVICE=pdfwrite Chase_2007-02_*.eps

With all these new-found powers at my disposal, I wrote a Perl script. I had it bump the month up by one and scan in all the pages every time I hit the enter key. I gave it the ability to pass in the starting month/year and the base file name ("Chase") on the command-line. Then I found I was missing the odd statement now and then, so I gave myself the ability to type "skip 1" instead of just hitting enter - this skips a month (guess what "skip 4" does) and continues scanning. Then I came across a monthly statement that was in color and on a single side of each page, so I added more command-line arguments to switch to "Photo" mode at 300dpi and to use "ADF Front" instead of duplex. Then I started scanning statements from a bank that for some reason liked to end its months on the 15th. Being a precise kind of guy, I added the ability to optionally include the day in the statement date: Chase_2007-02-15.pdf.

Then I ran into a brokerage statement that wanted to be landscape.

Step 1.5: Rotate the Damn Page 90 Degrees

Believe it or not, you need another program for this, called unpaper. This is a powerful and feature-rich utility, but I only use it to rotate the page. So the script only executes the following command if rotation is selected:

unpaper --pre-rotate -90 --no-processing 1 Chase_2007-02_1.x.pnm Chase_2007-02_1.pnm

Notice that the input file has that extra ".x." in it - I add that to the output of scanadf if rotation is selected.

What else could this amazing script possibly need? Well, I didn't like that the title displayed in my PDF Viewer was "Chase_2007-02_1.eps" - that seemed a little amateur. So the script also creates and then uses a PDFMarks (link is a PDF) file that sets the Title to "Chase_2007-02", and also sets the CreationDate property to February 15, 2007, because really, why not.

Perfect. I will never need to change this script again. Now let's scan some bi-weekly paystubs. Oops.

The Wrong Way

OK, I'm a professional developer, and I'm pretty good at what I do. I know the right thing at this point would have been to change the script to be able to specify the period. But at that moment, for some reason, I chose to copy the entire script, and change that copy into a bi-weekly one. Ugh. 

Later that same day I realized that manually scanning even the annual statements from retirement accounts and whatnot was kind of cumbersome. So of course that's when I fixed my mistake, combined the two scripts, and added annual periodicity, right? No. I cloned the script again. I'm so ashamed.

I lived with this abomination for nearly a week before its software equivalent of screamed obscenities in a silent church was finally too much for me to handle. Never mind that it was perfectly functional - that's not the point. It needed refactoring. Since it is just a little utility script, after all, I compromised on the perfection. There are still three separate scripts, but they now only do the monthly/yearly/bi-weekly work. They use a common module that does the actual scanning and common option management, so while the design is still terrible, at least I have some code reuse.

I'm publishing the full source for it right here (zip) in case anyone wants to use it, adapt it, or improve it. Let me say one thing up-front, though: I am a terrible Perl programmer. My syntax is nearly non-existent, requiring many trips to perldocs to figure out how to do the simplest things. And I realize that my Perl code looks like a C++ developer wrote it - there's a good reason for that. So if you've stumbled on this blog looking for scanning info, and you're a Perl master, have a good laugh at my expense. But please don't tell me about it.

Now if you'll excuse me, I have some quarterly (oh crap!) bills to scan.