hayley 365

home

my rails rumble 2013 project

18 Oct 2013

2013-10-18 23:56:59 -0500

I intended to write this long before the Rails Rumble started. Ya know, like yesterday, but thanks to all of my 10.8.5 problems, all of my free time got sucked up into trying to fix that. Basically, 10.8.5 introduced a whole bunch of random audio glitches for me when recording. A totally fresh install didn't fix it. And I had no joy in finding a copy of 10.8.4. Apparently 10.8.3 (and below) is incompatible with my mac.

So my only way forward was to just try OS X Mavericks.

Long story short, as of this writing, the audio recording problems seem to have resolved themselves in 10.9, but now another app I use on a regular basis may be producing jacked up video files. I haven't had time to even test it.

So if there wasn't even time for that, there sure wasn't time to write this. I could be programming right now. But I want to keep up my daily (public) writing goal, so here we are. AquĆ­ estamos.

so what is the idea

I'm working on future radar. This is actually related to a project I mentioned in a separate entry where I would've loved to do it for the Rumble because I'd love to get an idea of how expensive this project is going to be once the free hosting runs out.

Well, I managed to figure out an angle to this project that will both:

  1. make it eligible for the Rumble because I won't be using any previous code
  2. give me the ability to test the idea as a sort of stripped down MVP version

To elaborate on the second thing, the MVP is to basically rely on using freely available future radar imagery from government servers, rather than processing the raw data and creating my own.

See, by repurposing what's already out there, I get the benefit of not needing as much VPS horsepower to start out. And if I were starting from scratch, it would be a hell of a lot quicker way to get to an MVP.

But see, in reality, I've already half-finished the custom scripts (written in the awkward and often painful GrADS scripting language). Anyhow.

the beginnings of future radar

Geebs. I swear, one huge problem has just been thinking of a name. For one, I'm not stopping with just future radar. In fact, I got into the GrADS thing to begin with to do future cyclone/hurricane modeling. Then there was a tornado outbreak predicted and I got fascinated with being able to actually visualize what it might look like.

This was last year.

There's a whole lot of it that's done. Basically though, it ended up sitting because:

And both of those things kind of held each other back. I mean, why bother trying to get it going on a VPS when it's not even automated. And why bother automating it when it's going to be too costly to run.

See, I had previously borrowed one of my friend's Amazon VPSs during their free trial to see how well the lowest end server would run things and it took like 4 or 5 times longer than my underpowered laptop.

Flash ahead to recent times and I recently moved a site from Linode to Digital Ocean because one tiny site was losing me $20/month with so little traffic (there's a reason I architect things around Amazon S3 these days).

Well, at one point, I got the nerve to see how things would run on the Digital Ocean box. To my surprise, it did all of the processing faster than my underpowered laptop (after the Amazon AWS experience, I was expecting it to also be much slower). And coupled with the faster downloads, it was an overall win.

I, of course, also managed to completely kill the box at one point because I hadn't enabled swap and GrADS consumed all of the memory when I was trying to work with too much of a dataset all at once. Woops.

But the point is, I had figured out that the raw data processing could be accomplished on a $5/month box. What I don't know yet is if the wait is going to be acceptable. Like, once a new raw dataset is downloaded, how long is it going to take to process everything? And will that delay be acceptable? I've heard of a company with a similar project where they spin up a whole bunch of servers simultaneously when the new data is in.

See, that scares me. I don't have the money to bootstrap that kind of setup, nor the know-how at the moment.

And that's where I'm excited about what I hope to accomplish during the Rumble. The hope is to figure out a super cheap way to get this project off of the ground and then try to build an audience and/or test if there's even interest in it before I go so far as to imagine a future where I'm automating the creation and destruction of so many boxes all at once.

And here's a sign I need some sleep. I tried to pair "destroyal" with "creation". As far as I can tell, that word doesn't even exist. Ah, English.

And as I write that, I am starting to feel tired.

But I most go on.

I think with another 30 minutes, I can knock out an MVP of this thing.

the MVP

The barest MVP is this:

I'm basically to the point of building the bare-bones web interface. I'm also going to have to figure out some trickery to know that if this image is for forecast hour 13 and the forecast run time was X, then this particular image applies to X o'clock.

I'm too tired to even come up with a working example.

So yes, I better hurry up and get off of this thing before I lose consciousness.