So, I'm setting up a new filtering mail server. Nothing hard core. I'm trying to keep the resource usage down on it, so I'm not doing heavy content inspection this time.

Basically, the idea is that this server is sort of a first line of defense. Filter out all the obviously bad emails, and then forward what's left on to a bigger mail server that has lots of resources to do full content inspection.

So, I've got my test mail server, and the first thing I do, is set it up as a backup mail server (backup MX for those hip do DNS lingo). Now, I did this because I know for a fact that spammers try to send mail through backup mail servers first. I guess they think the security might be a little more lax on them. Funny thing is, this new setup is actually MORE strict than my current primary mail server.

Anyway, so I set the MX records in dns, designating this new server as a backup mail server for 2 of my client's domains. I litterally go over, stir some stuff on the stove for a few minutes, and when I come back to look at the logs, there's already spammers trying to send mail through the backup mail server.

So far, nothing's getting through. So I start thinking, only the REALLY bad spammers are going to try using the back door, so they're obviously going to get blocked easily. I want some REAL spam to try to get past this new server. But how do I do that, while leaving things the way they are? Easy, I set up a new subdomain and configure it to receive email via my new mail server.

Ok, so I have fake email accounts just waiting for attempts by spammers...but how do I get my email address spammed on purpose? I need to get my fake email addresses out on the web where they can be scraped by the slimeballs. I don't just want to go submitting my email address to websites. Sure that will get me spam, but some of it will be legitimate spam, since I you know requested it.

That got me thinking...where do spammers try to look? How about twitter! So i created a fake twitter account, and I've been tweeting random nonsense using popular trending hash tags, and making sure to also blatently tweet my email address. If that doesn't get me noticed by spammers, I don't think anything will.

Oh, and for good measure, here are some email addresses, in hopes that some bots will come scrape them and attempt to send spam to them:

john.j@labs.ds7.net

mike.d@labs.ds7.net

fred.rogers@labs.ds7.net

mkqdfs2sdf23@labs.ds7.net

gsdgklnlkfnlfk2324@labs.ds7.net

bobby.b@labs.ds7.net

pleasespamme@labs.ds7.net

eatmoreveggies@labs.ds7.net

freemoney@labs.ds7.net

Also, please feel free to make this post as public as possible. The more bots who stumble across my fake email addresses the better!

Ok, so I was "blogging" way before it was cool. Back then we called it a "news" section on your website.

Anyhow, enough hipster-isms. I've been meaning to get back into "blogging"...actually, strike that, I've been meaning to get back into writing really. But like everything in my life, I tend to find excuses not to do things, or more precisely, I start making up "prerequisites". Oh, I'll get back to blogging once I find a blogging service I like, or I'll start once I settle on a blogging system to setup and use, or once I settle on a layout to go with...

So I never end up doing anything. Well, I've been toying with posterous of and on for a while now, and I think I'm just going with it for now. So now I have no excuse. While I procrastinate on setting up my actual final blog (and website, etc), I will throw random things here.

Not that anyone in the world cares to know any of this...but that's not the point of blogging anyway...is it?

I sent a canned response to my local congressman regarding the new internet blacklist bill using the form from the demand progress website (http://demandprogress.org/). I got a canned response back:

 

Dear Mr. Stanley,

 

Thank you for contacting me with your concerns regarding internet privacy. Your views are important to me as I work effectively represent you and Ohio's 16th Congressional District. 

 

As you know, the internet has flourished since its beginning with minimal government intervention and has revolutionized our ability to communicate and move forward in a globalized world.  It has provided our businesses a competitive advantage to reach a broad and global market.  Open and widely-accessible access to the internet and modern telecommunications has become an important part to our freedom, prosperity, and innovation.

 

At the same time, the lack of any regulations on the internet can lead to violations of individuals' privacy and even identity theft.  These are serious offenses and should be fought through smart rules and appropriate laws that crack down on illegal behavior, while maintaining the openness that has allowed the internet to expand our access to information and a globalized world.  

 

I would like to thank you for sharing your thoughts with me internet regulation. Know that I will keep your thoughts in mind as I consider future legislation impacting technological issues. In the future I encourage you to visit my websitewww.renacci.house.gov where you can email me any other concerns that you have.

 

 

 

Sincerely,

Congressman Jim Renacci
Member of Congress

 

So, from reading that, it appears that my congressman was politely telling me, "thanks for your input, but I'm going to vote this other way". Maybe I was over reacting. Well, I replied anyway, and here's what I wrote:

Thanks for the canned response. I'm not really sure who you have as a
technological advisor, or what qualifications you have to be making
decisions that impact the technological future of the world, but I
doubt an internet "blacklist" will do much if anything to "protect
privacy" and "fight identity theft" on the internet. The only thing it
WILL succeed in doing, is limiting the freedom of normal everyday
citizens, and act as a form of censorship.

This type of legislation literally scares me. I thought I lived in a
country, that the rest of the world looked to as an example of
freedom. This type of legislation is far, far to close to the types of
things the Egyptian government was doing to prevent it's overthrow.

I sincerely hope you do not, under any circumstances, even consider a
bill like this. If you actually do represent me, and actual citizens
of this country, you won't consider it. If you do vote for a bill like
this, it will just be another example of large corporations, and media
conglomerates pulling the puppet strings of congress.

Please, PLEASE, think of the people you represent.

Sincerely,
Douglas Stanley

Maybe that was a bit harsh, I don't know. Then I started thinking, what if I come off as some conspiracy theory nut job. I should attempt to show that I honestly think that censoring the internet is a horrible idea, and that it really won't solve anything. Also, maybe I can show that I actually do know what I'm talking about (somewhat), as a long time computer science major. Which lead me to come up with an idea, so I wrote a follow up email:

Follow up thought to my previous email.

Identity theft is a very very large problem. Censoring where people
can go on the internet will NEVER solve it. That's a reactive
approach. Everyone knows reactive approaches don't work, and they
barely even help.

Why not instead work on a bill that will create a proactive defense.
We need a way to make identities "unstealable". If an identity can't
be stolen, then there would be no identity theft. Research needs to go
into ways to create cryptographically secure identities that cannot be
faked, or stolen. And ones that can be used to prove some one's
identity in person and online.

I would be very much behind any politician who would be bold enough to
propose an actual solution, instead of yet another wasteful
ineffective band-aid attempt at a solution.

Besides, searching for a solution like that would definitely create
jobs, whereas a censorship type approach, like the one currently
proposed, would only hurt business, and possibly lead to further job
loss.

Sincerely,
Douglas Stanley

So if anyone actually visits this blog entry, let me know what you think. Is it possible to create an "unstealable" identity using modern cryptography? What about cryptography + biometrics? If it's even remotely possible, why aren't we spending money researching it? Think of the problems it would solve outside of preventing identity theft. We could simplify the election process, and be absolutely sure that all votes are valid. Well, atleast we'd know the casting the vote part was valid, we'd still need to prevent tampering after the fact.

In any case, I think it's a better solution to focus on than wasting money on censorship...

So this weekend, my kids watched a movie on our hd tv in the living room. So what, lots of kids do that right? Well, what if I told you, that the movie they watched, was originally from a VHS tape, and that there's no VHS player connected to the tv? That's because I recently started the process of converting everything we have to digital form. Where am I storing all that data you ask? On a distributed storage cluster of course!

So, if you know me in real life, you'll know two things. One, I'm a geek. Not just a geek, but a hard core geek. I take being a geek seriously. So if you know me in real life, you also know that I tend to go overboard when it comes to computer hardware. No, I don't have a $5000 gaming rig. I think that's ludacris. I did spend $5000 on a computer once, but that was for a 1U rackmounted server I have hosted in a datacenter in florida. Yeah, I'm that kind of a geek. I don't just have a wireless access point at home. My wireless access point is on it's own dedicated network, with no access to my "real" internal network, except by vpn. Yeah, I'm that guy. I don't just have a "server" in my basement for "messing around" with. I have servers (plural). Before we moved in to this house, I spent several days in the crawlspace above the second floor, pulling ethernet cable from the basement. I'm that guy.

Second thing people who know me in real life know, is that I've been VERY busy lately. What have I been doing you ask? Well let me tell you. I've had this pile of old hardware in my basement that has been growing. I had a stack of hard drives...actually more like 3 stacks. Not a couple, more like 10+. I've been meaning to get down there and put some old stuff together and do something useful with it, but just haven't felt like it. It wasn't until I had something really important (like a take home midterm to write), that I got the motivation (need for procrastination), to get down in the basement and finally get some stuff done.

Ok, so I think I've talked enough about my geek cred, lets get to the point...most of you probably scrolled down to this point anyway :)

I started out with an old celeron 2.4ghz machine, with a promise sata card, 2 200gb drives and 2 250gb drives. I got it all put together, installed debian testing on it. I set up the drives in a software raid 5, and since you can't boot from this particular promise card, I put in a 1gb CF card in an IDE adapter to boot from. I installed glusterfs server on it and started messing around. After testing the throughput over gigabit ethernet, I was impressed. I was getting around 30-40MB/s. Not blistering fast, but much faster than your average older sata drive by itself, or atleast on par with it.

Impressed by the results, and wanting to have an actual "cluster", I needed atleast one more node. So I got together the parts for another machine. This one a P4 3.0Ghz, also with 1GB of ram. Now, I had another 4 hard drives, but only 2 onboard sata ports. So, I bought a 4 port sata card. This $30 purchase, was the only money I spent on this project. So my second node had 2 200gb sata drives, 1 250gb sata drive, and a 400gb sata drive (I litterally was just putting whatever I had together). I again, set up this node the same as the first. Software raid5, and a CF card in an IDE adapter to boot from. I also installed debian testing on this machine.

On both machines, the disk space that wasn't part of the software raid5 (I made both machines with an identical size raid5), I put all in just a plain old LVM group. So, it is more or less JBOD.

I then began testing my new two node gluster cluster. And I have to say, I'm still impressed! So, now I have this medium sized storage cluster (not HUGE by any means). I put an mpeg4 encoded version of a movie I copied from vhs with a haupauge hardware mpeg2 encoder on the storage cluster. Then, I set up a VM running Ubuntu Lucid. I installed the glusterfs client on the VM, as well as uShare upnp media server. I set up the ushare server to stream the files stored on the gluster file server, and viola, my xbox 360 can now stream movies and music stored on my gluster cluster.

Not only that, but any machine in my house can mount this gluster filesystem, and access anything stored on it (or create new content to be put on it).

Now if you read this far, you're probably thinking, why didn't I just take a 1TB usb disk with movies on it, and plug it straight into the xbox. Well, that wouldn't be hardcore, now would it?

Seeing a post on identica about some one's nerd score made me want to see how I ranked. Well, here it is:

Thought about possibly blogging while mobile, and remembered you can blog to posterous vial email.

So, I thought I'd try a post from my phone using my Gmail app.

I'm also thinking about migrating my "professional" blog over here, and just using posterous. I have been feeling less and less like hosting my own services.  So we'll see.

It was surprisingly simple to create, and so far, very fast. I'd also imagine it would scale quite well.

So if you're not familiar with either tornado or couchdb, you're probably not reading this, as the title was probably not interesting enough to get you to come here. However, if for some reason you chose to read this entry and have never heard of either of those, then I highly suggest you check them out. 

Tornado web is a VERY simple python based web framework that is based on an asynchronous design (basically how nginx works), where a single process can multiplex thousands of simultaneous network connections efficiently using modern features of the linux kernel (epoll). Basically what epoll does, is notify the process when new data is available on a network connection. So the single process is free to do whatever work it needs to do while it waits for more data from the client.

Now, the benefits of doing things asynchronously are lost if the process blocks for any reason (say you're transcoding video, or performing a horribly written db query). If the process blocks, then the clients starve. This is normally not an issue with multi-process or multi-threaded applications, as if one process or thread blocks, the others can continue to work. But, if there's only one process, this becomes an issue.

Most tasks, however, finish so fast, the client doesn't notice. It does mean though, that if you make network connections inside the server process (say to query a database), it's preferable if you can make them asynchronous as well. For this reason, I chose CouchDB for the database. Basically, CouchDB is a JSON document database (it stores objects, each object is a JSON encoded object), that speaks a simple RESTful (i.e. over HTTP) protocol. This was ideal, as I could perform asynchronous database queries.

Ok, so enough background, now for the nuts and bolts. Hopefully, if you were already familiar with these two projects, you would have just skipped down to here. So the basic architecture of my service is:

NGINX -> Tornado Web App -> CouchDB

All front end requests are handled by nginx, which proxies the request to the tornado app. The tornado app parses the URI for the base36 encoded string at the end, and then uses that as the document id in a couchdb lookup. So it basically sees a GET request from a client, takes the ID being asked for, does an async GET request of it's own to couchdb. When the couchdb request returns, it calls a callback function, which parses the JSON document from couch, pulls out the URL and sends a 301 redirect to the client with the URL that was fetched.

Now, at this point, I also added an extra step, just after it sends the redirect, it performs a PUT asynchronously back to couchdb to log the request with useful info (like user-agent string, remote IP, etc). All this info could be parsed from the nginx logs and stuck into couch later for analytics, but it doesn't seem to impact performance at this point, so I'm keeping it in there for now.

I've done some very basic preliminary testing, and I have to say, it seems to be just as fast, if not in some cases faster, than the top url shortening services out there. And since it's all asynchronous, it can scale up like nobodies business. In creased number or requests per second, just add a few more of the tornado app servers. If the couchdb server gets bogged down, you can load balance it as well.

Now, one final note here. This isn't a full blown url shortening service, as there is no way to ADD urls to the DB. Atleast not via a public interface at this point (there might never be). This is because this service is actually just going to be a part of a larger service/site that I'm working on, and so there will be no need to publicly create them (they'll be created on the back end automatically for things).

Where's the code you ask? Well, I may publish it in the future (right now it's kind of ugly). If anyone asks nicely (and therefore cares), I'll put the simple code up some where.

That's all for now though.

Ok, for the most part, it looks alot like pidgin. So it's not too bad. But I went to re-organize the order my buddy groups show up in to be more how I'm used to in pidgin, I realize they don't move. Then I realize the groups are in alphabetical order.

Sorry empathy, but you get a #fail from me. I'm immediately switching back to pidgin. Besides, I like things like off the record plugin for pidgin. Not even sure if empathy has something comparable!

Hey, I gave it a shot. I didn't just write it off because it was something new. I figured if the fine folks at Gnome wanted to switch default IM apps, fine, I'll try it. But it sucks. Well, for me atleast. Please allow me to re-organize the order that the buddy groups show up in, and maybe I'll give it another try.

I read a while back some one on twitter was looking for a service to archive their twitter feed (I think so that it could be used for publishing later).

That got me to thinking, how hard would it be to write something like that? I mean, you can grab a twitter feed as straight json right? Why couldn't you just simply stick that json you get back straight into a JSON oriented database like couchdb or mongodb?

So, as a test, I wrote a couple of lines of python in my favorite interactive python shell IPython, and it really was easier than you could imagine. I was only using stuff from python standard library too (urllib2 and httplib) to do all the work. Basically, just fetched json from twitter search, where I searched for messages from my own username, then turned the results into a dictionary using python's json library, took just the messages out, iterated through each one converting it back to json and creating a unique couchdb document for each message. I also made sure to use the unique twitter message id as the couchdb json document id.

So where's the code? I don't have any on me right now, left it in my other coat pocket. But I think I'll write a little daemon soon that polls twitter search periodically for a given search string, and copies all messages found into a couchdb database for archival. All the heavy lifting is actually done on twitter's side and couchdb, python is just the glue in the middle.

Watch for code to come in the future.