The Internet is Broken and Here’s How

Okay it’s not, that was a bit of an exaggeration. However, a fundamental way in which it’s used is broken, and it’s broken in quite a serious way.

Your Email is You and You are your Email
Most of the social networking services that we use (Twitter, Facebook, LinkedIn, etc.), as well as most subscription based services on the Internet, use a verified email address as means of user identification. You sign up with your email address and a password, they send an email to your account, that email contains a link, you follow the link, and hey presto! you are verified as being a real live human being. Well, an entity with an email address at any rate.

The Link Between a Person and Their Email Account is Unbreakable
Because an email address is for life, right? Well, I signed up to a lot of these services with an account I had at the garyshort.org domain, which I owned. Time passed and the registrar I used went bust. The “ownership” of my domain became unclear and I was unable to renew it when it was required. To cut a long story short, I lost the domain.

This was annoying, but it only handled my email and pointed to this blog, hosted on WordPress, so I set up a new email address, and thought no more about it.

Forced Password Resets are Good for Security
Time passed, quite a lot of time actually, and LinkedIn managed to get themselves hacked. I didn’t really care, it wasn’t a big problem for me, I use “once only” passwords for everything, so knowing my LinkedIn password only gained a hacker access to my LinkedIn account and LinkedIn had already secured that. I was happy enough, I’d change my password next time I logged in.

You see I’m not a security guy so, in my head, I thought the process would be that I’d log in, I’d be taken to a “change password” page where I’d input a new password and that would be that.

Of course, that’s a dumb idea, as the hacker has my password, so if it worked like that, the hacker could reset my password and lock me out. To stop that, what happens is, next time you log in, you get a message saying that a forced password reset has been triggered and a link has been sent, to your email address, for you to follow and reset your password. This makes perfect sense, only I don’t have that email address anymore.

Coincidently, and almost at the same time, I also got a forced password reset message from Twitter. Same thing. No, you can’t log in now, we’ve sent you a reset link to your email address.

Well that’s annoying, but that sort of thing must happen a lot, so there must be a way to fix it, right? Wrong! A search of Twitter’s support pages tells me I’ve pretty much lost my account, if I lose access to my email address:

Solution: Regain Control of the Domain
Seems the only way I can regain control of my Twitter account is to regain control of my domain, so I set about finding out who has it now. The current owners are hidden behind ShieldWhoIs, so I use the contact form there to see if I can get my domain back.

I get an almost instant reply telling me I can get my domain back for $1,500!

Would the Real Gary Short Please Stand Up?
At first, that just annoyed me and then I got worried, really worried. You see these guys are clearly not the most reputable bunch and all they have to do is issue a password reset to all the services I signed up to and the reset link will come to them, they can then reset it and bam! they are me. A script can be written to automate this, it’s trivial.

Not only are they me on every service I signed up to using that email address, but they are me on any service that I currently use Twitter OAuth to sign into, remember I can’t get into my account, so I can’t unauthorise any of these other services.

Relax, no one Wants to be You!
Right, no one wants to be me per se, but the account is valuable, here’s how. Twitter, in line with many services, has a sophisticated anti spam system that will hunt down and kill spamming accounts, but a real account, that a spammer has gotten control of, that could continue spamming for days or weeks before the spam posts outweigh the years of legitimate posts, and the spam hunting software hunts it down.

There’s Money to be Made Here!
Okay, so even if a spammer can use my account to spam for weeks before being caught, that can still only be worth pennies to him right? Hmm maybe. Let’s say that’s true and each high jacked account is only worth pennies, if you are doing it at scale, then there’s a fortune to be made, and it’s pretty obvious from the above email that whoever has my domain is only interested in making money out of it.

Think about it, there’s 500 million users on Twitter and 1 billion users on Facebook, the intersection of users and domain renewals must be large, and if the spammers pick up even a small fraction of them then there’s a fortune to be made.

Oh the Irony!
If it weren’t so serious, it would be funny. Twitter will not talk to me. They claim they can’t do a thing to help me if I don’t own the email address I signed up with. They do this all in the name of security. They say that the email address is the only way to verify that it’s me. Only it won’t be me, the spammers will use Twitter’s own security protocol to take control of my account, and that same security protocol will prevent me from stopping them.

Twitter Must be Worried, Right?
You’d think so, I mean this represents a potentially large spamming problem that would be very difficult to counter with software, and which could ruin their “pay to promote” advertising model. But as it turns out, they don’t give a damn.

As you can see from the image above, they have no interest in helping me regain control of my account if I lose access to my email. I contacted them days ago, via official channels, and haven’t even gotten an acknowledgement of my concern by way of return.

So What can we Do?
The short answer is nothing, if you’ve lost control of your email address your screwed, like me. If not, make sure you are using an email address that you stand the best chance of holding on to, use a Hotmail address or similar. Also, if there are other means of identification that a service offers you, then take them, no matter that it might seem a pain at the time, trust me, the alternative is worse.

Oh, and pass this around to your friends and family, make sure they see it too.

To end, I’d just like to say: “I’m Gary Short!”  

Give Camp –The Team Leader’s Dilemma

I spent more hours awake than is healthy working at the inaugural Give Camp UK event last weekend. I’m not going to describe it here, hop over to their site if you are interested in what Give Camp UK is all about, and here if you want an introduction to the whole concept.

Anyway, like I said, that’s where I was last weekend. I led a great team which built a secure forum site, with answer voting, for the Young Minds charity. Yeah I know what you’re thinking: it took you a weekend to build that? Seriously? I mean just take one of the open source StackOverflow clones, skin it and you’re done, right?

Well yeah, but therein lies the dilemma. You can do that, and you can be done in a few hours, then you can build in more of the features that the charity wants, things that they consider “nice to haves” you can get it all done in the weekend, ‘cos that’s what it’s all about, right, delivering value for the charity?

Yes! Definitely! But then, on the other hand… you need devs to be enthusiastic about the technology you’re using. You need them to be thinking, hey that project uses cool tech, tech that I want to learn and get some experience on. Also, if you catch the interest of the devs, then not only do they want to join your team for the weekend, they are going to want to stick around after the weekend is finished and work on the project going forward. So, even though you may not get so much done at the actual weekend, the charity wins in the long run, ‘cos the devs stick around to work on the project in their spare time. As they are doing that, they are building real world experience in the tech that they are interested in, which might enhance their CV in the future. Every one wins!

Anyone reading this want to stick around after the weekend and work on a skinned StackOverflow site? Nah, didn’t think so.

So what did I do? Did I go for maximising value to the charity for the 48 or so hours of the event, or did I have my eye on the long game?

As it turned out I went for the latter. So we build our forum with CouchDB on the backend, Node.js in the middle and KnockoutJS on the front end.

Was I right? Well we, pretty much, finished the project during the weekend and the devs are continuing to push stuff to the Git repo, but it’s really too early to tell.

So, what would you have done? Let me know in the comments.

Installing NodeJS on Windows

Until very recently, when I wanted to program in NodeJS, I did so on a Ubuntu VM running on my Windows machine. Firstly, because in the beginning, NodeJS didn’t run on Windows. Latterly, because to install it on Windows was four MSI files and to install it on Ubuntu was a case of running “sudo apt-get install NodeJS”. There’s no contest there is there?

However, yesterday I spent the day with Mark Rendle as he was in Dundee as part of his DevExpress sponsored speaking tour of the UK. During our many geeky conversations he mentioned Chocolatey to me and told me I should take a look, “it’s apt-get for Windows”, he said.

So I took a look, and right enough, to install “stuff” with it couldn’t be easier. Of course, first of all you have to install Chocolatey itself; turns out that’s a breeze though. Simply run the “cmd” command as Administrator and drop into Powershell (yes I know, bear with me though). Then from the Powershell prompt run:

Set-ExecutionPolicy Unrestricted

After that, run:

   1: iex ((new-object net.webclient).

   2:    DownloadString("http://bit.ly/psChocInstall"))

And “Bob’s your uncle”, that’s Chocolatey installed. Now you can run Chocolatey from, either Powershell, or an ordinary command prompt. Having installed it, we can use it to install a whole load of packages, including NodeJS. To do that, open a command prompt and type:

cinst NodeJs

then stand back and watch the magic happen (cinst is short for Chocolatey install by the way).

Having installed NodeJS, it’s time to test it with the time-honoured “hello world!” application so open up your favourite editor and type the following code:

   1: require('http').createServer(function(req,res){

   2:     res.writeHead(200,'text/plain');

   3:     res.end('Hello World!');

   4: }).listen('8080');

Now save the file, call it server.js, and run it by typing:

node server.js

at the command prompt. Now we can connect to localhost:8080 and check that everything is working okay:

Which it is, so that’s cool. And that really is all there is to installing NodeJS (and all the other cool packages available on Chocolatey.org), so what are you waiting for?! Come on in, the NodeJS water’s fine!

Software Performance Webinar

On April 28 I’ll be joining good friend of mine, Kendall Millar and his co-host Gael Fraiteur (of PostSharp fame) on their new webinar. In this webinar, Kendall will be speaking about The Natural Laws of Software Performance and I’ll be reminding you of Everything You’ve Forgotten About Algorithms Since You Left University.

If you are interested in being in the audience for this webinar then register now at the Eventbrite site, it’s sure to be a lot of fun. Looking forward to “seeing” you there!

I’ve Been Deep Fried!

Yes that’s right folks, the great guys over at Deep Fried Bytes have caught up with me, and persuaded me to talk about Technical Debt, as if ever getting me to speak was hard to do!

I had a great time with the guys, chewing over the where’s and whyfor’s of Technical Debt and if you’d like to listen to the podcast then just hop over here and take a gander, I’d recommend it, who knows, it may even save your project.

A Twitter oAuth Example in C#

I have a number of scripts that I run against my Twitter feed, stuff like bulk deleting my DMs, that sort of thing. Well of course they all stopped working last year when Twitter moved from a basic authentication model to an oAuth one. I thought it was about time I got them up and running again. Now I know there are libraries out there that will handle the Twitter oAuth stuff for me, but I wanted to get a handle on how it worked, for my own information – besides, how hard could it be, right?

Well the answer is… not that hard, but it’s tricky to get working and a total PITA to debug. So, having gotten it working I thought I’d write this blog post, mainly for my own information, so I know where to look when I forget how to do it the next time, but if it helps you guys out too then that’s cool. On the plus side, at least you don’t have to go around the loop of getting the token in the first place. If you are using code against your own account – as I’m doing here – then all the information you need is there on your app page after you register

I’ll walk you through the process, via the code, then I’ll post the full code listing below that way you can cut and paste it into your own solution if you want to use it.

Okay, so what’s first? Well, first off, we have to collect the parameters that oAuth is going to need…

//GS - Get the oAuth params
string status = "Your status goes here";
string postBody = "status=" + 
    Uri.EscapeDataString(status); 
            
string oauth_consumer_key = "YourConsumerKey";
string oauth_nonce = Convert.ToBase64String(
    new ASCIIEncoding().GetBytes(
        DateTime.Now.Ticks.ToString()));
            
string oauth_signature_method = "HMAC-SHA1";
string oauth_token = 
    "YourToken";
            
TimeSpan ts = DateTime.UtcNow - 
    new DateTime(1970, 1, 1, 0, 0, 0, 0,DateTimeKind.Utc
);
            
string oauth_timestamp = 
    Convert.ToInt64(ts.TotalSeconds).ToString();
            
string oauth_version = "1.0";

Couple of tricky things here. Firstly notice two string variables, status and postBody. The latter is what we’ll use when we POST the request later, but the first is just the status part, it’s separated like this because it has to be double encoded in the signature (more on that later) but single encoded everywhere else! Yeah, I know! And that’s not really documented anywhere. I only noticed on close examination of the example text on this page.

On the subject on encoding, the .Net Framework has 4 methods of url encoding, two on the Uri class and two on the HttpUtility class. Guess what? They all do it slightly differently. The one that I’ve found that works best is Url.EscapeDataString, however, even that’s not fully RFC3986 compliant, so if you are using my code and still having problems that might be a place to look.

Moving on, the oauth_nonce param is a string that has to be unique on every call, this is to stop play back attacks on your account. Here I’ve used the ticks on the current date and time. I think this should be okay, but if not I’m sure Barry Dorrans will have a better suggestion for you.

Next is the oauth_timestamp which is the number of seconds after the Unix epoch. Not surprisingly, there’s not a method in the .Net framework for that so you have to write your own. Remember to use the DateTime.UtcNow property to ensure it doesn’t use the local time on your machine, but uses GMT instead, regardless of where you are. This is important because, although the oAuth spec says the time has to be >= previous requests, some people are saying that, for Twitter, it has to be within 5 minutes, though I’ve not seen that in my limited use.

//GS - When building the signature string the params
//must be in alphabetical order. I can't be bothered
//with that, get SortedDictionary to do it's thing
SortedDictionary<string, string> sd = 
    new SortedDictionary<string, string>();
            
sd.Add("status", status);
sd.Add("oauth_version", oauth_version);
sd.Add("oauth_consumer_key", oauth_consumer_key);
sd.Add("oauth_nonce", oauth_nonce);
sd.Add("oauth_signature_method", oauth_signature_method);
sd.Add("oauth_timestamp", oauth_timestamp);
sd.Add("oauth_token", oauth_token);
                        
//GS - Build the signature string
string baseString = String.Empty;
baseString += "POST" + "&";
baseString += Uri.EscapeDataString(
    "http://api.twitter.com/1/statuses/update.json") 
    + "&";
            
foreach (KeyValuePair<string,string> entry in sd)
{
    baseString += Uri.EscapeDataString(entry.Key + 
        "=" + entry.Value + "&");
}
            
//GS - Remove the trailing ambersand char, remember 
//it's been urlEncoded so you have to remove the 
//last 3 chars - %26
baseString = 
    baseString.Substring(0, baseString.Length - 3);

Having gathered our parameters together, we need to form them into a string in order to sign them. Of course this isn’t straight forward either. The string has to be formed in the following pattern:

method&URL&ParamKeyValuePairs

Which looks simple enough, but it’s a little tricky. Firstly, the ampersands that split the three parts must *not* be url encoded, but the ampersands that split the parameter key value pairs *must* be url encoded. As if that wasn’t enough of a pain, the key value pairs themselves must be in alphabetical order by key, and then by value, if the key is repeated, this is because Twitter are going to replicate your signing to ensure your precious tweet hasn’t been tampered with, and so there has to be a scheme to follow for them to replicate what you did. Of course, we can use a sorted dictionary to handle that part for us. Also remember to pass status here here and not postBody as status must be double url encoded at this point, but not in the actual POST.

//GS - Build the signing key
string consumerSecret = 
    "yourSecret";
            
string oauth_token_secret = 
    "YourToken";
            
string signingKey = 
    Uri.EscapeDataString(consumerSecret) + "&" + 
    Uri.EscapeDataString(oauth_token_secret);

Next we have to create the key that we are going to instantiate the hash with. To do this you use the url encoded consumer secret, followed by a non url encoded ampersand, followed by the url encoded token. If you don’t need the token for the request you are making then leave it off, but you still need the trailing ampersand.

//GS - Sign the request
HMACSHA1 hasher = new HMACSHA1(
    new ASCIIEncoding().GetBytes(signingKey));
            
string signatureString = Convert.ToBase64String(
    hasher.ComputeHash(
    new ASCIIEncoding().GetBytes(baseString)));

Next we are going to sign the request, nothing complicated here, as you see, just remember to take a base64 string of it.

//GS - Tell Twitter we don't do the 100 continue thing
ServicePointManager.Expect100Continue = false;

Then we have this line. This needs to be included to stop Twitter throwing 417 Expectation failed errors.

Well that’s about it. There’s not much to say about the rest of it. It’s just a standard POST request from here. Twitter do recommend that you send the oauth params (remember they have to reproduce what you did to verify your request) via the Authorization header, so of course, you have to url encode it all up again, and wrap it in double quotes too.

Right, that’s all I’ve got to say. I’ve posted the full code below, feel free to use it if it’s any use to you, of course it comes with no warranty whatsoever, other than to say, it works on my machine for what I use it for, YMMV.

class Program
{
    static void Main(string[] args)
    {
        //GS - Get the oAuth params
        string status = "your status";
        string postBody = "status=" + 
            Uri.EscapeDataString(status); 
            
        string oauth_consumer_key = "YourKey";
        string oauth_nonce = Convert.ToBase64String(
            new ASCIIEncoding().GetBytes(
                DateTime.Now.Ticks.ToString()));
            
        string oauth_signature_method = "HMAC-SHA1";
        string oauth_token = 
            "YourToken";
            
        TimeSpan ts = DateTime.UtcNow - 
            new DateTime(1970, 1, 1, 0, 0, 0, 0);
            
        string oauth_timestamp = 
            Convert.ToInt64(ts.TotalSeconds).ToString();
            
        string oauth_version = "1.0";

        //GS - When building the signature string the params
        //must be in alphabetical order. I can't be bothered
        //with that, get SortedDictionary to do it's thing
        SortedDictionary<string, string> sd = 
            new SortedDictionary<string, string>();
            
        sd.Add("status", status);
        sd.Add("oauth_version", oauth_version);
        sd.Add("oauth_consumer_key", oauth_consumer_key);
        sd.Add("oauth_nonce", oauth_nonce);
        sd.Add("oauth_signature_method", oauth_signature_method);
        sd.Add("oauth_timestamp", oauth_timestamp);
        sd.Add("oauth_token", oauth_token);
                        
        //GS - Build the signature string
        string baseString = String.Empty;
        baseString += "POST" + "&";
        baseString += Uri.EscapeDataString(
            "http://api.twitter.com/1/statuses/update.json") 
            + "&";
            
        foreach (KeyValuePair<string,string> entry in sd)
        {
            baseString += Uri.EscapeDataString(entry.Key + 
                "=" + entry.Value + "&");
        }
            
        //GS - Remove the trailing ambersand char, remember 
        //it's been urlEncoded so you have to remove the 
        //last 3 chars - %26
        baseString = 
            baseString.Substring(0, baseString.Length - 3);
                       
        //GS - Build the signing key
        string consumerSecret = 
            "YourSecret";
            
        string oauth_token_secret = 
            "YOurToken";
            
        string signingKey = 
            Uri.EscapeDataString(consumerSecret) + "&" + 
            Uri.EscapeDataString(oauth_token_secret);

        //GS - Sign the request
        HMACSHA1 hasher = new HMACSHA1(
            new ASCIIEncoding().GetBytes(signingKey));
            
        string signatureString = Convert.ToBase64String(
            hasher.ComputeHash(
            new ASCIIEncoding().GetBytes(baseString)));

        //GS - Tell Twitter we don't do the 100 continue thing
        ServicePointManager.Expect100Continue = false;

        //GS - Instantiate a web request and populate the 
        //authorization header
        HttpWebRequest hwr = 
            (HttpWebRequest)WebRequest.Create(
            @"http://api.twitter.com/1/statuses/update.json");
            
        string authorizationHeaderParams = String.Empty;
        authorizationHeaderParams += "OAuth ";
        authorizationHeaderParams += "oauth_nonce=" + "\"" + 
            Uri.EscapeDataString(oauth_nonce) + "\",";
            
        authorizationHeaderParams += 
            "oauth_signature_method=" + "\"" + 
            Uri.EscapeDataString(oauth_signature_method) + 
            "\",";
            
        authorizationHeaderParams += "oauth_timestamp=" + "\"" + 
            Uri.EscapeDataString(oauth_timestamp) + "\",";
            
        authorizationHeaderParams += "oauth_consumer_key=" 
            + "\"" + Uri.EscapeDataString(
            oauth_consumer_key) + "\",";
            
        authorizationHeaderParams += "oauth_token=" + "\"" + 
            Uri.EscapeDataString(oauth_token) + "\",";
            
        authorizationHeaderParams += "oauth_signature=" + "\"" 
            + Uri.EscapeDataString(signatureString) + "\",";
            
        authorizationHeaderParams += "oauth_version=" + "\"" + 
            Uri.EscapeDataString(oauth_version) + "\"";
            
        hwr.Headers.Add(
            "Authorization", authorizationHeaderParams);

        //GS - POST off the request
        hwr.Method = "POST";
        hwr.ContentType = "application/x-www-form-urlencoded";
        Stream stream = hwr.GetRequestStream();
        byte[] bodyBytes = 
            new ASCIIEncoding().GetBytes(postBody);
            
        stream.Write(bodyBytes, 0, bodyBytes.Length);
        stream.Flush();
        stream.Close();

        //GS - Allow us a reasonable timeout in case
        //Twitter's busy
        hwr.Timeout = 3 * 60 * 1000;

        try 
        { 
            HttpWebResponse rsp = hwr.GetResponse() 
                as HttpWebResponse;
            //GS - Do something with the return here...
        }
        catch (WebException e)
        {
            //GS - Do some clever error handling here...
        }
    }        
}

NoSQL Conference – Dundee Nov 20th 2010

The first NoSQL (Not only SQL) developer day will take place at the University of Dundee on Saturday 20th Nov (http://developerdeveloperdeveloper.com/nosql1/). Come and join other developers and database professionals and learn more about this emerging technology area. Seven sessions across the day covering VoltDB, Hadoop and HBase, Windows Azure, Cassandra and Hector, CouchDB, RavenDB and a session on is NoSQL the future of data storage.