Installing HDInsight Emulator on Your Local Machine

In this video I demonstrate how to install the HDInsight Emulator on your local machine, via the Web Platform Installer, and how to run the wordcount example to test it’s installed properly.

Posted in Developement, Hadoop, HDInsight, Vlog | Tagged , , | 2 Comments

Call with a recruiter…

You have to admire his honesty, if nothing else. :-)

Him: Hi, it’s Ben Dover here, this is just a courtesy call to find out a little about your skill set, so that I can help you get your next role.

Me: Cool, where’d you get my number?

Him: (Laughs) Oh don’t worry, it’s nothing sinister, it’s right here on your CV.

Me: So you have my CV in front of you?

Him: Sure do!

Me: And you’re phoning to find out about my skills?

Him: Sure am!

Me: Uh huh, so right under where it says “Contact Details”, where you got my phone number from, there’s a section titled “Skills”, that section pretty much details my, well… skills.

Him: Ah, yeah, no, totally read that, just it didn’t really mean that much to me so thought I’d give you a call.

Me: As a courtesy..?

Him: Riiiight.

Me: And you want to find me my next role?

Him: Yes, you’ve got it! That’s why I’m calling, as a courtesy…

Me: And terms like “Hadoop”, “Pig”, “hive” and “Python” don’t mean anything to you?

Him: Not a thing, that’s why I thought I’d call you.

Me: Yeah, I’m not sure you’re going to be able to help me, but thanks for your… courtesy.

Posted in Recruitment | Leave a comment

Sorry if You Disagree, but…

… the UK is a ‘Christian Country’. Why do I mention this, you may well ask? Well, recently, David Cameron wrote an article for the Church Times, where he stated:

“I believe we should be more confident about our status as a Christian country, more ambitious about expanding the role of faith-based organisations, and, frankly, more evangelical about a faith that compels us to get out there and make a difference to people’s lives.”

Nothing too controversial there you might imagine; however, defining the UK as a ‘Christian country’ seems to have angered a section of the population, including a group who published an open letter in the Telegraph Newspaper.

Well I don’t really have a lot of time for either politicians or organised religion but, on this occasion, Cameron is correct, we are a ‘Christian country’. The reason I say this is twofold. Firstly, the Church of England is established in this country, 26 of it’s bishops sit in the House of Lords, helping to pass the laws of the land.

Secondly, the majority of people, the majority of the time, self-identify themselves as being Christian.

Now you may not like these facts, but facts they are and they cannot be denied simply by putting your fingers in your ears and shouting “No we’re not! No we’re not! No we’re not!”

Posted in Data Science | Leave a comment

Speaking at DDD South West

dddsw_medium.jpgI’m delighted to say I’ve been accepted to speak at DDD South West, where I’ll be delivering my talk on “Hadoop and Big Data for Microsoft Developers”, I hope I’ll so you there; if so, don’t forget to stop and say “hi!”.

Posted in C#, Community, Data Science, Developement, Hadoop, HDInsight | Tagged , , , , , , | Leave a comment

The Day HDInsight Died (Again)

So here I am this morning, all ready to get my Data Science Fu on, I pop onto my local HDInsight installation, and it tells me:

Hadoop Not Running

Hmm, time to see if the services are running:

Hadoop Services Not Running

Nope, all stopped. Sigh, right, start them. When I try to do that, I get an error that tells me that there’s been a “log on failure”. A what now? Same log on worked yesterday. Hmm, worked yesterday but not today, I wonder if HDInsight installs a default user with the password set to expire?

Hadoop User Must Change Password Next Logon

Yes it does! I’m not sure if HDInsight installs this way, or if some policy applied on my work machine forces this, but unchecking the “User must change password at next logon” and checking “Password never expires”, should solve this problem.

Services Started

Services are up and running again…

Hadoop Is Working

And HDInsight is back!

So, if your local instance of HDInsight stops working, then check that the hadoop user password hasn’t expired.

Posted in HDInsight | Tagged , , | Leave a comment

How Alex Salmond Answers The Hard Questions on Scottish Independence

As you’ll know, at least hopefully you will, should you vote yes on September 18th, in the Independence Referendum, you won’t be voting for independence per se; what you’ll be voting for is to give Alex Salmond the right to negotiate the terms under which Scotland will eventually become independent.

Alex, has set out his position on each of the major areas under negotiation, this is the right and proper thing to do, and it is to be welcomed. However, what he hasn’t done is to set out a “Plan B” if you will, or informed us what his “Red Lines” would be.

What do I mean by that? Well let’s look at the need for a “Plan B”. First, let’s take the issue of Europe. An independent Scotland will have to negotiate entry into the EU. Alex’s position is that we’ll be accepted and the process will be quick. Recently the EU President, a man in a position to have some idea about what it takes to expand the EU membership, said that, at best, the negotiations would be long and complicated.

Given that, when questioned on what would happen if Scotland were refused entry, or even what would happen to our trade if EU membership took, say, 5 years to complete; Alex should be able to say, “Well if my negotiating position isn’t entirely successful, or if I lose completely, an Independent Scotland would…”. Instead, he waves his hands and claims all will be well, much like the video.

Having dealt with that, let’s look at what I mean by “Red Lines”. To do that, let’s look at the situation with the currency. First Alex claimed that the pound was a “millstone” around the necks of the Scottish people, and he couldn’t wait to rid us of it. Once focus groups told him that ditching the pound was a barrier to people voting yes, that idea was immediately scrapped and we were told we’d be keeping the pound. Notice, as with all these things, Alex never says, this is my negotiating position, he just asserts that, “we will…”, like it’s a foregone conclusion.

However, the other political parties have stated that you can’t have a currency union without a political union, and since the SNP are not interested in the latter then an independent Scotland can’t have the former. By the way, despite what the SNP say, this is not the other parties bullying us Scots, it’s a principle followed by the EU too, if you want to be part of the Eurozone, then you have to be part of the political union, since we know Alex wanted to join the Eurozone not that long ago, we know he’s not against a political union per se, just with the rest of the UK it seems.

So, Salmond’s position is we’ll have a currency union, the remainder of the UK’s position is we won’t, so what’s Salmond’s alternative, should his negotiating position fail? There isn’t one, it’s more hand waving a la the video and more assertion of “There will be a currency union”. That brings me back to the idea of a “Red Line”, if we can’t be in a currency union with the UK, and we can’t join the Euro, even supposing we wanted to, immediately, would that constitute a “Red Line”, a situation whereby Salmond would say, “to carry on under these circumstances would be so damaging to Scotland that we won’t continue with Independence”? I fear not, in fact my fear is that there are no “Red Lines”, my fear is that Salmond wants independence at any cost, and since he won’t tell us what, if anything, his “Red Lines” are, I fear I’m right.

So, before I go, just remember this. If you are voting yes in September, you are betting that Salmond wins every argument, and the result of every negotiation is that he gets his own way. When was the last time any politician achieved that? Hell, when was the last day you won every argument you took part in? The sad truth, that no one seems to be waking up to, is that if you vote yes in September, you have no idea what you’ll be getting, so be careful what you wish for.

I’m voting no, only because “hell no!” isn’t an option.

Video | Posted on by | Leave a comment

The Day HDInsight Died

So there I was, casually working away on my HDInsight sessions for a conference next week. Happily submitting my jars and watching my map reduce jobs running, like a happy little data scientist. I had just finished writing my umpteenth job and I wanted to submit it, so I clicked to open the HDInsight dashboard; but it failed to open, reporting that there was nothing on the end of localhost:8085. Hmm, odd. so I opened the job tracker: dead too, and so was the namenode.

I opened the services manager under Windows and saw that, although all the Hadoop services were marked as automatic they were all stopped. Weird, but okay. I clicked to start the first one and it failed to start, reporting a logon failure. “Well that’s nonsense” thinks I, but I’ve not got time to fix that, I’ll just reinstall.

So I uninstalled HDInsight dev preview and the HDP, then fired up Web Platform Installer and reinstalled it. Job done. Well, not quite, because although the namenode and jobtracker and the command line shortcuts installed okay the dashboard was missing. Since I knew the URL I just added it and click to launch it:


But no, the dashboard hadn’t been installed. Weird, but not to worry, so I Googled it with Bing, or I might have Binged it with Google, I can’t really remember, but the upshot was that I found this post:

Which basically said I had to had go here:


and hack the package. So I went there and low!


The package isn’t there. So, that’ll explain why it didn’t install, but doesn’t explain why it hasn’t been downloaded, nor why HDInsight suddenly stopped working in the first place, after having been happily mapping and reducing it’s little heart out for weeks.

If I get to the bottom of it I’ll let you know.

Posted in HDInsight | 5 Comments