Abertay University Dundee, Dare to be Digital.

Dare ProtoPlay is the UK’s biggest Indie games festival, taking place in Dundee’s Caird Hall and City Square every year over 4 days in August. As well as showcasing the games produced by the student teams on Dare to be Digital, the festival also showcases many other indie games from local, national and international developers and offers a varied educational programme for the public. Most programmes are free to enter.

Posted in Uncategorized | Leave a comment

Bitcoin drops more than 10% after security breach

http://www.bbc.co.uk/news/business-36962254

Posted in Uncategorized | Leave a comment

ADW Loading Patterns

Good advice from the CAT…

https://blogs.msdn.microsoft.com/sqlcat/2016/02/06/azure-sql-data-warehouse-loading-patterns-and-strategies/

Posted in Big Data Digest | Leave a comment

Advice to Data Science Noobs

Recently I was at Dev Day in Poland; there was a speaker there, Chad Fowler, and he gave a talk on the Passionate Programmer 10 Years On. The talk was good, but in one particular place, he spoke about advice a mentor had once given him, he’d said do these six things for success in your field – Chad then went on to talk about the six things.

His list was particular to him and not relevant here, but it got me thinking, what six things would I tell a noobie data scientist to learn. Well I thought about it, and here are mine:

  1. Learn statistics
  2. Learn a statistics language (R, SPSS, SAS)
  3. Learn a scripting language, preferably Python
  4. Learn a machine learning library preferably scikit-learn
  5. Learn an RDBMS
  6. Learn a document database

I hope this list helps you, feel free to leave yours in the comments

Posted in Data Science | Tagged | 1 Comment

Why Everyone Should Calm Down About IndyRef2

Nicola Sturgeon announces a “possible” timeline and actions that “might” trigger a second referendum and everyone goes mad. The Separatists go mad because it’s another opportunity to see their dreams come to fruition, and the Unionists go mad because the last referendum cost 13.3Million, fully occupied the Scottish Government (SNP) for 4 years, when they should have been focused on running the country and, well… it’s not even been a year since we had the last one.

Barely a month ago the SNP came in for sharp criticism because they refused to debate a second referendum at their conference then, just last week, Sturgeon was targeted by her own Cybernats after welcoming the Queen to Scotland to open the new Borders rail line. The SNP know that they have a powerful weapon in the form of the Cybernats, but they are fundamentalists when it comes to the question of independence and they will turn on anyone who they see as standing in their way, including the SNP leadership.

This is the context for the announcement regarding the timetable for a second referendum. Sturgeon has to settle her own troops before they openly rebel. Of course we’ll have to wait for the actual wording of the manifesto to she what she has to say, but it very much looks like a sop to the Cybernats – the wording of announcement is couched very much in terms of “possible”, “might” and “maybe”. The fact that this announcement came the same weekend as the new Labour leader was announced, shows us that it’s also a show of strength to Labour, than it is a series commitment to a second referendum.

So why should we calm down? Well, for the simple reason that the SNP were defeated on the fundamentals of independence and those have not changed; namely:

  1. The breath and depth of the defeat. The SNP lost in every demographic except one, including the youngest voters, which put the lie to the fact the SNP were “doing for the next generation”. The next generation spoke, and they said we’re happy with the Union thanks very much. On top of that, they lost in every constituency bar four. The SNP know these numbers and they know that they have a lot of work to do in order to win a second referendum.
  2. The European Question. The SNP want to claim that an “out” vote in the forthcoming UK wide EU referendum would trigger another independence referendum, as Scotland should not be dragged out the EU against the will of it’s citizens. However, the SNP must then deal with the issue that Scotland becoming independent would result in her falling out of the EU with no guarantee of an an early re-entry or that it would be on such favourable terms as the UK negotiated.
  3. The Currency. Salmond famously asserted that there would be a currency union with the rest of the UK. The Westminster Government stated, categorically, that this would not be the case. The voters didn’t believe Salmond’s assertions and the lack of a, credible, “plan B” caused some to vote “no”. The question of what will an independent Scotland use for a currency still has to be  answered.
  4. The Economy. The SNP, famously, asserted that Scotland pays in more to the UK than it gets out and that oil would be $130 a barrel. Well the SNP’s own figures subsequently showed that we only paid in more than we got out 3 times in the last 15 years, and the price of oil crashed. The SNP will have to come up with a new, credible, economic plan for an independent Scotland now that the voters have seen what would have happened under the old one.

The SNP leadership will have access to this information, and a lot more besides, and they’ll be in no hurry to have another referendum, so we should all just calm down about it and see it for what it is, an announcement designed as a show of strength to the new Labour leadership and an attempt to whip the Cybernats back into line.

Posted in Politics | Leave a comment

Died After Being Found Fit to Work. Really?

In this post I want to deal with this Guardian story (note: this story was also covered in other newspapers).

image

In this article the Guardian claim that 2,380 people died shortly after being declared fit to work by the Department of Work and Pensions (DWP). Okay, whilst every death is a tragedy for the individuals’ families, my initial reaction to this is: so what, people die shortly after having consumed breakfast, that doesn’t mean the two things are correlated? To find out if this is actually a problem or not, we need to dig into the figures.

The source data for this article comes from this freedom of information release, made by the DWP. There are a couple of pieces of information contained in the release, but the figures we are going to analyse are contained in the first request for information, namely:

Information request 1
The total number of people who have died within a year of their work capability assessment since May 2010

The answer to which was given as:

Total number of individuals with a WCA decision between 1 May 2010 and 28 February 2013: 2,017,070
of which: Number who died within a year of that decision. 40,680

To find out if that figure of ~40K is high or not, we have to do the following things:

  1. Using the UK’s population pyramid for 2015, break the ~2M down by age group.
  2. Annualise that figure for the dates provided (assuming uniform distribution).
  3. Normalise the above rates for working age population.
  4. Find out the Age Standardised Mortality Rates (ASMR) for each of the above groups.
  5. Calculate the expected deaths in each age group.
  6. Sum those and compare to the annualised figures.

Phew, okay, let’s get started. Using the above link to the UK’s population pyramid, and assuming it’s not changed that much between 2013 and 2015, annualising the figures and then normalising for the working population, we can say that the ~2M figure breaks down, annually, as follows:

image

Using the above link to the ASMR from the Office of National Statistics (ONS), we can find the death rate per group and from that, calculate how many people, from each group, we would expect to die:

image

Summing each age group, we find that we’d expect 1,811 claimants to die per year. Our annualised death figure show that, in actuality, 10,539 claimants died, a variation of 8,728.

The sub-text of the news article was clearly that the DWP test is unfair and is forcing people, “on death’s door”, back to work. On the face of it, these figures would support that, but before we come down on one side of this argument or the other, we have to look to see if we can account for the delta between expected and actual numbers in any other way.

Firstly, the DWP figures don’t account for what caused the death. The claimant could have finished the test and been hit by a bus on the way home; the resulting death was nothing to do with their claim. True, but then the ASMR rates take into account deaths from all causes, so we have that covered.

Next, the DWP figures show deaths after a decision was made, but it doesn’t say what that decision was. Some of these claimants would have continued to receive the benefit, or would have been moved onto other benefits, they were not all necessarily “forced back to work”.

There are also a couple of things that would have depressed our number of expected deaths. Firstly, we assumed that the claimant population mirrors the working population; it doesn’t. We know that the claimant population contains more older males, two categories (older and male) with increased death rates.

On top of this, we assumed that the claimant population mirrors the working population in terms of health (and so risk of death). This, clearly, is not true

These things will have accounted for some of the ~8K “extra” deaths; how many? Well we don’t know. I think we have to put a “health warning” on these figures. So, what can we say with confidence? Well firstly we can say that more people die in the claimant population than in the working population, however we need to do more research to discover if this difference is significant. The other thing we can say, with confidence, is that the source data does not backup the Guardian’s article.

Well that’s it for this post, ‘til next time, keep crunching those numbers! Open-mouthed smile

Posted in Data Science, Statistics | Leave a comment

Always Check Behind the Headlines

So STV are carrying a news story saying 53% of Scots are now in favour of Independence. Hmm, thinks I, unlikely given the level of upfuckery conducted by the SNP around education and the NHS right now, not to mention the crash in oil price that would leave Scotland 7.2Bn poorer than we are now.

However, there are those dyed in the blood separatists who either have cognitive dissonance around the SNP’s failings, or who don’t care, they’d rather live in abject poverty in an independent Scotland, than wealthy in the UK; so, off I trot to look into the data.

Right enough, the headline is strictly true, 53% of those asked did say they’d vote for independence, buuuut, 3% are undecided and there’s a 3% error margin on top of that, so really this poll, puts independence neck and neck with unionism, which is pretty much where some polls had us before the referendum, and we know how that turned out. Also, whilst 53% (+/- 3%) said they would vote for independence, only 50% said they wanted another poll within 5 years.

On top of that, the Unionists respected the settled will of the Scottish people after the vote, whilst the Separatists totally ignored the result and carried on campaigning; so this poll is nothing to get worked up about, you would expect one side of a debate to make ground on the other when they are campaigning hard and the other side are not.

STVPollDataEven so, I still wanted to see the data, so I could see for myself what was actually asked and what was answered. Luckily STV supplied a link where I could download the data, and here’s what I got… I shit you not, that’s it. Umm, okay, in no way can that be construed as “the data”. So now you can colour me suspicious of the whole thing.

Posted in Data Science | Leave a comment