Thu 05 March 2015

Filed under social_research

Tags evaluation research

I only just came across the work of Judea Pearl (which shows how ignorant I am because he won the Turing Prize in 2011). I think his work is sensational and is essential reading for all scientists but in particular for social scientists and evaluators too.

Basically he says that science has suffered because statistics has failed to formally deal with causation, leaving it as a kind of mythical thing we only talk about in whispers. Pearl provides a robust and practical notation for causation with the do() operator and develops a complete set of theorems around it. In particular he shows under what conditions correlational data can indeed be used to draw causal conclusions.

Wait, he does what?

One of the most frustrating paradoxes in the whole of philosophy, and perhaps problem number one in the philosophy of science, is Hume’s depressing observation that we can’t actually observe causation. So how are we to do science? How are we to actually learn anything?

Pearl drops a bombshell. He says that we have just been lazy in assuming this impossibility. He shows how observational, correlational data can under certain circumstances provide evidence for causal statements. This is a very big deal.

AI systems can’t sit about getting frustrated by Hume’s paradox any more than a living, learning human can. Pearl and colleagues needed to know how to set up an AI system so that it can sift through observational data and indeed make causal hypotheses on the basis of some of it. What characteristics does observational data need to have in order to support causal hypotheses?

One massively important consequence of Pearl’s approach is that the randomised control trial loses its alleged status as the unique and golden road to causal evidence.

But Pearl had plenty of other things to say which should make social scientists, and evaluators in particular, sit up and listen. Against a background of a lot of rather airy discussions of chaos and complexity in the evaluation community, he points out that our knowledge of the way the world works is built up of surprisingly simple yet surprisingly stable chunks.

He isn’t making this up: he is one of the parents of modern AI. Intelligent systems right now are all about how to learn to work out new rules in new situations. Pearl’s algorithms are helping AI systems to do just that. We humans do it all the time. Both humans and AI systems understand the world at least partly in terms of relatively simple rules of thumb - mini-theories.

These mini-theories can by the way be seen as grist to the mill of realistic (Pawson and Tilley 1997) evaluation theory. Perhaps Pearl also has some ideas for the problem which is always facing evaluators (and social scientists in general): how to synthesise the kind of mini-theories from which theories of change are built; and more generally, how to synthesise qualitative information.

Plus he does it all with structural equation models which are fun to look at and easy to work with, (and which can be seen as the basis for the logframes and logic models which evaluators have to use every day).

Look at the second part of this annotated bibliography to see the sort of things he has been dealing with. , e.g. Pearl J. and E. Bareinboim, “Transportability of causal and statistical relations: A formal approach,” Proceedings, AAAI-11, 2011. Reduces the classical problem of external validity to mathematical transformations in the do-calculus, and establishes conditions under which experimental results can be generalized to new environments in which only passive observation can be conducted.

Anyway, the book might seem really hard (I am working through it but very slowly) but I just discovered there is an Epilogue right at the back of the book which provides a great summary. You can read it in an hour or two and it will definitely change your life.

Pawson, Ray, and Nicholas Tilley. 1997. Realistic Evaluation. Sage Publications Limited. http://books.google.com/books?hl=en&lr=&id=GXagvZwC2bcC&oi=fnd&pg=PR9&ots=ZQKmPF0ABw&sig=0vnJeAQ-BR77Ozqjq2WUDDJLp84.

Comment

Sun 09 November 2014

Filed under nerdvana

Tags R reproducibleResearch

Here is my custom build system for compiling rmarkdown files in sublime-text 3 on Linux.

"cmd": ["R", "-e", "library(knitr);rmarkdown::render(input = '$file',output_file = '$file_base_name.pdf')"],
    "selector": "text.html.markdown",
    "path": "/home/steve/.cabal/bin/:/home/steve/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin ...
Read More

Sat 01 November 2014

Filed under mAndE

Tags research evaluation

Assumptions and context

Assumptions

A potential assumption in a logic model is a variable A, not under our control, e.g. tomorrow's weather in our town, (and a designated value or set of values d of that variable, e.g. fair weather with no heavy rain or storms) which ...

Read More

Fri 03 October 2014

Filed under nerdvana

Tags reproducibleResearch opensource zotero

The problem

Every student, researcher and scientist has to read and make notes and highlight or scribble on texts. In long-gone days I spent ages working on Uni literature and making summary charts of what I read, and I really wish I still had some of those notes at my ...

Read More

Fri 03 October 2014

Filed under nerdvana

Tags reproducibleResearch opensource zotero

The problem

Zotero is a really great tool for organising your scientific and professional documents and their citations, and keeping documents and citations together.

Now usually we want to store documents in some kind of hierarchical or tagged structure. Up to now it has been possible to do that either ...

Read More

Wed 10 September 2014

Filed under nerdvana

Tags reproducibleResearch opensource

Wordpress is a great blogging engine. But I spend almost all my time with plaintext files in markdown. Whatever I am working on, from my CV to statistical reports, I have it here on my hard disk and I can compile it as pdf or Word or whatever I feel ...

Read More

Sun 10 August 2014

Filed under resilience

Tags evaluation development

This article is work in progress! Feel free to add a comment. There should be a pdf at this link.

Background and motivation…

Why should we care about resilience?

Self-healing systems - This is what we want!

After the terrible earthquake in Haiti in 2010, billions of dollars of aid money ...

Read More

Wed 11 June 2014

Filed under odds and ends

Tags ggplot R reproducibleResearch

Bar charts and histograms are easily to understand. I often write for non-specialist audiences so I tend to use them a lot. People like percentages too, so a bar chart with counts on the y axis but percentage labels is a useful thing to be able to produce.

But how ...

Read More

Tue 10 June 2014

Filed under nerdvana

Tags opensource tech ubuntu

I quite often get asked (and ask myself) what is the best GANTT software for basic use.

Recently we have been comparing a few solutions so here are a couple of quick tips.

If you have a license for Microsoft Project I would use that, though it is a bit ...

Read More

Mon 28 April 2014

Filed under nerdvana

Tags R reproducibleResearch

Screenshot from 2014-04-28 09:13:14

If you love knitr and rstudio and use them to produce long reports, you probably know that you can produce a table of contents in your html (and pdf) documents. In the newer rstudio (Version 0.98.801 or later) you do it by requesting a toc in the doc ...

Read More

Thu 06 March 2014

Filed under total_speculation

Tags future psychology socialCapital society

Is the free-text search box the defining invention of the last twenty years? I think it probably is. Now more than half the entire Western world (and a lot of the rest of it) can find the answer to more or less any text-based question that occurs to them within ...

Read More

Thu 12 December 2013

Filed under nerdvana

Tags reproducibleResearch zotero

Well, for now I have regretfully given up on docear for writing papers - although it fantastic to be able to mindmap your ideas and citations, it still leads you into a dead-end when you want to actually get your paper finished - sadly, the export functions are not very pretty and ...

Read More

socialdatablog © Steve Powell Powered by Pelican and Twitter Bootstrap. Icons by Font Awesome and Font Awesome More