by Graham Teskey

My friend and colleague Lavinia Tyrell recently posted a note on LinkedIn, highlighting a recent World Bank Independent Evaluation Group report, which reflected on various methods of monitoring and evaluation currently used in development.[1] In so doing, Lavinia referenced this diagram:

---

As a fan of diagrams, as well as a long-time user of both the London Underground and the Washington Metro, I looked at it closely. There are several stations on the map that I have never been to and probably never will. ‘Regression Discontinuity’ seems to be far too deep in the East End, while ‘Empowerment Evaluation’ sounds very Sloane Square or Friendship Heights. I may take a trip to QCA once the line is built, even if only to find out what the abbreviation stands for. But without a doubt my favourite stop must be on the Northern Line – the yellow one – ‘Logical Frameworks’ (logframes).

In all the jobs I have held, the only training that has ever stayed with me was a three-day course on logframes, held in a very pleasant beach hotel on Fiji’s beautiful coral coast. This was a few months after I joined what was then the Overseas Development Administration, DFID’s forerunner. Three days on logframes. Yes really. Our Pacific team were gathered to learn this new skill. The course was designed not only to help us think rigorously about how change happens, but also to ensure that we had a shared understanding of what constitutes good design.

Twenty-five years later I am aware that logframes are out of fashion: references to the ‘tyranny of the logframe’ abound, its vertical determinism, and its lack of flexibility. Donors seem to have replaced logframes with things called ‘Program Logics’, as in the schematic on the right. You don’t have to be able to read it to get the picture… (and it’s logframes that are supposed to be too vertical…?).

I would argue that program logics like this constitute a retrograde step. My main criticism is that, well, they are just not logical. Program logics contain no explanation of why outcomes at higher ‘levels’ will be forthcoming. The above diagram simply says, “If we successfully deliver the outputs, then we will deliver the outcomes”.  Sometimes a bunch of heroic assumptions may be listed somewhere in the diagram but with no interrogation regarding their political feasibility. This is a theory of action, not a theory of change. A ‘Theory of Change’ (ToC) refers to an informed assessment of why some particular set of activities we think will lead to some higher-level development goal. Why do we judge that relevant actors will make the changes assumed within the program?  Simply giving actors the formal right or skill to do things differently is not the same as those actors using those rights or skills in practice.  We call it a ‘theory’ because it is both explanatory (why things are as they are and not like something else) and predictive, generating predictions that are capable of proof or falsification.  

The basic requirement of any ToC is that it must contain an “if, then, because” statement. For example, “If we deliver effective and valued training to staff in organisation X (the Activity), then we will deliver a certain number of staff with new skills and competencies (the Output). If these staff subsequently apply their skills effectively and consistently in the workplace (the Intermediate Outcome), then we will improve organisational performance (the end of project outcome or EoPO). We judge this to be the case because (i) the senior management of the organisation has introduced a rigorous performance management system; (ii) the organisation is under intense public pressure to perform; and (iii) “poorly performing staff have recently been dismissed and a cohort of capable graduates has been recruited”. These ‘because’ statements are absent from program logic diagrams. There is no consideration of whether the changes are politically likely, organisationally feasible, or socially possible. This is the discipline that the logframe demands.

I would go as far as to suggest that a good logframe will tell you most of what you need to know about any initiative: its purpose, its ToC, how to measure progress and success, the sources of the data, and how because statements will be tested. There are six basic rules of logframes:

• There should be one goal and one goal only. If there are two goals, then at best we run the risk of inconsistency of effort as priorities and context change. At worst we run the risk of having two separate projects.

• There can be no double hierarchy.  This is a common error in many program logics: i.e.,”We will do A in order to achieve B”. Sorry, not allowed. If you want to do A, then fine, do it. But B must appear at the next level up. That is what ‘logic’ means. Recently I reviewed a design which had, at the EoPO level: “Municipal health services have the incentives, capability, and systems to enable improved delivery of the essential services package’. This is a classic double hierarchy, one thing (incentives, capability, and systems) to enable another (improved service delivery).

• There can be no Outcomes without Activities to deliver them. Outcomes do not appear from nowhere – they require planning and delivery.

• Outputs do not need indicators. An Output is just that – a thing in and of itself and which is self-defining. It is either delivered as a direct result of the Activity or it isn’t.

• The Goal (or impact statement) must be tracked. I am dismayed that there seems to be a trend to leaving the Goal statement ‘unmeasured’. While project monitoring and learning efforts should be focused at the Output – Intermediate Outcome -EoPO level and their inter-relationships, the Goal must be tracked. Otherwise, how will we judge success? Tracking requires identifying measures at the design stage, which are publicly available, will be collected consistently, and will indicate investment impact.

• While the Goal in most circumstances will remain constant, Activities and Outputs can change through the life of the project. No logframe is ever set in stone. I do not understand why some claim that they are.

The arrows drawn on the schematic on the right show how to read a logframe. It does not proceed vertically. On the contrary, it is to be read in a zig-zag fashion: “if we successfully deliver Outputs, then we will deliver Outcomes because we judge the change is both technically desirable and politically feasible (this is our ToC). The column on the extreme right hand of the logframe is the probably the most important. We judge this investment has a good chance of being delivered because…. And these ‘because’ statements must be interrogated as rigorously and frequently as are the ‘deliverables’ in the narrative summary column. I accept that there may be as many lousy logframes as there are lousy program logics – usually because the right-hand column is labelled ‘assumptions’ and they are merely listed and then forgotten. A good logframe will summarise why the results chain has a chance of coming about. Without that the logframe becomes just another way of presenting a program logic.

So, a plea to everyone waiting at Evaluation Station Central. Please take the next Northern Line train to Logical Frameworks. Mind the gap as you alight there. Indeed, by alighting at this station you will fill many of the gaps in the current obsession with program logics.

[1] World Bank 2020. Evaluation of International Development Interventions: An Overview of Approaches and Methods Jos Vaessen, Sebastian Lemire, and Barbara Befani