Your occasional update on our network, upcoming events, professional development and the world of impact evaluation
Members of the Impact Evaluation Practitioners Network,
We were thrilled to see so many people at our first seminar. There were 124 people on the call to hear Professor Nick Biddle, Bethany Jones and Andrei Turenko speak about their work. Both presentations sparked a lively discussion and there was lots of interest in data access and data sharing. We will share more resources on those topics in future newsletters and seminars. Until then you can review Nick Biddle’s slides [PDF, 426 KB], which are attached to this email, and watch the seminar recording here. We are planning another seminar for November, read on to find out the details.
This month Melbourne will host the Australian Evaluation Society conference. If you are attending, it is an excellent opportunity to meet other network members. For anyone in Melbourne we have organised a post‑conference catch up on the Wednesday evening. Details are below.
We hope to see you in person or online soon,
Yours in impactful impact evaluation!
The IEPN Secretariat
Network update
Our new secretariat
At our first network seminar this month we announced the new members of our secretariat. The full secretariat is:
- Harry Greenwell, The Australian Centre for Evaluation
- Tony Bertoia, Australian Education Research Organisation
- Driss Ait Ouakrim, The University of Melbourne
- Gosia Bucki‑Smith, Transport Accident Commission
- Alejandra Mendoza Alcántara, Monash University
- Harrison Hansford, University of New South Wales
- Campbell McNolty, The Australian Centre for Evaluation
To our new secretariat members, thanks for dedicating your time to the network. The team are working on our next seminar, the terms of reference and a program of events for the next 12 months.
Membership breakdown
The network is growing quickly and already takes in a broad range of sectors. Australian government employees make up 26% of the network and state government a further 16%. We also have representation from academia (23%), independent research bodies (16%), consultancy (10%) and philanthropy (9%). Impact evaluation takes many forms and each of our sectors has unique expertise to share. There is so much we can learn from each other.
Upcoming events and training
Network drinks in Melbourne
Join network members for a meal or a drink in Melbourne. Following the first day of the Australian Evaluation Society conference we will be gathering at a pub close to the conference venue. All are welcome and some colleagues from other government departments will be joining us too. If you are in Melbourne we would love to see you there.
Date: Wednesday 18 September, 2024
Time: 5:30 – 8:30 pm
Venue: The General Assembly, 29 S Wharf Promenade, South Wharf VIC 3006
There’s no need to RSVP just come along and meet your fellow impact evaluators.
Australian Evaluation Society Workshop: Planning impact evaluations of social policy
Zid Mancenido and Dan Carr from the Australian Education Research Organisation present a workshop tailored for public servants and evaluators. This day‑long workshop offers a deep dive into the nuances of thoughtfully implementing experimental and quasi‑experimental evaluation methods to identify the impact of social policy programs. We will go beyond theories of cluster randomised trials, regression discontinuity designs, and differences‑in‑differences approaches to illustrate how to design rigorous impact evaluations tailored to program rollout, policy context and the intention of the evaluation itself.
Date: Tuesday 17 September, 2024
Time: 9 am – 5 pm
Venue: Melbourne Convention Exhibition Centre
For more details and to register, go to the AES conference website.
The second IEPN seminar
Please join us for the second seminar hosted by the Impact Evaluation Practitioners Network. After a short discussion of network business, the seminar will feature two guest speakers:
Dr Aidan Cashin from UNSW will present Target trial emulation: a framework for causal inference from observational data
Dr Barbara Broadway presents her paper with Anna Zhu, Spatial Heterogeneity in Welfare Reform Success
Date: Monday 18th November
Time: 1 – 2:30 pm AEDT
Venue: Online via Microsoft Teams.
What we are reading
As we develop the format of the newsletter we are trying out some new content ideas. Here are a few things that caught our interest in the last month. If you find this reading list interesting or you would like to submit something for the next edition, please contact us at evaluation@treasury.gov.au
Assessing macroeconomic policy: E61 released a research paper on lessons from Covid‑19 Economic Supports for future economic downturns. It is not strictly an evaluation but does try to measure the effectiveness of policy where causal inference methods are hard to implement.
Instrumental variables and the exclusion restriction: a recent paper by Jonathan Mellon points to the risks of using weather, such as rainfall, as an instrument. The paper was also discussed on the Development Impact blog, which asks whether using lightning strikes as an instrument may face similar problems to rainfall.
Differences‑in‑differences: Scott Cunningham has a series of posts on his substack that expand on a DiD checklist developed by Pedro Sant’Anna. And a recent paper by Amanda Weiss asks “How Much Should We Trust Modern Difference‑in‑Differences Estimates? “, highlighting the risk that, in practice, DiD studies may often be under‑powered.
Combining Experimental and Observational Data: Experimental evidence is great but it can be costly or impractical to collect long‑term outcomes in experimental settings. This 2023 paper by Graham Van Goffrier, Lucas Maystre and Ciarán Gilligan‑Lee offers a way to combine experimental data with long‑term outcomes from administrative data. The work of Susan Athey, Raj Chetty and Guido Imbens takes a similar approach.