Your occasional update on our network, upcoming events, professional development and the world of impact evaluation
Dear members,
On Monday we hosted our second network seminar with a great turn out from members. We had lots of excellent feedback on both presentations. Harrison Hansford spoke about Target Trial Emulation and Dr Barbara Broadway presented her paper on Spatial Heterogeneity in Welfare Reform Success. Thanks to our speakers and to the secretariat members who pulled everything together.
In this update, our plans for the network in 2025 and lots of reading to fill up your summer. We have a special section on Target Trial Emulation, for those who were excited by Harrison’s presentation.
Most importantly, we are launching our network survey. We want to find out what you want from the network, please take 5–10 minutes to fill it out.
Best wishes for the upcoming holiday season. We are looking forward to a packed 2025 for the IEPN.
Yours in impactful impact evaluation!
The IEPN Secretariat
Network update
Network survey
Today we are launching a survey of network members. It’s your opportunity to shape our agenda for 2025, so please respond.
Click here to take the IEPN Survey
The survey includes an option to join a new member directory, to help network members find collaborators and shared interests. We plan to set up the members directory within our members platform next year. The survey itself is anonymous, if you are interested we will contact you separately to gather information for the directory.
The survey closes on Friday the 20th of December, please make sure you get your responses in on time to have your say.
Upcoming initiatives
At our second network seminar this week, secretariat member Dr Driss Ait Ouakrim gave an update on our plans for 2025. Here is a quick summary of some of the initiatives we have underway.
Randomised Coffees and Teas (aka RCTs!)
Early in the new year, we will randomly assign network members into small groups to help you meet other members. It will be up to group members who are interested to organise a time to catch up for a coffee, tea or any other refreshments. If the RCTs are a success, we will run additional rounds in the future.
Members collaboration platform
The Australian Education Research Organisation (AERO) have kindly agreed to host a Microsoft Teams group for the network. This will give us a place where members can ask questions, chat and share resources. It will be an easy place to share files, including seminar recordings and other network resources.
We will trial the group from December into the new year with a small number of members before opening it up to the whole network early in 2025.
If you would like to volunteer to be part of the trial and help us test the functionality of the platform, please reach out to Tony Bertoia on tony.bertoia@edresearch.edu.au
What we are reading
If you find this reading list interesting or you would like to submit something for the next edition, please contact us at evaluation@treasury.gov.au
Diving deep on Target Trial Emulation
Following an insightful presentation on Target Trial Emulation at our most recent seminar, we have lots of reading to share. Here are some resources for people who want to dig deeper into TTE.
TTE for evaluating health policy: This academic paper by Seewald, McGinty and Stuart discusses the application of TTE to evaluating policy initiatives (paywalled – see the full paper attached to this email), particularly those in public health.
More information on the Target Trial Emulation Guideline: Enter your details here to receive more information on the guideline as it develops.
Application of TTE in a study of men’s mental health: The Australian Institute of Family Studies used a TTE approach to evaluate the provision of additional mental health care services.
Other reading
Five key questions before starting an impact evaluation: Felipe Alexander Dunsch and Simone Lombardini dive into the practical considerations of delivering an impact evaluation. A useful guide for those about to commence or commission an impact evaluation.
A method for selecting control variables when analysing a randomised controlled trial: David McKenzie discusses the use of the Post‑double selection Lasso (PDS Lasso) to choose the best set of covariates for inclusion in a regression. While researchers often consider many options for covariates, this method typically selects only a small number of covariates for inclusion (mean = 3.6, median = 2)
Balance checks – a better approach to joint orthogonality tests: David McKenzie (again!), this time writing about balance checks and the tendency of F‑tests and similar approaches to over‑reject the null hypothesis and what to do about it. The post summarises a recent paper by Jason Kerwin, Nada Rostrom and Olivier Sterck.
The growth in causal claims in economics: Over thirty years the proportion of economics papers making empirical causal claims has grown from 5% to 28%. This and many other features of the credibility revolution are explored graphically on a new webpage about causal claims.
Who’s Afraid of Policy Experiments: Research from Robert Dur, Arjan Non, Paul Prottung and Benedetta Ricci finds that voters in the Netherlands are in favour of randomised policy experiments, particularly for policies that they don’t already have a strong opinion about.
Using PLIDA for public policy research and reporting: This research paper from the Productivity Commission provides an overview of the Person Level Integrated Data Asset (PLIDA) from a user perspective. It takes a case study approach to identifying the strengths and limitations of PLIDA for different areas of Australian public policy research and reporting. It also notes emerging directions that stand to enable greater use by researchers over time.