On this page
The Commonwealth is the custodian of a vast array of data. When deciding how to collect and analyse data for an evaluation, it is important to consider what information already exists and to be aware of existing legislation and governance arrangements designed to ensure its use is ethical, culturally appropriate, efficient, and adheres to privacy principles.
The evaluation questions established in the planning process will help you determine what data to collect to provide the most accurate answer to those questions.
It is important to select methods and tools that are fit for purpose, robust, ethical and culturally appropriate. You also need to ensure that data collection strategies and analytic methods are both practical and possible (within available resources).
Things to consider
How can I select methods and tools that are fit for purpose, robust, ethical and culturally appropriate?
What are the best measures to use in your evaluation?
- What credible sources of data and evidence will help you evaluate your program/activity?
- What performance information already exists?
- Has the most feasible and credible approach been identified for your evaluation (that is, is collection timely and not too onerous or expensive)?
- What evidence/data do you need to answer your key evaluation questions?
- What are the things you need to be able to measure?
- What are the things you need to be able to judge?
- Which are the most important to get a balanced view?
Have you got a plan for what to measure, when and how to collect data, and from whom?
- Who owns the data?
- What’s your budget for data collection?
- What are the most important pieces of evidence or data you need to answer your evaluation questions?
- How will you establish a reliable baseline?
How will you design or source your data/evidence collection tools?
- What tools, skills and knowledge will you need to collect credible data/evidence to answer your evaluation questions?
- Do you need to design a new data/evidence collection tool?
- What data access requirements might you need?
- Are your collection methods culturally appropriate?
- Are there restrictions on what data can be provided, or in what form?
- How will you store your data (including future access requirements)?
What to measure
Many evaluations are 'mixed methods' evaluations where a combination of quantitative and qualitative data is collected.
Quantitative data
Quantitative data (numbers) will tell you how many, how much or how often something has occurred.
Collection methods include:
- outcomes measurement tools
- surveys with rating scales
- observation methods that count how many times something happened.
Qualitative data
Qualitative data (words) will tell you why or how something happened and is useful for understanding attitudes, beliefs and behaviours.
Collection methods include:
- interviews
- focus groups
- observation
- open-ended questions in a questionnaire
- ratings against quality assurance standards
- benchmarking studies comparing the processes of high-performing programs and low-performing programs
- comparisons of best practice.
Select data collection methods
Before you decide on your data collection methods, consider:
- your review of existing evidence/data and the priority evidence/data sets needed to address your evaluation questions – this will help to identify any data gaps and create a data development plan
- the skills, resources and knowledge available (or necessary) to collect and analyse the data in an appropriate way.
For statistical techniques to make valid calculations, data must be collected in an appropriate manner (for example, correct sampling). Therefore, identifying the types of analyses that will be needed early in planning your evaluation is essential.
If establishing cause and effect relationships is key to your evaluation, it is advisable to seek expert advice on which techniques should be used and how the data should be collected.
You also need to be aware of any administrative data (data collected through the routine operations of managing and/or implementing a program or activity - including any linked datasets) that may be available to help answer your evaluation questions, as well as how to access this data.
Ideally, program managers will have set up systems to capture administrative data that will help with the evaluation (for example, program guidelines, application forms and progress reports).
It is best practice to check with any target groups about what collection methods they would prefer.
Culturally appropriate, ethical and unbiased
Ensure your collection methods and tools are culturally appropriate, ethical and unbiased. At a minimum, you should discuss the evaluation with participants and pilot any tools or questionnaires that you will use. For more information, see:
- AIFS resource on evaluation with Indigenous families and communities
- AES Guidelines for the Ethical Conduct of Evaluations [PDF 497KB]
To help define your data requirements, you might consider developing a data/evaluation matrix.
Data/evaluation matrix
A data/evaluation matrix is a tool for organising evaluation questions and sub-questions and developing plans for collecting the information needed to address them.
To do this, think about:
- developing evaluation questions from the program logic
- identifying performance indicators and data sources
- articulating data collection responsibilities and timeframes
- identifying the required data, including limitations of particular sources.