Fit-for-Purpose Evidence: The Standards Auckland Council Must Apply to All Data Used in Analysis

Auckland Council must apply clear, enforceable standards to all data used in analysis and decision-making — including material submitted by advocacy groups.

Currently it does not. Officers are required to review and summarise all feedback received, and decision-makers must consider the views and preferences of people likely to be affected (Local Government Act 2002, ss 76–78 and 82). There is no parallel requirement to assess whether the evidence offered is fit for purpose. The result is that convenience samples, self-selected online polls, and questions that almost everyone answers the same way routinely enter summary reports and can influence options analysis and recommendations. That is a failure of process, not an inevitable feature of local government.

The minimum standards that should apply

Any quantitative claim about public opinion, preference, or support that is presented as evidence for a decision should meet basic survey quality criteria drawn from Stats NZ’s quality dimensions (relevance, accuracy, consistency, interpretability), the Research Association of New Zealand Code of Practice, and the standards Auckland Council already applies to its own commissioned research (Quality of Life Survey technical reports, People’s Panel methodology notes).

1. Sampling and representativeness

A survey of people who attended a bike rally, signed an online petition, or completed a form promoted only through advocacy networks is a convenience sample. It measures the views of people who chose to participate. It does not measure the views of Aucklanders in general, drivers, older residents, people who do not cycle, or any defined population unless the sampling frame and response rate are disclosed and the results are weighted or caveated accordingly.

Council’s own surveys routinely report sampling method, achieved sample, response rate, and weighting. External material that claims “Aucklanders support…” or “the community wants…” without equivalent disclosure should be described in reports as non-representative and given limited weight.

2. Question design

Questions must be capable of distinguishing between different views. “Do you think safety is important?” fails this test. Almost every respondent answers yes. The question is vague, leading by implication, and produces data with no analytical value. Professional standards require questions that are specific, balanced, and free of leading language or social-desirability pressure. Leading or double-barrelled questions should be flagged and the results treated with corresponding caution.

3. Disclosure of methodology

Every survey cited in analysis or advice papers should state: target population, sampling method, sample size, response rate (where known), fieldwork dates, weighting applied, and any known biases or limitations. Failure to provide this information is itself a quality limitation. Auckland Council already requires technical detail for its own research; the same expectation should apply to material it chooses to rely upon.

4. Fit-for-purpose assessment in advice

Policy and project advice papers should explicitly assess the strength of evidence, including external surveys. The Council’s earlier Quality Policy Advice standards (and the current national Policy Quality Framework) already expect analysis to be evidence-based, objective, and clear about limitations. This should be applied consistently: a non-representative survey should not be presented as equivalent to a probability sample or to the Council’s own weighted research.

Legal and practical basis

The Local Government Act does not require formal consultation in every case, but it does require decisions to be made after identifying options and considering relevant views and preferences. Information that is systematically biased or incapable of supporting the inference drawn from it is not relevant in the legal sense for the purpose for which it is offered. Placing material weight on it without clear caveats risks a decision that fails to consider relevant information properly. Judicial review remains available on that ground and on Wednesbury unreasonableness, even if courts give councils wide discretion in practice.

The Significance and Engagement Policy already requires the Council’s own questions to be objective. Extending the same principle to the treatment of external material is consistent with that policy and with the Council’s existing practice of commissioning rigorous research when it wants reliable population-level data.

What should change

• Require officers to record, in summary reports and advice papers, the sampling method and key limitations of any external survey that is cited as evidence of wider public views.
• Treat non-representative or poorly designed surveys as expressions of the views of the people who responded, not as estimates of Auckland-wide opinion, unless the methodology supports that inference.
• Align the treatment of external quantitative evidence with the standards already applied to the Council’s own commissioned surveys and with Stats NZ quality dimensions.
• Make methodology disclosure a practical expectation for any advocacy group that wishes its survey results to be given weight beyond the views of its own supporters.

Advocacy groups are free to survey their members or rally attendees and to present those results. They are not free to have those results treated as equivalent to representative research without scrutiny. Auckland Council already knows how to design and report surveys to a professional standard. Applying that standard to the evidence it relies upon is a straightforward improvement in the quality of analysis and a necessary safeguard for the integrity of decision-making under the Local Government Act.

Submitted by Mark Pervan, Founder, City Builders Association, Auckland.