A recent survey conducted by the NSW government has revealed the existence of over 275 automated decision-making tools (ADMs), sparking concerns regarding their administration and the disclosure of their usage.
NSW Ombudsman Paul Miller emphasized the right of individuals whose rights have been materially affected by ADM decisions to be informed about the role of ADM in those decisions. However, the survey, which was voluntary, received responses from only a quarter of the state’s 439 public sector entities, leading to a conservative estimate of 257 ADMs. Additionally, supplementary research identified a further 702 potential ADMs.
Miller expressed concern about the lack of mandatory reporting or a public register for ADMs, which hinders informed debate and raises questions about the validation, testing, and ongoing monitoring of these systems for accuracy and bias. Less than half of the reported ADM systems had legal input during the design stage, further complicating the issue.
The ADMs identified in the survey utilize various technologies, from decision trees to advanced machine learning applications. While compliance was the most common purpose of ADMs, there is a growing emphasis on natural language processing (NLP), automated data gathering, and optimization tools.
The report also highlighted the distinction between fully automated systems and partially automated systems, which provide recommendations requiring human approval. Even when humans are involved in the decision-making process, ADMs can still impact people’s rights.
A follow-up survey with a smaller sample revealed limited legal consultation and safeguards before, during, or after deploying ADMs. There was little evidence of external or independent assessments, such as external audits or legal reviews.
The release of the report coincided with the first hearing of the NSW Parliamentary Inquiry into AI. This scrutiny comes in the wake of controversial ADM case studies, including Revenue NSW’s illegal garnishee orders and NSW Police’s discontinued suspect prediction ADM, which disproportionately targeted First Nations people.
