Automated decisions fail on data, not on the algorithm

A South African court has already struck down an automated decision system, and it did not fall on the algorithm. It fell on the inputs. On 23 January 2025 the Pretoria High Court declared the automated bank verification and database checks behind the Social Relief of Distress grant unconstitutional and invalid.

The commentary you will read this month says AI regulation is coming to South Africa. That is true and it is also the less useful half of the story. The rule you are waiting for arrived as a judgment, and the reasoning in it is about data quality rather than about software.

What the court actually found

The South African Social Security Agency launched the Social Relief of Distress grant in May 2020, and regulations published on 22 April 2022 moved it to online-only applications with automated verdicts treated as definitive. Eligibility was tested by checking applicants' bank accounts and government database records automatically.

Successful applicants collapsed. As the Institute for Economic Justice argued and the court accepted, the process produced irrational and arbitrary denials of assistance to people legally entitled to it. In the month the new regulations took effect, approvals fell from about 11 million to 5.6 million.

Two findings are worth reading closely if you run a business.

The first is about a rule that looked reasonable. Any bank deposit was treated as income. A deposit can be a temporary loan, or money held on behalf of somebody else, and the system had no way to tell the difference. The logic was sound and the data underneath it did not mean what the logic assumed.

The second is blunter. The government databases the checks relied on were error-ridden and outdated, which likely indicated employment where there was none.

The government has appealed, and the Supreme Court of Appeal hearing was set down for 25 August 2026 in Bloemfontein. The law here is not settled.

Why this is your problem and not only the state's

Nothing in that reasoning depends on the decision being a government one. It depends on a system making a determination about a person using records nobody had recently checked.

Private businesses make those determinations constantly. Whether to extend credit to a new account. A CV into the shortlist or out of it. A claim flagged, a supplier put on hold, a shift worker given the overtime.

The Conversation piece notes the same risks apply in the private sector, where operations are more shielded from public scrutiny than a grant scheme is.

The two questions in the wrong order

Most businesses approach this as a legal question first. Am I allowed to let software decide this? Then, once someone says yes with conditions, they go looking for the software.

Run it the other way. Ask what the decision is actually reading, and whether that record is true today.

A credit hold that reads a payment history from an accounting package nobody reconciles is the bank deposit problem with different labels. A shortlisting rule that reads a job title field last standardised in 2019 is the outdated database problem. In both cases the software will be confidently, defensibly, traceably wrong, and it will be wrong at speed.

What POPIA does and does not settle

POPIA has a section on exactly this, and the number is worth carrying. Section 71 says a person may not be subject to a decision with legal consequences, or which affects them substantially, taken solely on the basis of automated processing intended to profile them. The section lists what profiling covers: performance at work, creditworthiness, reliability, location, health, personal preferences and conduct.

The exceptions are narrower than people assume. The decision can stand where it is taken in connection with concluding or performing a contract and the person's request has been met. It can also stand where appropriate measures protect their legitimate interests, or where a law or code of conduct provides those measures. Those measures have to let the person make representations, and give them sufficient information about the underlying logic of the processing. Whether your particular process clears that bar is advice work rather than something to settle from an article.

What is already clear is the surface area. Bizcommunity's legal desk points out that using AI in human resources necessarily involves processing personal information, from CVs and interview recordings through to biometric data, medical certificates supporting sick leave, and performance metrics.

That is most of an HR function. Meanwhile the practical position, as one ICT commentary puts it, is that phoning your bank, medical aid or insurer already carries a good chance the voice is not human. The deployment is ahead of the rulebook, which is the ordinary state of affairs and not a reason to wait.

The audit that comes before the automation

The finding that should change how you sequence this work is that the SRD system failed on inputs. That is exactly what a discovery audit surfaces, and it surfaces it before anything is built rather than in a courtroom afterwards.

For each decision you are thinking of automating, write down four things. Which field the decision reads. Which system owns that field. Who last verified it and when. What happens to a person when it is wrong.

The fourth is the one that decides whether the decision should be automated at all. If being wrong means a customer waits an extra day, automate it. If being wrong means someone loses income, a shift or a job, the system can prepare the decision and a named person signs it.

That distinction costs nothing and it is the difference between a defensible process and an indefensible one. It is also why Zorah treats data quality as the first phase of any automation work rather than something to tidy up later.

What to do on Monday

Pick the one decision in your business that software already makes without a person looking at it. Most businesses have at least one and have forgotten it exists. Credit holds and automatic account suspensions are the usual candidates.

Find the field it reads. Open that field for ten records and check it by hand against something you trust. If more than one of the ten is wrong, you have your answer about whether to automate the next decision, and you did not need a lawyer to get it.