OTTAWA — Reviewing the rules for the federal government’s use of artificial intelligence every six months is too hard, the officials in charge of it say, and they propose to cut that frequency to once every two years.

The reasons are laid out in the third review of the federal “directive on automated decision-making,” which The Logic obtained under access-to-information legislation after reporting on an unfinished draft last October.

That was nine months ago and the third review isn’t finished yet. More than three years after the directive went into effect, according to its own schedule, the government should be on the sixth.

The directive is an internal order for the federal public service on how to use algorithmic data-crunching tools and it’s maintained by the Treasury Board Secretariat, which manages the federal government’s workforce and internal policies.

In a nutshell, the directive says that the heavier the potential consequences of using an algorithmic aid for decisions affecting the public, the more scrutiny the tool is supposed to get. For example, a tool for deciding whether a form for a rebate on an LED lightbulb might require hardly any examination; one that helped decide whether federal inmates should get parole would demand a lot.

The directive requires that these “algorithmic impact assessments” be published for all to see, and that the rules laid out in the directive itself be reconsidered every six months.

That frequency, according to the current review, “is intended to ensure that the instrument remains relevant and responsive to the evolving risks and challenges of automated decision-making in the federal public sector.”

Unfortunately, doing two reviews a year “presents significant operational challenges,” the Treasury Board officials wrote. The small team in charge of the directive is always in “review mode,” they wrote, and rules that get reconsidered every six months never get a chance to settle down and just be used.

“As with any administrative policy, the requirements of the directive should display a degree of stability and reliability, enabling federal institutions and the clients they serve to plan and act with a reasonable degree of confidence,” the current review document says.

Additionally, there isn’t much AI in use in the federal government, which is “evidenced by the number of [assessments] published on the open-government portal.” (As of mid-July, there were five.) Finding things to fix in the directive depends on people using it and assessing how well it works for them, the review says.

It could be that uncertainty about what the directive will say in six months is keeping public servants from proposing algorithmic tools, the reviewers suggest.

Though the public service has had stability, in a way, because the reviews have been so slow. Amendments after the first review took over 18 months to write, approve and implement, the document says, and by the time they were in effect, “the federal AI landscape had significantly changed.” And therefore—the logic is tricky here—fewer reviews are in order.

If something unexpected happened, the federal government’s chief information officer could order a re-examination of the directive before two years was up.

The Treasury Board Secretariat couldn’t answer questions from The Logic about the status of the third review (or the missing fourth, fifth and sixth reviews) by deadline. The online version of the directive was last revised in April 2021 and doesn’t include language changes the third review proposes. 

Teresa Scassa, a University of Ottawa law professor who focuses on information law and policy, agreed in a blog post that less frequent reviews would be OK. “While more frequent reviews were important in the early days of the [directive], reviews every six months seem unduly burdensome once initial hiccups are resolved,” she wrote.

It’s not just public servants who are kept busy by more reviews than they can handle, she added. “Being asked to comment on reports and proposed changes every six months seems burdensome for anyone—including an already stretched civil-society sector,” she wrote. That is, for the relatively small number of experts, like Scassa, who pay close attention to these things.

Like the unfinished draft from last year, the more complete review includes language expanding the directive’s application to cover algorithmic aids for decisions affecting the federal workforce, not just decisions for citizens outside the government. It includes a requirement that algorithmic impact assessments include consideration of model biases—the possibility that flaws in the algorithm might reflect flawed thinking, like racist or sexist assumptions the government wants to get rid of, not embed in computer systems that are supposed to help make better decisions.

Besides slowing the pace of reviews, the newer draft adds a requirement that peer reviews, outside assessments of algorithmic tools that are required for those with more extensive consequences, be summarized in plain language and published.

And it specifies that the algorithmic impact assessments be published before anybody starts using the tools they describe. That’s been encouraged, the review says, but not explicitly required anywhere.

“The earlier an [assessment] is released in the lifecycle of a system, the better for transparency and accountability,” the review says.

Scassa already has ideas for what additions she’d like to see in the next review, including a review of the handful of algorithmic impact assessments that have been published. Some “are clearly highly divergent in terms of the level of clarity and detail provided,” she wrote, including one that doesn’t even really make clear how AI is used.

“If the [assessment] is to be a primary tool not just for assessing [algorithmic decision aids] but for providing transparency about them, then they need to be good,” Scassa wrote.