If you're of a certain age and tend not to pay attention when parking, you've likely heard of DoNotPay. Built by a young Stanford savant, the app uses an automated chatbot to help users contest parking tickets. So far, so good? Not for the authors of System Error, a tome of "provocative insights and concrete solutions" for the most important tech problems of our day. DoNotPay is "exactly the type of story that gives us pause," the authors tell us: Parking tickets serve "important, legitimate purposes." What if too many citizens successfully contest their tickets and infrastructure revenue collapses? In fact, it's a lesson in the unintended consequences of technology.
It's also an odd grievance for a book that's ostensibly about the importance of liberal democracy for containing tech excesses. DoNotPay empowers citizens, not major tech corporations, and improves on an opaque, often infuriating process. The authors don't argue that parking ticket laws should be enforced fairly and efficiently—rather, they blame the technologists for exposing those ticket laws.
This pattern continues throughout the book. Often its authors' criticisms of Silicon Valley and concentrated power are spot-on, but they can't come up with coherent responses and refuse to apply the same critical lens to government actors.
The problem with tech, authors Rob Reich, Mehran Sahami, and Jeremy M. Weinstein allege, is a culture of optimization. This culture springs from the actual problems that programmers and engineers are solving algorithmically. Spend all day teaching computers to find more efficient solutions to problems, and eventually one thinks in those terms too, without regard to values or unintended consequences. It's not a new insight, but it's a useful reminder of how culture and shared language can shape the way we approach problems.
Although tech is a dangerous world, the authors tell us, government regulation is not. "Regulation is just a loaded word for an important thing: the actions taken by those we elect to transform our shared values into rules that serve the common interest."
It's frustrating, then, that the authors fail to consider that the same mindless optimization can also afflict the very regulators they believe can solve big tech issues. One of the heroes of System Error is Lorena Gonzalez (D.), a California assemblywoman who pushed for legislation to rearrange gig working classifications for drivers at Lyft and Uber. Reasonable minds can disagree on the merits of the move, but Gonzalez cheered the firings of 200 workers from jobs she decided were "never good" and completely invented statistics to support the bill. All this, despite mass outcry from the gig workers who were affected by the law. What is this if not mindless optimization?
In some ways, System Error makes the same rhetorical move about government that technologists make about their technology. Both invoke democracy and mass support to cover up rule by experts: System Error simply wants those experts to be government bureaucrats, not Silicon Valley content moderators.
The sections on algorithmic bias and automation are decent overviews of current debates, although the authors struggle to outline clear policy proposals. Faced with 21 distinct definitions of fairness, the authors conclude: "We must pay attention to what fairness means in particular social contexts." What this means in practice is left unsaid.
But it's the section on speech online that is predictably terrible, as the authors push eagerly for more speech censorship from the tech overlords. "Even if our worst fears are not borne out by the data," they say, we should still censor disinformation far more aggressively. They draw a line between hate speech and "genuine creativity and expression" and suggest that line should be easy to police. Among the policy proposals: "dedicated anti-trolling laws to police mob-style attacks on journalists" and "reducing virality" of posts that don't satisfy an internal fact-checker. Governments must "balance the importance of free expression against other essential goals."
Of course, one of the loveliest features of American democracy is precisely that free expression is not the government's to police, outside of strict bounds. One would hope a book about the future of our democracy would recognize that truth. System Error is a useful primer for understanding political questions over tech. Just don't expect it to provide many answers.
System Error: Where Big Tech Went Wrong and How We Can Reboot
by Rob Reich, Mehran Sahami, and Jeremy M. Weinstein
Harper, 352 pp., $27.99