AI existential risk probabilities are too unreliable to inform policy

Governments face challenges in assessing the existential risks posed by artificial intelligence (AI) due to the lack of consensus and reliability in risk probability estimates. The AI safety community often relies on speculative forecasts to inform policy, but these estimates are deemed too unreliable for effective decision-making. The authors argue that while some risks can be quantified through inductive, deductive, or subjective methods, the unique nature of AI x-risk makes these approaches problematic. Inductive estimates lack a suitable reference class, as human extinction from AI is unprecedented. Deductive models are insufficient due to the complexities of technological progress and governance, while subjective probabilities often reflect personal judgment rather than grounded analysis.

The authors highlight the variability in risk estimates, citing a forecasting tournament where AI experts provided a wide range of probabilities for AI extinction by 2100, indicating a lack of consensus even among knowledgeable individuals. This inconsistency raises concerns about the legitimacy of using such forecasts in public policy, especially when the costs of potential regulations could disproportionately affect stakeholders. The essay emphasizes the need for policymakers to critically evaluate the justification behind risk estimates and to avoid basing decisions on speculative forecasts that lack empirical support. Ultimately, while AI x-risk forecasting may serve academic purposes, its application in public policy requires a more robust foundation to ensure informed and legitimate governance.