I’m about 2/3 way through Nick Bostrom’s Superintelligence. He talks about this exact scenario. Given how foreseeable this whole situation was, the brakes mechanism should have been put in place a long time ago?
Thanks for reading, and for the first ever comment on my Substack! Bostrom is the right reference here. He laid out the control problem over a decade ago, and the scenario he described is very similar to what Anthropic's 'When AI Builds Itself' paper is now talking about.
The difficult part is your question: why wasn't the mechanism built earlier? I think the answer is the competitive market. Nobody had an incentive to build the brake while everyone else was pressing the accelerator. That's not an oversight, it's a structural outcome of the market and geopolitical pressures involved. The letter is landing now because the evidence has caught up with the theory. Whether we're still in the window to act on it is the question I keep coming back to.
That is the most worrying question. We already have an out of control truck speedy down a steep hill on a highway, only building the brakes now, while there’s not even runaway truck ramps off the side.
I’m about 2/3 way through Nick Bostrom’s Superintelligence. He talks about this exact scenario. Given how foreseeable this whole situation was, the brakes mechanism should have been put in place a long time ago?
Thanks for reading, and for the first ever comment on my Substack! Bostrom is the right reference here. He laid out the control problem over a decade ago, and the scenario he described is very similar to what Anthropic's 'When AI Builds Itself' paper is now talking about.
The difficult part is your question: why wasn't the mechanism built earlier? I think the answer is the competitive market. Nobody had an incentive to build the brake while everyone else was pressing the accelerator. That's not an oversight, it's a structural outcome of the market and geopolitical pressures involved. The letter is landing now because the evidence has caught up with the theory. Whether we're still in the window to act on it is the question I keep coming back to.
That is the most worrying question. We already have an out of control truck speedy down a steep hill on a highway, only building the brakes now, while there’s not even runaway truck ramps off the side.