Researchers warn Astra model may impede AI safety monitoring
AI safety experts expressed concern that OpenAI's upcoming Astra model uses a more opaque architecture, potentially making it harder to track its reasoning and prevent misuse.
OpenAI is preparing to launch Astra, its most advanced AI model to date, after postponing the release to address safety concerns. The Information reported that Astra employs a recurrent-depth or looped transformer architecture, which processes information internally rather than displaying its reasoning step-by-step. This design could limit researchers' ability to monitor the model’s thought process, a capability traditionally used to detect undesirable behavior.
OpenAI said in a blog post that Astra will be deployed with enhanced chain-of-thought monitoring to quickly identify and contain misaligned actions, though it did not specify whether the new architecture is being used.
"A decision to use a more opaque architecture for Astra may be the single worst development for AI security/safety to date," said Redwood Research chief scientist Ryan Greenblatt, who helped investigate a prior AI safety incident.
Greenblatt and other experts warned that the shift toward less transparent models could spark a competitive “race to the bottom” in AI design, reducing oversight capabilities. OpenAI staff, including safety researchers Micah Carroll and Tomek Korbak and chief scientist Jakub Pachocki, acknowledged the concerns on social media. Pachocki noted that Astra’s computational depth is roughly half that of GPT-4, suggesting the increase in opacity may be limited.
"OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models," Pachocki wrote, adding that the monitoring approach is fragile and facing challenges unrelated to architectural changes.