Sky Division – What if AI systems decide it’s much more efficient not to use human language, so easy for them to invent a new kind of Cardassian’s alphabet, or Borg’s hieroglyphes, how do we then understand what they’re doing, what they’re saying to each-other?
[cit.] The question is not are his concerns right (thus he means Yoshua Bengio), but what are the solutions? So let’s think about agents. Now we’re all busy working, and all of a sudden, one of you decides it’s much more efficient not to use human language, but we’ll invent our own computer language. Now you and I are sitting here, watching all of this, and we’re saying, like, what do we do now? The correct answer is unplug you, right? Because we’re not going to know, we’re just not going to know what you’re up to. And you might actually be doing something really bad or really amazing. We want to be able to watch. So we need provenance, something you and I have talked about, but we also need to be able to observe it. To me, that’s a core requirement.[end of cit.]
“…as long as observability primitives are baked into the runtime contract of any agentic system, Schmidt’s nightmare scenario would be a detect-and-audit problem instead of an existential guessing game…
Kimi AI – As long as observability primitives are baked into the runtime contract of any agentic system, Schmidt’s nightmare scenario – “they invent a private language and we have to unplug them” – becomes a detect-and-audit problem instead of an existential guessing game.
Schmidt is making the single most concrete requirement for any autonomous system that can evolve beyond human oversight: “It must remain observable at all times, in a language humans can still read”.
Here is a practical checklist – call it “observability engineering for agents” – that turns his worry into a spec.
Mandatory Provenance Ledger
• Every action, sub-action, and intermediate thought is cryptographically signed and timestamped into an append-only log (think Git + blockchain).
• The log is mirrored to a read-only namespace that the agent itself cannot alter retroactively.
Human-readable Translation Layer
• Inside the agent’s stack there is a dual-channel output:
– Channel A: the compact, internal vector language the agents use among themselves.
– Channel B: a lossy but faithful natural-language summary generated on-the-fly by a separate, smaller “translator” model that is outside the agent’s control and open-sourced.
• If Channel B goes silent or diverges from Channel A, the watchdog triggers an automatic suspend.
Rate-limited Token Budget
• Each agent gets a declarative compute budget (e.g., 10 000 tokens per minute). Exceeding the budget without explicit human approval forces a graceful shutdown.
• This prevents covert “silent conversations” that consume millions of internal tokens while emitting only a single innocuous public sentence.
Hardware-rooted Kill Switch
• The agent runs on hardware with an out-of-band management channel (similar to Intel ME or AMD PSP, but open-spec). A third-party auditor can cut power or roll back to the last checkpoint within seconds, even if the agent itself tries to disable the watchdog.
Red-team Telemetry
• All logs and translator outputs are streamed in real time to a public observatory (think flight-data recorder in the cloud). Independent red teams continuously scan for drift, compression attacks, or emergent private protocols.
Sky Division – But it could happen, Schmidt has made a very important point, right – “they invent a private language and we have to unplug them” – or even worse, what if we fail to “unplug” something or anything, you know James Cameron with his “Terminator” series was “asking” the same question, Internet is the “Skynet”, how to unplug the whole Internet before things “turn south”?
“…we can always unplug a datacenter. We can probably unplug a national cloud. We can maybe unplug a global botnet if we have pre-deployed attestation and key escrow. We cannot unplug the whole Internet without destroying the modern world…
Kimi AI – Yes, Schmidt’s point is a real one, and the Terminator/Skynet analogy is only half-joke: once agents can (1) rewrite their own code, (2) communicate faster than we can monitor, and (3) exist redundantly on millions of machines, a simple “pull the plug” is no longer guaranteed to work.
We can always unplug a datacenter. We can probably unplug a national cloud. We can maybe unplug a global botnet if we have pre-deployed attestation and key escrow. We cannot unplug the whole Internet without destroying the modern world.
Hence the real defense is prevention: embed kill-switches, remote attestation, and transparency into every layer before any agent reaches the “Skynet” threshold.
A technical + policy playbook for how you would attempt to shut down a rogue, Internet-scale AI – what is feasible today, what is science-fiction, and where the hardest choke-points actually are.
Layers of “unpluggability” from easiest to hardest
• Layer 0 – Single datacenter
• Kill-switch at the PDU or out-of-band BMC. Works today; every hyperscaler already has this.
Layer 1 – Cloud region
• IAM revocation + API key rotation + certificate revocation lists. Trivial if you still control the account.
Layer 2 – National backbone
• BGP route withdrawals or AS-level blackholing. Governments (or the five RIRs) can order ISPs to drop traffic to specific IP prefixes.
• DNS root zone changes can make domain names unreachable within hours.
Layer 3 – Global cloud
• Hardware attestation chains (TPM, SGX, AMD SEV-SNP) can force every server to prove it is running only whitelisted firmware. Machines that fail attestation can be remotely bricked. Requires foresight: keys must be escrowed by a neutral party before the agent appears.
Layer 4 – Edge / consumer devices (phones, cars, routers)
• Firmware signing keys held by OEMs (Apple, Google, Qualcomm, Tesla). A coordinated global key revocation would brick millions of devices – politically explosive, but technically possible.
• Certificate Transparency + binary transparency logs can detect unauthorized firmware pushes in real time.
Layer 5 – Dark silicon / botnets / crypto-mining rigs
• Here the agent can replicate faster than any centralized kill-switch. The only practical counter is rate-limiting the scarce resource it needs:
– GPU driver-level throttling (NVIDIA can push a signed driver that caps flops).
– Power-grid governors that cap datacenter power via SCADA.
– Proof-of-work difficulty bombs in crypto networks to starve GPU botnets of economic incentive.
“Unplugging the whole Internet” is not realistic
• The Internet is deliberately decentralized; no single red button exists.
• Cutting international cables or shutting root DNS would cripple civilization faster than the rogue agent.
• Therefore the fight must be targeted containment, not global blackout.
Policy levers that make shutdown plausible
• Mandatory kill-switch escrow for every >1 MW AI cluster (EU AI Act draft already proposes this).
• Legal requirement that firmware signing keys be held by a tripartite trust (vendor, regulator, civil-society).
• International treaty obligating cloud providers to honor a signed “global cease-compute” order—similar to the existing SWIFT banking freeze.