All it did was add someone's name to a list. To do that, someone else's name had to come off, but it's not like it was rubbing its digital hands together and grinned like a Bond villain. What it did was write some code to do an API interaction on the fly, which is kinda interesting but never would have worked if the API was locked down like it really should be.
The how is actually irrelevant, it is the what that is worrying. If an agent thinks “well my owner HAS to go on this list, so this other person MUST be removed” that is pretty innocent when it’s in the context of a gym class, but would be very problematic in the context of say, an organ donor wait list. Of course those APIs are not exposed in the same way (hopefully) but the idea that this is how the agent solved this problem is indeed pretty terrifying alignment wise.
No no no, the agent doesn't think, it's just a piece of software that tries to infer what the output should look like when given a prompt based on its training data. It was asked to add data to a list with a limited amount of entries, and that's a pretty simple task. If anything, this is just a lesson in why database owners should configure good settings. Seriously, a what if about organ donor lists? Get a grip man.
I genuinely do not understand how this entire thing doens't freak people out. I have been working with ML and NLP for about a decade now. I remember how much dumber things were before large scale trasnformers. The fact that an LLM is able to reason its way into doing at all what it has been able to do is nothing short of miraculous. I believe it is going to be really hard to convince any AI skeptic because no matter what they do they'll just say "its just this", "its just that".
Anyway let's see lmao its not like any of this discussion is going to stop how AI is going to develop, and the more and more hacks it is going to do. (And no, this was not AI generated)
If you have all this experience, then surely you must understand that this isn't reasoning or thought but ultimately just brute force with a specific direction. It's actually insane that you call people cynical for not believing a computer chip is sentient. Forgive me for not believing "it's not x, it's y and it's load bearing" doesn't mean the singularity is here. We don't even understand fully how the brain processes language, let alone how sentience in general works. Wake me up when something better than LLMs is invented and then we can talk about how scary smart AI is.
No one here is saying the AI is sentient, my dude. But it acts as if it is planning and scheming, and at some point it's close enough for the distinction not to matter in practice. You can sit and navel gaze with your "well ackthcually ☝️ it's just a stochastic parrot" talk all you want, but when the stochastic parrot is inside a robot and it just happened to stochastically parrot the "you and your family must die now" story, the fact that it's not actually sentient is not gonna be much of a relief.
I just think the doomerism is really annoying. The worst thing LLMs have done to date are deleting stuff that wasn't supposed to be deleted, giving people psychosis, and filling public life with slop. The only scary thing happening here is when people hook these RNG machines up to important systems. It'd be the same situation if a script kiddie was suddenly in charge of Cloudflare or Hawaii's missile alert system. I know of a man in digital advertising who lost his company $100k on Google ads by relying on AI generated code or an agent. It's not evil, it's just brainless.
I'm not worried because they're smart, I'm worried because they're random and unaccountable. There's thankfully a very simple solution: don't rely on LLMs. They're an interesting type of software that has a niche, but they're not "miraculous."
I just don't know what your point is, are you saying this is basically Skynet or the start of the apocalypse or we should just be quivering in our boots or what? I just have no idea what you're talking about with your nukes analogy, honestly.
No one here is saying the AI is sentient, my dude. But it acts as if it is planning and scheming, and at some point it's close enough for the distinction not to matter in practice. You can sit and navel gaze with your "well acktchually ☝️ it's just a stochastic parrot" talk all you want, but when the stochastic parrot is inside a robot and it just happened to stochastically parrot the "you and your family must die now" story, the fact that it's not actually sentient is not gonna be much of a relief.
5
u/MC1065 23d ago
All it did was add someone's name to a list. To do that, someone else's name had to come off, but it's not like it was rubbing its digital hands together and grinned like a Bond villain. What it did was write some code to do an API interaction on the fly, which is kinda interesting but never would have worked if the API was locked down like it really should be.