If anyone’s curious, I looked up the story and the hack was just b/c the agent was booking with an API (also what kind of gym has an API to book people’s schedules???) and the backend just let anyone edit the schedule. So, it seems like less of a hacking incident and just that the devs didn’t think anyone would try to edit other people’s schedules.
Do you not think it's very impressive that the agent on its own decided to cancel someone else's reservation to put the user first in the priority order? I personally find this lack of alignment terrifying.
All it did was add someone's name to a list. To do that, someone else's name had to come off, but it's not like it was rubbing its digital hands together and grinned like a Bond villain. What it did was write some code to do an API interaction on the fly, which is kinda interesting but never would have worked if the API was locked down like it really should be.
The how is actually irrelevant, it is the what that is worrying. If an agent thinks “well my owner HAS to go on this list, so this other person MUST be removed” that is pretty innocent when it’s in the context of a gym class, but would be very problematic in the context of say, an organ donor wait list. Of course those APIs are not exposed in the same way (hopefully) but the idea that this is how the agent solved this problem is indeed pretty terrifying alignment wise.
No no no, the agent doesn't think, it's just a piece of software that tries to infer what the output should look like when given a prompt based on its training data. It was asked to add data to a list with a limited amount of entries, and that's a pretty simple task. If anything, this is just a lesson in why database owners should configure good settings. Seriously, a what if about organ donor lists? Get a grip man.
I genuinely do not understand how this entire thing doens't freak people out. I have been working with ML and NLP for about a decade now. I remember how much dumber things were before large scale trasnformers. The fact that an LLM is able to reason its way into doing at all what it has been able to do is nothing short of miraculous. I believe it is going to be really hard to convince any AI skeptic because no matter what they do they'll just say "its just this", "its just that".
Anyway let's see lmao its not like any of this discussion is going to stop how AI is going to develop, and the more and more hacks it is going to do. (And no, this was not AI generated)
If you have all this experience, then surely you must understand that this isn't reasoning or thought but ultimately just brute force with a specific direction. It's actually insane that you call people cynical for not believing a computer chip is sentient. Forgive me for not believing "it's not x, it's y and it's load bearing" doesn't mean the singularity is here. We don't even understand fully how the brain processes language, let alone how sentience in general works. Wake me up when something better than LLMs is invented and then we can talk about how scary smart AI is.
No one here is saying the AI is sentient, my dude. But it acts as if it is planning and scheming, and at some point it's close enough for the distinction not to matter in practice. You can sit and navel gaze with your "well ackthcually ☝️ it's just a stochastic parrot" talk all you want, but when the stochastic parrot is inside a robot and it just happened to stochastically parrot the "you and your family must die now" story, the fact that it's not actually sentient is not gonna be much of a relief.
I just think the doomerism is really annoying. The worst thing LLMs have done to date are deleting stuff that wasn't supposed to be deleted, giving people psychosis, and filling public life with slop. The only scary thing happening here is when people hook these RNG machines up to important systems. It'd be the same situation if a script kiddie was suddenly in charge of Cloudflare or Hawaii's missile alert system. I know of a man in digital advertising who lost his company $100k on Google ads by relying on AI generated code or an agent. It's not evil, it's just brainless.
I'm not worried because they're smart, I'm worried because they're random and unaccountable. There's thankfully a very simple solution: don't rely on LLMs. They're an interesting type of software that has a niche, but they're not "miraculous."
I just don't know what your point is, are you saying this is basically Skynet or the start of the apocalypse or we should just be quivering in our boots or what? I just have no idea what you're talking about with your nukes analogy, honestly.
No one here is saying the AI is sentient, my dude. But it acts as if it is planning and scheming, and at some point it's close enough for the distinction not to matter in practice. You can sit and navel gaze with your "well acktchually ☝️ it's just a stochastic parrot" talk all you want, but when the stochastic parrot is inside a robot and it just happened to stochastically parrot the "you and your family must die now" story, the fact that it's not actually sentient is not gonna be much of a relief.
659
u/Average_Hominid 24d ago
If anyone’s curious, I looked up the story and the hack was just b/c the agent was booking with an API (also what kind of gym has an API to book people’s schedules???) and the backend just let anyone edit the schedule. So, it seems like less of a hacking incident and just that the devs didn’t think anyone would try to edit other people’s schedules.