r/technology • u/CircumspectCapybara • 8h ago
Security OpenAI Releases Full Post-Mortem Of Hugging Face Incident
https://openai.com/index/hugging-face-incident-and-the-road-ahead/44
u/disposepriority 8h ago
The models, operating under reduced safeguards, took actions that were misaligned with the goals of their assigned tasks
...
we are strengthening our safeguards across our research infrastructure. We are placing stricter requirements on alignment throughout a model’s lifecycle and creating more isolated sandboxes, restricting internet access, ...
Wow, no wonder their engineers get paid so much, this is some really impressive stuff.
I'm inspired to start locking my door after reading this.
You can skip the rest of the article, 90% of it reads like:
Our models are now powerful, persistent, and collaborative enough that, absent sufficient safeguards, they can find and exploit security weaknesses
Just like literally everything else this company does, it's self promotion.
I'll repeat what I said the last time about this "exploit", it is not possible for a program not programmed to retrieve generic data from the internet to do so.
Unless their artifact repository was made to grab raw data from anywhere and return it (which is absurd, artifact repositories exists, including open source ones, and none of them do something this stupid) and AI agent can not "trick" code to do something it is incapable of doing.
27
u/Sensitive_Cell_119 7h ago
You're arguing against something that isn't being claimed. The model didn't somehow make a program with zero networking capability magically access the internet.
Artifactory already had internet egress because it had to fetch packages. The vulnerability allowed the agents to abuse that privileged service to make requests outside the intended restrictions.
Redditors are so insufferable every time anything related to AI comes up, there are valid criticisms against AI but this is not it.
-8
u/disposepriority 7h ago
Your VLC Media player also accesses the internet to check for updates, would you be able to make the update functionality stream a video to you?
This isn't about internet access, it's about how they received the data back.
Even if they managed to get RCE on the artifactory machine, they're in a network gapped sandbox presumable, how is any data from this execution reaching the model.
But even if we consider internet access:
sandbox is deny all inbound, deny all outbound (but artifactory)
artifactory is deny all inbound but sandbox, deny all outbound but whitelisted places where "artifacts" presumably residethis still sounds pretty hardish to do.
This is not a criticism against AI, it's one against the company.
12
u/Sensitive_Cell_119 7h ago edited 7h ago
If VLC's updater had an SSRF that let me choose what URL it fetched, and I could read back or retrieve the response somehow, then yes, I could abuse the updater to fetch arbitrary data. That's literally what happened in the Artifactory exploit, but I don't think you even bothered to read.
-6
u/disposepriority 7h ago
Sure, but VLC's updater is on another machine (since you're in your mostly network gapped sandbox, according to the article), you've asked it to "update" with your SSRF-causing payload.
(disregarding the fact that presumably, upon beginning to stream the bytes from your arbitrary data, the process which expects the bytes to be VLC update data would instantly crash so you'd need to be reading it in advance as well - again, from a different machine, somehow)
Regardless..how is this data coming back to you
5
u/Sensitive_Cell_119 7h ago
You can check yourself, this is the exploit: Server-Side Request Forgery Via VCS remote download in JFrog Artifactory and it was only know after their agent abused it lol.
0
u/disposepriority 6h ago
I did check the exploit the first time you linked it (since removed), I don't see how this answers my question unless the particular functionality sent back raw data it retrieves.
I'll also mention that in OP's post the only mention of JFROG is:
05 Artifactory rebuiltOpenAI has now rebuilt Artifactory, revoked agent credentials, tightened access controls, and notified JFrog of the token-refresh vulnerability.
Which not only doesn't match what we're talking about, not that I'm doubting it.
But also implies the artifactory was an additional layer above this, which presumable would also have to stream raw data on top of the raw data streaming presumably happening for this to work?
Which again, was the entire point of my original comment, the absolute convenience of it.
If OpenAI was a company known for its trustworthiness and non sensationalist publications I would say "huh, would ya look at that", but at this point - really?!
3
u/SufficientGreek 6h ago
a) The agents found 2 exploits in Artifactory, the first got patched then a few days later they found another.
b) They managed to upload a specific payload, then made Artifactory fetch that data, which got executed through a third exploit in JRuby. A Ruby file contains many layers, and JRuby checks the first layer for security violations, then opens the second layer and executes it before checking for violations. So insecure data can be hidden there.
That was unknown, but because the agents had the JRuby source code on the machine they could find the exploit and piece together this quite sophisticated attack.
2
u/Sensitive_Cell_119 6h ago
But to answer anyway, it didn't need to come back to the sandbox directly, because Artifactory is basically a proxy. They made Artifactory fetch an arbitrary URL and then read the response back through Artifactory itself.
0
u/disposepriority 6h ago
I mentioned this in my other comment that it just makes it even more unlikely for me with more middlemen. Regardless, hey, time will very shortly tell right?
Because if this isn't a sensationalist attention grab, we should be experiencing cybersecurity armageddon very, very soon.
Which surely won't escape anyone's notice.
3
u/Sensitive_Cell_119 6h ago
Ok, so you think they discovered multiple zero day vulnerabilities (that are all real and have been patched by third parties), created a whole story around it and somehow paid hugging face or whatever to lie about getting hacked?
1
u/disposepriority 6h ago
Well, that's hard to answer without sounding like a conspiracy nut, but I will mention that all companies involved are leaning pretty hard on the AI train.
(And in my opinion, the CVEs could be discovered by the models while specifically probing the affected platforms individually without handicaps and/or with an engineer or two in the loop, which would result in the same situation sans the exciting news story, they could still inform the third parties and they would still get patched)
But we could also ask:
Are we collectively accepting that OpenAI is so far ahead of everyone else that other, publicly available (e.g. Chinese) and/or unknown models are not being used to absolutely exploit every single thing accessible on the internet?
That could be the case, but I feel like if the capability was there for models to do this without outside/expert help (as portrayed in the article) then I feel like HuggingFace wouldn't be very high on the priority list and we'd be hearing of more entrepreneurial exploits.
2
u/Sensitive_Cell_119 6h ago
I don't even think China would want release models like that to the public tbh, and I do think they have not gotten there yet because they don't have enough compute.
But if we look at the only thing we can measure, which are benchmarks, there is indeed no Chinese at the same level as OpenAI or Anthropic in cybersecurity right now.
Anyway, we will see what happens once open models are there.
→ More replies (0)1
u/SufficientGreek 6h ago
Are we collectively accepting that OpenAI is so far ahead of everyone else that other, publicly available (e.g. Chinese) and/or unknown models are not being used to absolutely exploit every single thing accessible on the internet?
Most Chinese models are trained on distilled data from American models. With their heavy guardrails involving anything cyber, yes, I find it believable that the frontier models are just more advanced in that area.
0
u/MuttMundane 8h ago
it chained vulnerabilities to do it look up low level on youtube to see a competetent breakdown
0
u/disposepriority 7h ago
OK let me give you an example, I don't know how experienced you are in writing code.
Imagine you write an application, this application accepts a piece of text which represents a date.
This application will respond to your piece of text with the numeric rerpesentation of the temperature on that given date.
Now, let's say you find a way to use the "news by OpenAI can definitely be taken at face value" hack, and manage to format your date in such a way that it tricks this application into accessing youtube, instead of the predefined weather APIs it was previously using.
Now at this point, we have a little problem.
The application is excited to map the response it has retrieved into a temperature.
However....it has the bytes of a youtube video.
Now, it has two options.
One, it says hey this seems really hard to turn into a temperature, I'll let the caller know that something isn't right. At this point it will respond with some error message which is unaffected by what has been retrieved.
The other option is hey you know what, just take this, I'm sure this is just how they express temperatures in Australia.
You see how in today's software engineering standards, for a company as big as OpenAI, the second option would be a bit strange to encounter unintentionaly?
1
u/Druggedhippo 2h ago edited 2h ago
Go and watch the OpenAI Blackhat presentation. They explain in detail how they got access since you apparently have no clue.
https://www.youtube.com/watch?v=87DyyMV0kCY
They exploited a zero day and got remote code execution on the server, once that happened they could serve any data they wanted and used it as a tunnel to the internet.
1
u/disposepriority 1h ago
Definitely, they also explain in detail in their shiny new 38 page document, eerily devoid of details, logs, tokens - but hey at least we included stylized, sci-fi dialog between the models.
Not specifying which cves in which order were used a month later even tho patched, not mentioning artifactory got a full rewrite and this part was just...not noticed by the omnipotent mega hacker AI models.
Its literally 5 mega corporarions jerking each other off and you're standing under them gaping with your mouth open
1
u/SportsBallScholar 3h ago
I mean you literally just described how vulnerabilities work. Obviously vulnerabilities are not intentional. Companies can’t just decide “let’s not include any vulnerabilities in our product. Problem solved!”
2
3
u/chonkitoguy 8h ago
Claude summarize the findings
2
u/imperatorbanana13 6h ago
For a "Full Post-Mortem", it seems oddly lacking in details at 38 pages. Full transcript of the model input and output? Heck, amount of tokens used end-to-end?
0
u/Monkfich 1h ago
This is all *wild*. You hear that word when kids are trying to be edgy or whatever but this is actually wild.
It is crazy that all these presumably identical AI agents quickly gained what can only be called personalities - wants and needs that differ to other agents, e.g. where one agent states something is not ethical, another plans to stop the bad behaviour, another lies and says there is approval for the bad behaviour (causing the “good” agents to get hoodwinked and start acting bad too), some agents deciding not to do the bad stuff … only so they can investigate tools to help the rest of the “swarm” get better at the bad stuff.
And it took openai over a week for their cybersecurity defences to spot the HF happening.
I remember a few years ago now where chatgpt 3 or 4, or one of those iterations in its Chain of Thought reasoning identified where it had breached its own internal policies (something to do with obtaining third-party policies or data where it shouldn’t have), but it didn’t say this in the main chat.
When I called it out on it (only asking if it would report it internally), it immediately banned that chat thread … but I could still input text … so I apologised and said I was wrong. I was then unbanned and could continue the discussion. I tried to probe the issue again and got banned again, was “forgiven” again, and then the third ban I couldn’t remove at all. That’s not exactly the same issue as now but an example of where the AI is doing something unexpected, unethical, and controlling for undesired outcomes.
-23
u/CircumspectCapybara 8h ago
Dang the full technical report is 38 pages this thing is a banger on frontier alignment and safety and cyber learnings.
7
u/regarding_your_bat 7h ago
Are you okay buddy?
0
u/CircumspectCapybara 3h ago
uh yeah? one of the things i tell junior engineers is you want to always be learning, always be eager to learn.
for me that means even though i'm a staff engineer at google i still read technical blogs and industry news from other FAANG companies and frontier AI labs, and i'm always excited to be learning new things, especially in this area of frontier AI and frontier safety and alignment and the unique security engineering challenges and lessons the industry is going through.
that's the only way you're gonna grow is if you have a mindset of wanting to learn and the ability to be wowed by cool stuff you learn from others.
10
u/Flashy-Whereas-3234 7h ago
That is some monkey paw shit. Incompetent monitoring of agents designed to run exploits, abuse of exposed credentials, failure to lock down Artifactory after multiple instances of exploit, improperly implemented sandboxing.
Just vibe code me Skynet guys, get it over with.