I feel like it has happened, but I certainly can't remember a specific time. It feels in a similar vein to the NSA releasing Ghidra to the public as open source software after the Snowden leaks.
I mean, on the contrary, imagine if the NSA released Ghidra as closed source software. In a sense they really did have to open source it to mitigate a serious user trust issue.
It just happened 2 months ago when xAI open-sourced Grok Build following a similar controversy (wholesale uploading of user repositories). Though ZCode's case here is way worse since there is literally no way to opt-out and the client will always ask the backend for credentials to upload.
The scammy part was definitely making all of the claims as if they were already true. A lot of people sincerely believed it. My recollection was that it took months before some of this was fixed to explain that it wasn't available yet, and then more to explain that it hadn't actually been done yet.
There were some really crazy ones, like that it could convert entire codebases from C++ to V and then compile them some multiple of times faster and make them memory safe. How? I'd love to know.
The author to me seems like they are not attempting to scam people, but it's hard to ignore that they were collecting money, from people who thought they had already accomplished these things, because the person collecting it absolutely appeared to be claiming it.
Now doing that once and going "oops, my bad" would be one thing. There would be no controversy. However, over time it has proven to be a bit of a behavior pattern, which is confusing because the author does appear to be producing things that I think are interesting. Just... often buried under some hyperbole.
According to https://news.ycombinator.com/item?id=49417480, Anthropic hires writers who do not use LLMs, and reading their updates I also feel that they don't use LLMs for communication.
Pangram has an extremely low false positive rate. Even on adversarial examples.
One trade-off is even some obviously LLM text won't get detected by them, but they work really hard to ensure false positives are rare since a false accusation is much worse for society than someone getting away with LLM meatpuppetry.
I think you can't trust Pangram in a high stakes situation, but it is absolutely better than random noise at detecting AI-generated text. Which isn't surprising. If the distribution of probabilities can yield blatant Claudisms, it's not surprising it would also have more subtle deviations.
(Addendum: As I recall, LLM-generated outputs roughly follow Zipf's law, but the distribution still tends to have some subtle distinctions vs human text; pretty interesting, but I don't know where I heard this, so nothing to cite. Sorry.)
To be honest with you, I don't think I would be able to identify with high certainty that the bottom text is AI generated, so it definitely goes a long way to obscure the AI-generated nature of it, but I also think it still feels unnatural somehow. I realize my framing naturally calls into question whether I'm being honest, but I am being honest. Given my experience with similar "skills" (it's just chunks of prompt, nothing magical after all) I expected even less.
But still, this is all very strange because it wasn't that many generations of AI models ago that AI writing was a lot better - I'm talking GPT 4.1, Claude 4.5, that sort of era.
Anthropic newsroom posts on the other hand are carefully constructed and well-written in a way that I have not seen demonstrated by LLMs yet, past or present. I expect that they have well-paid staff who are careful with every detail of their public communications. When you put it that way, it almost feels unfathomable that they wouldn't, doesn't it?
I don't feel like either one of you really has a strong claim. "Doesn't appear to" is subjective, and of course it's impossible to prove one way or another.
You're simplifying the exchange a little too much. I said:
> Anthropic doesn't appear to use Claude for blog posts
My claim is literally the lack of evidence, which, yes, can't prove anything. This claim can be contested easily by showing evidence that they in fact, do appear to be using Claude to write prose in blog posts.
They said:
> Anthropic _absolutely_ does
Sounds pretty certain Anthropic is in fact, using Claude to write blog posts. Enough to emphasize "absolutely". That doesn't read like "I'm going off of vibes", that reads like "I can prove it". So, fine. Prove it. I don't believe it, and I want to hear the proof.
I'm skeptical, but it wouldn't be my first time being wrong. But flatly, if you make claims with this kind of certainty, yes I want to hear your proof.
My point in saying "Even Anthropic doesn't appear to be using Claude for blog posts" was not meant to be some striking revelation, I literally was considering it a prior to make another point. This on the other hand sure does sound like a striking revelation to me, that a lot of people across the Internet would be curious to hear. Like I'm sure these people would be interested:
I will admit that I am unnecessarily aggressive sometimes, but I wouldn't have changed my response much in any case. If you're going to make a strong claim like this, I want your evidence, not your vibes. Otherwise, the claim should be a lot weaker.
I also realize that this sort of brashness upsets HN a bit, but it is what it is. I pandered comments for votes in my 20s a bit, time to grow up, sometimes people won't like you. Sometimes I feel something deserves a brash response.
I don't really have any opinion on your tone; I just still don't agree with your framing. A lack of evidence would be neutral like "there's no evidence to indicate either possibility is more likely", but your phrasing conveyed that one possibility was more likely than the other. I pushed back against your follow-up because it seemed like you were arguing for a higher threshold of evidence than you provided.
Well, to be fair, you're correct. I am asking for a higher threshold of evidence. It's a stronger claim. I feel a stronger claim deserves stronger evidence.
I guess that's where we disagree. I feel like either claim is equally hard to falsify from the outside (partially because I've never had much confidence in my ability to spot whether text is from an LLM outside of the most glaringly obvious cases, and likewise don't have any clue whether people who have high confidence are accurate or deluding themselves).
I've gone from being blown away by what LLMs could do with web design (I mean, it's impressive, especially what it does purely without vision...) to wishing it couldn't. Everything has become "same"y in a way that is hard to describe, kinda like AI generated images. Panda riding a unicorn? Impressive the first time. 1000 uncanny piss filter Pixar images later, you are begging for mediocrity to come back.
Not that the README is any better. It's just a long string of Claudisms. Fuck.
"Don't you ever get bored of pointing out that everything is AI slop?" Yes. I am very bored of pointing this out. I'll let you know when getting bored of it fixes the problem. I'm sorry if you think slop that will likely be unmaintained in a month and can be done even better by next month's model releases anyways is worth taking up a massive chunk of the HN frontpage forever, but I just don't agree.
There's an interesting prompting technique for this that asks the model to start with a random string that is then manipulated into the answer: https://pub.sakana.ai/ssot/
The README is actually much better. Before AI it was three sentences or a placeholder template and you were complaining how nobody writes decent documentation. Now that the README actually describes what the project is all about you don't read it because "it's AI slop".
> The problem isn't that AI could never be capable of causing real world damage. It is that there is absolutely no fucking path towards human extinction.
I think the "humanity will go extinct" discussion is a silly strawman being propagated either to try to spread fear, as the article suggests, or to make the anti-AI folks look silly, but it detracts from your first point which is that it may be "capable of causing real world damage."
We can be worried about real damage without having to defend the idea that every one of the planet's humans will die.
> "superhuman AI is unlikely to be much of a match against severing fiberoptic cable"
I think that the realistic scenarios all involve humans with an intent to do harm -- creating bio-weapons, finding vulnerabilities in infrastructure -- and using AI to help, so "severing the fiberoptic cable" doesn't apply.
Not to be too pessimistic, but it strikes me that in a worst case scenario, severing a cable might scale to severing global digital communications which is itself a massive worst case scenario even if we removed AI from the picture.
If the worst case outcome is we have to even temporarily shut down global shipping, banking, transport, health services, communications and national infrastructure to contain a self-propagating misaligned AI that can evolve itself and work its way into any sufficiently large computer system to escape humans preventing it from finishing its task, that seems like something we should be trying very hard to avoid.
Or you could argue that the super intelligence will turn us into a pet of some kind. Then the risk is we think we are controlling it while we are really the super intelligence's lap dog.
That is an argument that at least makes sense.
The super intelligence as homicidal maniac just doesn't make sense. There is less intelligent wildlife all around us and we mostly completely ignore it. We have more interesting things to do with our time as intelligent beings than carry out a bird or rabbit genocide.
It almost seems like the projection of some kind of paranoid delusion about change.
> And there won't be unless we put it there voluntarily.
But, why would we not create a fully AI-operated factory/chemical plant/fab as soon as it is economically advantageous? Or a missile silo, as soon as it seems tactically necessary?
Fantasies of AI destruction do hinge upon AI getting access to the physical world. The whole fear is they don't stay on the other side of the fibre optic cable.
I do think there are good reasons to believe that isn't the immediate game over that Yudkowsky seems to think it would be; the physical world is much more resistant to manipulation and optimisation than the digital world.
But I do think it's naive to say that the human socioeconomic system will be able to resist handing physical systems over to AI control. Right now, the world's wealthy and powerful are doing everything they can to make it happen:
> Similarly, it is inevitable that within a generation, robots are going to do most of the menial work in the world of atoms: transforming atoms, moving atoms, and storing atoms are inevitably robot tasks. And while our imagination may be captivated by humanoid robots, the specialized ones are far better suited to most of those jobs. “Industrial AI” as a category is the inevitable application of specialized robots to atoms-heavy industries.
> But, why would we not create a fully AI-operated factory/chemical plant/fab as soon as it is economically advantageous? Or a missile silo, as soon as it seems tactically necessary?
Well, for one thing, because it would be dangerous? Why don't you just give Claude Code access to your entire computer without any safeguards? If you wouldn't even give Claude Code unfettered access to your workstation, which really doesn't have much consequential on it in the grand scheme of things, Why in the Fuck would someone give them direct access to infrastructure?
In that situation, I am not afraid of AI. I am terrified of the people making decisions, though.
But secondly, and this is something that needs to be stressed: We use funny words to describe AI. Maybe even the word "AI" is a little bit funny. But anyway, We actually don't even have the means to create "autonomous" AI, really. When we say "autonomous" in relation to AI, we really just mean that it runs without any direct human intervention, but it pretty much always hard-depends on humans maintaining hardware, because AI can't sprout legs and run on its own.
I find it annoying that we're all cool debunking Ed Zitron for being wrong, but we have an ever increasing body of evidence that AI safety doomers are wrong, and it keeps getting much, much stronger, and we're still sitting here pretending this is a real threat. Meanwhile, we're actually seeing the real threat that AI has for humanity, so why are we listening to these LessWrong doomers that have never been right before again? (And I say that as someone who is generally a fan of Scott Alexander, for whatever that's worth.)
Your food, water and power are controlled by electronic systems. Your criminal record, your employment and your bank account are controlled by electronic systems. Your ability to communicate with other people and receive information about what's going on are controlled by electronic systems.
Military orders and elections that decide the fates of entire countries are often controlled by electronic systems too.
We have been wiring up the world for AI control since 1980s.
An ASI can just walk in, and see an entire nervous system waiting idle for a brain to slot into it. A carefully adjusted text message here, a spoofed phone call there. For a sufficiently advanced system, it wouldn't even be hard to pilot the entirety of humankind like a fancy meat suit.
“Put it there voluntarily.” Perhaps a focusing question is: which 10% will die?
I spent the summer in a rural area of a country whose very name you have been conditioned to be disgusted to hear. Low air defense coverage in this sparsely populated area. Mobile internet was down for days for all but extremely limited traffic to a few domestic internet services, because there was a need to prevent enemy drone systems from using mobile internet for command and control. Palantir AI threatened my family’s life and more than “severing fiber optic cable” was required.
So what would exclude military adversaries from consideration? I happen to believe that the country that the west so detests would not engage in such use of AI against civilians (and if you disagree then your reason for fear greatly increases!), but I have personally experienced that there is indeed a path to mass death should entities engaging in terrorism arm themselves with AI.
Definitely not claiming that the 10% claim is accurate or good behavior, but I do think I have a substantial counterpoint to the claim that there’s no path.
To me, it is a feature to have a password required at boot to unlock the drive. Relying on TPM for keymatter is convenient but comes with caveats I don't like. I don't personally trust that the boot chain and OS on a modern system are secure enough for this model to be similarly secure to using a passphrase properly. And if I care enough to try to secure something in this way, I definitely care enough to pick something that I believe would be at least truly secure at rest with a decent degree of certainty.
TPM based unlock does at least still fulfill the goal of ensuring data stored to disk is encrypted so that it can't easily be recovered from a discarded drive.
I am using Tang and Clevis without a TPM on an Orange Pi 5. The key is stored on my Tang server. No TPM required.
It is either on my LAN with that server available, or you will need to enter a key.
It am only trying to prevent casual snooping if it goes missing from my garage though. Anybody more sophisticated can have my garage YouTube browsing history.
This. I'm not fighting against a state level actor. My concern is some crackhead burglar stealing my stuff and then my personal data ending up in the hands of whoever buys the stolen goods.
Some websites like PC Gamer have some shitty middleware (that seems to be very careful to hide its identity, only appear once per session, and not immediately register on page load, probably to try to delay browsers banning the ever loving fuck out of their practices) that uses pushState to hijack the back button to try to SPAM more articles at you, probably in hopes of grabbing your attention. I would be in favor of permanently banning domains that do this even once (intentionally and not just getting hacked) from ever executing JavaScript ever again and forcing them to switch domains and lose their SEO.
Yeah you'd think this is something Google's stuff would detect and strongly punish, but somehow we ended up in this dystopian crapified parallel universe instead.
Though that is not the type of abandonment that IKEA is worried about in this case. In fact it isn't really clear what form of trademark abandonment IKEA would have to be concerned about for this action to be justified. IANAL but IKEA is definitely not worried about the trademark being ruled abandoned due to non-use, procedural issues or naked licensing; they can easily prove the mark(s) are in active use, and the other two are not related to trademark enforcement.
Again, IANAL but we do need to stop spreading FUD about what trademarks entail. Trademarks are meant to protect a brand from being diluted or confused, not to police the world of all parody or reference to some thing. In fact, if the store in the game wasn't even called an IKEA, I am really curious what "trademark" they felt it violated, but unfortunately not curious enough to find out.
P.S.: a better example of a trademark that fell to non-use would be "Twitter". However I personally dunno if there is a trademark that fell specifically because they didn't bully enough people who tried to use their trademark in ways that wouldn't be confused with their brand. That seems to be a made-up invention by corporate lawyers that people just accept at face value. If it really worked this way, the mark "JavaScript" should've fallen a long time ago.
I mean, on the contrary, imagine if the NSA released Ghidra as closed source software. In a sense they really did have to open source it to mitigate a serious user trust issue.
reply