In my experience, they all do this with dathasheets. Even if they read the actual datasheet, they misunderstand them gravely. I can't relie on them to do unusual setups or chaining stuff properly. It's true I did these attempts a couple of months ago, maybe they're better now.
Take the approach Geohot is suggesting. Take a shipping container, make a standard layout, cooling and compute load. Find a cheap source of electricity.. Place it and have compute.
It has been done... We used to get our POP gear built out from Dell (?) in shipping containers - pre-racked, wired, and cooled - just add network/power feeds. We'd have them dropped places we needed more capacity but there wasn't space available in the DC.
No, that is probably one of the worst cases they probably saw. Most likely the subscription inference cost is much lower than you expect. If you look at costs for similar open models they are much lower than what you get by buying from anthropic, so that is the real cost basis I expect.
It's likely Amazon is making a fucking killing though.
While $5000 is a lot, the people who rack up close or just over a thousand "API equivalent cost" are pretty common.
> Most likely the subscription inference cost is much lower than you expect.
This is probably not true because they'd be screaming it off every rooftop were that the case.
Same deal with the API inference. Even the "profitable on inference" claim is sourced back to hearsay of informal statements made by OpenAI/Anthropic staff. No formal announcements, nothing remotely of the "You can trust what I'm saying, because if I'm lying the SEC will have my head" sort.
Yet making such statements would be invaluable. If Anthropic can demonstrate profitability before OpenAI, they could poach most of the funding. There's no reason to keep it a company secret.
And API inference is only part of the total costs, not even bringing in training and ongoing fine-tuning. If they're not even profitable on inference, how could they hope to be profitable overall.
I'm going to be a dickhead for a moment here, apologies, there's no way to say this that isn't rude to you. This is still the same hearsay "In an interview, somewhere."
> Let’s say half of your compute is for training and half of your compute is for inference. The inference has some gross margin that’s more than 50%.
But the context, the very previous sentence is:
> Think about it this way. Again, these are stylized facts. These numbers are not exact. I’m just trying to make a toy model here.
Here, Amodei is in effect using weasel words. He is not giving any actionable claims about Anthropics margins, merely plucking an arbitrary number. Why 50%? Is 50% reasonable? Is 50% accurate to the company? Those are all conclusions the listener draws, not Amodei.
> I don't know about SEC rules
The main premise is that, as a CEO, there are some regulations you are beholden to. You're not allowed to announce you've made a trillion dollar profit, sell all your stock, and then go "teehee just kidding". The SEC prosecute you for securities fraud if you do that stuff.
This makes such weasel words as earlier suspicious. Because the exact statement Amodei gives is not prosecutable. He's not saying anything about the company, just doing a little "toy model".
The degree to which it is intentional that this hearsay travels and is extrapolated from "Well he picked 50% because it's a reasonable figure, and because he's CEO, a reasonable figure would have to be a figure akin to what his company can achieve" into "Anthropic has 50% margin", that's up for debate. Maybe it is intentional, maybe Amodei is exactly the same kind of shitweasel as Altman is. Probably he's just a dumbass who runs his mouth in interviews and for whatever reason cannot issue the true number in an authoritative statement to dismiss this misconception.
Hence my original comment; If the real number were better than the hearsay rumours of the number, Amodei would immediately issue a correction; It'd be great for the company. Hell, even if 50% were about the margin, that'd be great! To promote that from mere hearsay to "we're profitable, go invest all your money" would also be huge. Really, any kind of margin at all would put him ahead of OpenAI.
But he doesn't issue a correction. He doesn't affirm the statement. Perhaps he has other reasons for that, but a rather big reason could be that the margin number is in fact pretty bad.
Now, the observant reader will note I am also using a weasel word there. I do not know whether the number is good or bad, your take away should be "it could be bad." Not "it is bad". Go pressure Amodei into giving us the real number.
Self reply as I could've explained the SEC thing better:
Anti-fraud regulators like the SEC give an inherent trustworthiness and credibility to CEOs and other market participants. You can trust that they're not lying to you, because they would be sent to jail if they were.
Another example are general anti-fraud regulations; Consider how one would trust North American or European steel suppliers more than Chinese steel suppliers.
It's not that the Chinese are "evil lying people" and Americans are "saints who never lie", it's that you can trust American, Canadian, and European courts to hold the liars accountable by regulations even if you're not in any of those regions. But the Chinese liars won't be held accountable by regulations.
Thus also the opposite, if someone opts out of this credibility granted to them by anti-fraud regulations, their words may not be quite so truthful.
SEC rules means CEO cannot lie or deliberately hide the cost of something.
50%+ Margin statements have basically been "We are making 50% on delivering it." This does not include ANY of the costs of getting to this point, training, scraping, datacenters, people and so forth.
They are basically saying "Oh yea, the cost of GAS in the car is only X so charging Y per mile is great margin" while ignoring maintenance, cost of acquiring the car and so forth.
but comparing your margin of charging to drive a mile to the price of gas makes a lot of sense? that is the only variable cost in the equation. training / scraping / people are all pretty much fixed costs.
That's a tad naive. CEOs can and have and often lied about everything:
Sam Bankman-Fried, Elizabeth Holmes, Kenneth Lay - and hundreds if not thousands more.
The SEC is a regulatory agency, not able to bring criminal charges. The above-named for the most part had to be prosecuted by the Department of Justice or sometimes state attorneys.
> While $5000 is a lot, the people who rack up close or just over a thousand "API equivalent cost" are pretty common.
I think if you're not Anthropic and you don't have access to the actual data, then you can't say for sure. A bunch of anecdotes on terminally-AI people on twitter is not making a convincing case for me, IMO.
On the other hand, if similarly sized models cost much much cheaper than this, why, in the world, would Anthropic have much higher costs than that?
Also, counterpoint, maybe they want you to think that they have higher costs so you're more willing to actually pay for it?
> Using profanity indicates a weak vocabulary. A lack of discipline. A degree of unrefinement unbecoming
Says you. I think not swearing obviously indicates a weaker vocabulary since there's a lot of things you can't or won't say.
Of course, swear words can be offensive if you're their target, and I don't really enjoy that side. But they are _very_ effective in communicating frustration, anger, surprise. I think using them brings a bit of spice to life and a well placed "Fuck!" can feel extremely cathartic and... dare I say... pleasant.
You can and _must_ if you want competitive costs. Musk famously overpaid in order to get speed of deployment.
I was reading geohot's musings about building a data center and doing so cost effectively and solar is _the_ way to get low energy costs. The problem is off-peak energy, but even with that... you might come off ahead.
And that dude is anything but a green fanatic. But he's a pragmatist.
And 80-90% of the adherence to 80-90% of those laws are self-imposed.
Once upon a time I used to buy tamales from a guy on the street corner. He was probably breaking half a dozen business licensing and preparation laws by doing his little street corner business.
HN dwellers would mostly debate for days about getting the right license or some other silly nonsense. Meanwhile tamale man is cooking, tamale man is selling, tamale man is doing his thing. Is he paying his taxes? Who knows. No one ever bothered to find out and from what I can tell nothing ever happened to him.
There's also probably the majority of the US who just make up laws that don't exist and then enforce it against themselves. Most people think they have to give an ID to a cop if he asks for it on the street.
They assume everything is illegal and self police because that's what the tyrant programmed them to do.
Same effect in HN comments also. Many people hold back from expressing their real views here because they are afraid to run afoul of the Flag Police.
They actually did change that law. Citizens of the Great Empire are now required to identify themselves to any cop who asks their name. For the childrens.
The "show him your government ID which you are now required to keep on you" part hasn't yet been instated, but it's coming. That will be introduced along with a whole heap of other big anti-freedom changes (like Central Bank Digital Currency) during the coming World War.
While I am a big fan of nuclear, I think the issue of land usage for solar is overblown. We use a lot of land for far less useful things. In the end, anything that helps us burn less fossil fuels, I am happy with.
You're also taking away farmland that could be used to produce all kinds of things. Most of the prime solar areas are the same prime areas for agriculture. By creating massive solar farms, you're at the same time, reducing acreage that could be used for range animals and other agriculture:
Modeling by the American Farmland Trust (AFT) finds that 83% of projected solar development will be on agricultural land, of which 49% will be on land AFT deems “nationally significant” due to high levels of productivity, versatility, and resiliency. In May 2024, the U.S. Department of Agriculture’s (USDA) Economic Research Service (ERS) reported that between 2009 and 2020, 43% of solar installations were on land previously used for crop production and 21% on land used as pasture or rangeland.
In a few years we'll have to deal with an impending disposal issue on farmland:
Forecasts suggest that 8 million metric tons of solar panels will have reached the end of their lifecycles by 2030. The National Renewable Energy Laboratory reports that less than 10% of decommissioned panels are recycled. Many end up in landfills at the end of their lifecycle, which could be problematic, according to researchers with the Electric Power Research Institute because panels could break and leak toxic materials like lead and cadmium into the soil. If decommissioned panels are not disposed of properly, they could contaminate the surface and groundwater in the surrounding area, making disposal a major issue for farmers and rural communities who rely on groundwater for needs ranging from crop irrigation to drinking water.
Agricultural land in large parts of the US is going through a massive degradation cycle. We are heading for dustbowl 2.0 especially now that a bunch of the weird land universities have been shut down. In short its being used wrong and left empty too long, meaning the top soil is blowing away. Not to mention the land drains stopping proper soaking leading to flash flooding and runoff events.
Depending on how the panels are put in place, the land and soil quality will increase significantly because its reverting to fallow and long rooted stabilising plants will have 25 years to build up the biome again. Converting land back to farming is pretty quick.
I understand the point your making, and I do agree with the end of life cycle issues. THere is going to be a lot of lead leaching into water courses if not dealt with properly.
If you replaced ONLY existing fields used to grow corn for ethanol, and turned those into solar panels, you would already exceed the entire current US demand for electricity.
Solar energy is a phenomenal use of land, of which we have enormous amounts of in this country.
reply