Companies that have no interest in growth and are already heavily entrenched have no purpose for increasing output though. They will fire everyone and behave the same?

On the other hand that means they are weaker if competition comes along as it's expected that consumers and business would demand significantly more due to comparisons.

comparing a laptop to a A100 (312 teraFLOPS) or H100 (~1P FLOPS) server is a stretch to say the least.

An M2 is according to a reddit post around 27 tflops

So < 1/10 the performance of just computation. let alone the memory.

What workflow would use something like this?

They aren't going to be using fp32 for inferencing, so those FP numbers are meaningless.

Memory and memory bandwidth matters most for inferencing. 819.2 GB/s for M2 Ultra is less than half that of A100, but having 192GB of RAM instead of 80gb means they can run inference on models that would require THREE of those A100s and the only real cost is that it takes longer for the AI to respond.

3 A100 at $5300/mo each for the past 2 years is over $380,000. Considering it worked for them, I'd consider it a massive success.

From another perspective though, they could have bought 72 of those Ultra machines for that much money and had most devs on their own private instance.

The simple fact is that Nvidia GPUs are massively overpriced. Nvidia should worry a LOT that Apple's private AI cloud is going to eat their lunch.

> comparing a laptop

Small correction: the M2 Ultra isn't found in laptops, its in the Studio.

I find the entire field lacking when it comes to long-horizon problems. Our current, widely used solution is to scale, but we're nowhere near achieving the horizon scales even small mammal brains can handle. Our models can have trillions of parameters, yet a mouse brain would still outperform them on long-horizon tasks and efficiency. It's something small, simple, and elegant—an incredible search algorithm that not only finds near-optimal routes but also continuously learns on a fixed computational budget.

I'm honestly a bit envious of future engineers who will be tackling these kinds of problems with a 100-line Jupyter notebook on a laptop years from now. If we discovered the right method or algorithm for these long-horizon problems, a 2B-parameter model might even outperform current models on everything except short, extreme reasoning problems.

The only solution I've ever considered for this is expanding a model's dimensionality over time, rather than focusing on perfect weights. The higher dimensionality you can provide to a model, the greater its theoretical storage capacity. This could resemble a two-layer model—one layer acting as a superposition of multiple ideal points, and the other layer knowing how to use them.

When you think about the loss landscape, imagine it with many minima for a given task. If we could create a method that navigates these minima by reconfiguring the model when needed, we could theoretically develop a single model with near-infinite local minima—and therefore, higher-dimensional memory. This may sound wild, but consider the fact that the human brain potentially creates and disconnects thousands of new connections in a single day. Could it be that these connections steer our internal loss landscape between different minima we need throughout the day?

Yes... The field lacks the HOLY GRAIL (long-horizon problems). But we don't need a mouse-brain to sort spam emails. The Hail Mary 2B+ parameter models and above are still niche uses of these algorithms (too heavy to run practically). There is plenty of room for clever and small models running on limited hardware and datasets to solve useful problems and nothing more.

Models that change size as needed have been experimented with, but they are either too inefficient or difficult to optimize at a limited power budget. However, I agree that they are likely what is needed if we want to continue to scale upward in size.

I suspect the real bottleneck is a breakthrough in training itself. Backpropagation loss is too simplistic to optimize our current models perfectly, let alone future larger ones. But there is no guarantee a better alternative exists which may create a fixed limit to current ML approaches.

I'm not proposing a change in model size; rather, I'm suggesting a higher dimensionality within the current structure. There’s an interesting paper on LLM explainability, which found that individual neurons often represent a superposition of various data elements.

What I’m advocating is a substantial increase in this aspect—keeping model size the same while expanding dimensionality. The "curse of dimensionality" illustrates how a modest increase in dimensions leads to a significantly larger volume.

While I agree that backpropagation isn’t a complete solution, it’s ultimately just a stochastic search method. The key point here is that expanding the dimensionality of a model’s space is likely the only viable long-term direction. To achieve this, backpropagation needs to work within an increasingly multidimensional space.

A useful analogy is training a small model on random versus structured data. With structured data, we can learn an extensive amount, but with random data, we hit a hard limit imposed by the network. Why is that?

I believe they do know this.

However the real challenge would be:

1. bring this mapping into a AI framework for inferencing 2. We don't know the "OS" on how it runs. Just randomly triggering a neuron probably wouldn't work as there is a lot of other factors that trigger neurons.

continually learn with a fixed computational budget over massively extended task horizons.

No, but it could help me make such a thing.

Possibly, but as of now it's a completely unsolved problem and to my knowledge nobody has shown even a tiny model being able to perform it.

Based on the top page today I may even be able to make the argument we can't even simulate the abilities of a fruit fly.

The absolute frontier models can perform only a fraction of a fraction of what a typical work day looks like for a human. I calculated the chance that a frontier model today has about a 1^-29 chance of performing a single day of connected tasks based on GAIA benchmarks.

with these methods the issue is the log scale of compute. Let's say you ask it to solve fusion. It may be able to solve it but the issue is it's unverifiable WHICH was correct.

So it may generate 10 Billion answers to fusion and only 1-10 are correct.

There would be no way to know which one is correct without first knowing the answer to the question.

This is my main issue with these methods. They assume the future via RL then when it gets it right they mark that.

We should really be looking at methods of percentage it was wrong rather then it was right a single time.

This sounds suspiciously like the reason that quantum compute is not ready for prime-time yet.

It's RL so that means it's going to be great on tasks they created for training but not so much on others.

Impressive but the problem with RL is that it requires knowledge of the future.

So I feel I strongly fall in a poor performer interview category any time any code problems come up. How would I convince you I do not have a fraudulent resume?

I study hours every day for many years now. I know many complex systems however studying algorithms bore me to tears.

I've built HPC clusters, k8s clusters, Custom DL method, custom high performance file system, low level complex image analysis algorithms, firmware, UIs, custom OS work.

I've done a lot of stuff because I can't help wanting to learn it. But I fail even basic leetcode questions.

Am I a bad engineer?

There seems to be no way for me to show my abilities to companies other than passing a leetcode but at the same time stopping learning DL methods to learn leetcode feels painful. I only want to learn the systems that create the most value for a company.

I imagine if you interviewed me you would think I wrote a fraudulent resume. Not sure how I am supposed to convince someone otherwise though. Perhaps I've been dumb in not working on code that can be seen outside of a company.

There are people who literally say this and then you hire them -- they turn out to be complete duds. I'm genuinely curious because I'm hiring right now: by what mechanism would I discover that you have these skillsets and are good at what you do?

I was somehow reminded of this guy who once wowed me in an interview by coding up a small graphics demo rather quickly. Turns out later that that was the exact one program he could code without being hand-held EVERY SINGLE MOMENT. I laugh whenever I remember that incident (from early on in my career, in my defense).

You have to build a repertoire of questions that defeat rote memorization, prove real experience, and show genuine ability to solve unseen problems...

I remember TA'ing African exchange students in Haskell.

They could remember the exact type signature of standard library functions.

They could define a Monad instance from memory at the speed they could type.

But you couldn't ask them a single question outside of what was presented at lectures.

They couldn't solve a single assignment. They were stuck on rote learning.

I'd blame their educational system, because it was quite consistent (sample size = 3).

The classical example: Teacher says X, the whole class repeats in choir X.

There's an interesting bit in "surely you're joking, mr Feynmann" where he is amazed how primary school kids are learning physics. Only, as it turns out, nearly noone in the entire country has any actual understanding of physics, because from elementary school upwards all they do is memorize the material.

https://v.cx/2010/04/feynman-brazil-education

I've had a lot of success asking about fuckups, worst thing they've had to debug, and general "fuzzy" questions that specifically do not have a single answer, or the answer is so relatable/specific to a person's experience. Then you have to watch them as they answer.

What's so bad about getting a feel for someone and hiring them on a probationary basis? See how they go on discrete - but real - tasks for a week or two, and then make the call on going forward or not. It's how most hiring has always worked (and continues to work) in almost all other fields.

Also, re technical questions, I don't think anyone is saying that you can't ask any technical questions whatsoever, I think the concern is about giving people abstract, theoretical CS problems that will never actually come up on the job, on the very iffy assumption that their performance while being asked to dance for a committee in high-pressure job interview situation is going to be reflective of their actual skills. (And more broadly, that a good programmer must be quick on their feet with schoolboy-style CS puzzles that are basically irrelevant to most roles.)

I have no idea if this is accurate or not, but there is this general stigma from both sides that "no decent programmer worth hiring fulltime in hotter markets/companies would/should agree to a contract/temp/trial job".

I think de-facto fulltime jobs i've seen end up kind of like that - if you don't work out in X months, nobody will shed a tear about kicking you out - but this is still perceived as a more expensive operation that should be avoided by having a "proper" interview process. On which nobody can ever agree, but that's the sentiment.

A trial period is part of every permanent employment contract herearound. No matter the job.

Both parties can terminate the employment agreement within a week for (usually) the fist three month of employment.

No stigma involved.

I think it’s fair for both parties, they both need evaluate each other.

IMO this is where a comprehensive System Design interview weeds out the hopeless. Minimum 2 hours, don't just use a "design facebook/twitter/insta" scenario because anyone can memorise those, dig into the weeds as and when it feels appropriate, and keep the candidate talking through trade offs, thought process etc so you can really peak inside their head. The catch is that you also need to be very competent and know the design and all permutations of it inside and out.

Leetcode et al. are just testing rote memory, there's no need to have candidates actually type out solutions its a waste of time. So long as they can articulate what solution they would use, why, and what other solutions they considered that's all you really need to be concerned with.

So to preface, I'm not looking for a job (trying to build my own company)

When I do interviews (probably limited compared to you but some) I do it like I wish someone would interview me.

I focused purely on curiosity. how many things disparate things are they interested in, the things that overlap with my knowledge I probe deep. I believe in Einsteins quote.

"I have no special talents. I am only passionately curious."

If someone knows about how RDMA and GPU MIGS work they are probably pretty damn interested in how HPC clusters function. More importantly can they compress this information and explain it so that a non technical person could understand?

There are so many endless number of questions I could ask someone to prob their knowledge of a technical field it kind of upsets me that most of the time people ask the most shallow of questions.

I believe this is because most people actually study their fields a very limited amount, because most people are honestly not truly interested in what they do.

The biggest implication of this is that I may be able to tell if someone has this trait but I understand that the majority of people could not as they literally don't know the things they could ask.

Asking system designs of me if you aren't knowledgable of the field would probably be the easiest to see the complexity of systems I can build.

It depends on the role. In web development, for example, there are lots of non technical things people can be good / great at that a coder might not be. Things like CSS, HTML, accessibility, semantics, things like that. Lots of JavaScript people have no idea how to leverage CSS and over engineer things or find it impossible to implement certain features or if they do, it's 1000s of lines of code when it's really just a few lines of CSS.

Take home project limited to 2 hours.

Chat gpt can solve any take home problem. You need to be on video making interactive feedback and demands to see that someone can actually write code and understands programming even a little bit.

Hey, what are you hiring for exactly? May I email you? My email is in my profile.

Are you self-taught? How did you write the on-the-job code? Do you do well with take-home exercises?

Do you struggle with work projects that feel boring or abstract?

Self taught, come from EE background.

Was originally building firmware and hardware for previous company while testing DS on their systems. They liked my work and I switched to DS and ended up a team lead.

Honestly have never had a take home exercise but would love it if I could. I basically make my own work, if I don't get work I will build other projects for them and try to sell it to the company.

I normally make good value projects and can sell it. It's how I went from hardware to DS lead of a team in a few years.

You sound pretty cool and useful to companies that get it. My suggestion is to find ways to present on some of the things you’ve done. Then you can point to the presentation video/slides, plus you’ll probably receive inbound inquiries from companies that could actually use creative thinking and building.

The take home exercise/work sample companies are out there. Here’s a list I helped contribute to. https://github.com/poteto/hiring-without-whiteboards

Yeah I'm EE and CompE background, and been in software for 20 years, it gets old with the 26yo "senior" dev interviewing you rolling his eyes like you are some big liar for not wizzing through his CS quiz questions (oh and for front end positions, lol). Studying EE was much harder and well-rounded engineering program compared to CS degree at the time I was in school.

Why are you doing all those things and what jobs are you applying for? Can you solve fizz buzz in an interview setting?

I'm not bad at getting jobs, just feel when I am looking I believe people think I am lying about my project experience.

I do these things because I see a place in the company to grow value and do it.

I can write almost all basic coding and my true skill set is in custom complex DS pipelines at scale.

Fizz buzz is certainly not a replacement for expertise, which is what the above commenter seems to be emphasizing.

Naturally it's a "low bar" but it's an awfully low bar for a job that isn't entry level.

That's why it's so concerning when you have people who confidently say they're great engineers who are experts in firmware, UIs, operating systems, and networking, yet when you ask them to write a 3sum or remove from a binary tree they freeze up and blame it on an unfair interview structure.

I'm a bit reminded of a chef who claims he can cook pasta, sushi, and pastries, yet when asked to fry an egg says that it's not fair to expect him to do so on the spot. A chef who can't cook when needed is about as useful as an engineer who can't code when needed.

There's a common double standard here though - when I've asked backend-type devs to do a UI sample/question (these are often "full stack" roles), the leet code geniuses dismiss my concerns as that not mattering if they flop it.

I imagine we could do this now but not the way you think.

have a human created story and text as a guideline.

With that have genAI make the text per stage, you would get different statements every time and would stay on track.

Would be interesting to play a game where all players say the same information in slightly different ways every single playthrough.

Doing any job for more than an hour without completely forgetting it's goals and tasks

How long do you expect LLMs/agents to be unable to do this?

Good question, I'm working on exactly this, I suppose you could call it the replacement of RAG.

It's actually not very easy to achieve this. I could give a very long winded answer (don't tempt me) but suffice to say it's a resolution problem.

All AI have a fixed resolution on creation. Long running tasks focus on a very particular narrowing space per step, the resolution required for an infinite task is infinite resolution.

No 9s of error will ever fix this.

Funny enough, small animals do this with ease so I strongly disagree the idea that our AI outcompete even small mammals in every way.

Personally, I think that phenomenon (along with "hallucinations") is fundamentally baked into LLMs writ large.

I think LLMs are a dead end on the path to AGI.

I agree. Whenever people complain about LLM hallucinations they behave like they never seen one in humans.

Not only humans hallucinate all the time, humans also have persistent hallucinations as evident from the presence of opposing beliefs in various slices of society.