Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Once I start a conversation about Postgres query plans, maybe 95% of the knowledge will be almost certainly not needed, and for an inference provider there will be many more concurrent queries with reasonably large overlap. Maybe future architectures will be able to take advantage of this so not all parameters are needed in memory for almost all instances.

If we are talking about all knowledge, then I agree that the compression ratio is very impressive already.

 help



Mixture of Experts is already used by pretty much all modern LLMs to address exactly this phenomenon.

Hopefully, future models can be trained to be even more aware of external knowledge, accessible through web search / RAG / whatever it will be then, and might not need to internalize much knowledge at all.


> future models can be trained to be even more aware of external knowledge

Then you need longer contexts, which is proving to a much more stubborn problem than general knowledge compression.


I suppose the right approach would be to run queries like the ones you want your model to be able to do, sort weights/experts by usage frequency and then reareange the weight so that all but the most frequently used can stay on disk. Tricky though, it could be that a e.g. database question uses 90% of the network at some point or another

Edit: I guess at the moment this is just having an LRU cache of experts


Note that MoE is a sparsification mechanism and doesn’t actually have to do with expertise or what would commonly be considered areas of expertise.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: