Unfortunately you are wrong. Time marches on and progress in silicon does too. You may not like the timeline, but it will happen.
While not exactly an “SSD”, High Bandwidth Flash will make it to market and make big sparse MOE models even more accessible. Everyone knows bandwidth is the game right now and you think hardware vendors are going to sit on their heels and not take part of the market? Absolutely silly to discount progress both on the software and hardware sides.
That’s what I was seeing too, and my cache hit rate was below 25% during that same time, leading to significant burn of my weekly limit (Pro 5x plan) via all the uncached input. Doesn’t prove that the overload caused the cache failure, but it does seem to point to some common infrastructure cause. No such problems (cache hit or system overload) with Terra or Luna.
Was that Codex/subsidised-usage or API? I do get overloaded in Codex-account from time to time, but API is rock solid.
They obviously load shed a bit of Codex-sub during peak times, and for the amount of tokens you get for a sub, I don't mind. I just mean the API where you pay-per-token is rock stable.
Right now I intend to maintain this as a library. The examples are just that, examples for programmers/agents. If someone in the community wants to step up to maintaining release binaries I will gladly have that support, it's just impossible to do as a sole maintainer
That's up to you of course, but is it that much more work to compile the cli binary at the same time as you compile the libraries? How am I supposed to actually use the Cuda binaries available in the releases section, through a separately downloaded Python wheel?
It's not that much extra work to compile, the extra work comes from the maintenance and feature requests. By not shipping the binary directly I am defending my time until other contributors want to step up and maintain things. I am one person with limited time and I don't want to spend all of it in front of a computer
Yes you use them through a wheel. If you have specific questions on packaging and how to use things lets move it over to the discussions/issues in the repo itself so it can be more broadly accessible to more people and we can make the packaging of the library as useful as possible
Wow, it's amazing to hear that even though I released this so recently people are already using it properly! Thanks! Please let me know any issues you run into
word boosting will probably come on a much longer time horizon, but streaming is here!
I'm really hoping someone either contributes a good server example to the codebase (and is willing to help with issues) or use transcribe.cpp or the bindings to create a robust server in another language :) would be happy to link it from the main project directly as well
Every model with open weights has some code which can be used to inference it. So we download the published weights and run against inference library they suggest, be it transformers, Nemo, etc
Largely this is out of scope for the library, mainly because I’m not aware of many models supporting this. but if there are models which support this would be happy to support
Yeah I’m working on it, Linux is a big pain point especially Wayland
Once things are more or less ironed out on MacOS and Windows a lot of attention will be turned towards Linux
I know a lot of Linux PRs are open it just takes me so long to get around and test them. And often multiple different implementations trying to fix similar issues which is a lot of overhead sometimes
Is there any way people can help? From your last sentence, it sounds like another PR isn't it and the opposite might be needed. But would love to contribute with testing if helpful. I'm regularly jumping between XFCE, KDE, GNOME, Niri, etc..
Testers by far as the most needed thing, I do maintain a list of per platform people who help to test so if you drop a GitHub username (or email me) I will add you to the list and ping for help
Basically the biggest blocker is me being the sole maintainer and reviewer at the moment and it just ends up taking a lot of time for the scale of the project. Which is why it moves slow and features typically are much slower than someone can vibe code. I know each added feature inevitably has bugs so I try to be careful with them.
But also Linux has historically been a minefield, fixing something for someone breaks for someone else so yeah testers really needed. Or anyone with deeper Linux DE knowledge than I have. I’m much more accustomed to server based Linux distros
I have a personal fork of hyprvoice[0] which I use almost everywhere now (w/ the big cohere-transcribe running on a local vLLM instance). It does a similar thing, but that's not why I'm mentioning it; I think it's worth looking at because it's a clean reference for the few elegant ways you can implement text injection in modern Linux (wayland).
It supports ydotool[1], wtype[2] and "clipboard fallback with clipboard restore". The first two you can probably think of as AHK equivalents - they wire in at the input layer and inject keystrokes when injecting text. wtype is wayland-only and a bit less invasive, ydotool supports non-wayland also apparently, but I haven't tried it. Neither approach provides 'instant text' - you have to watch the text get typed out, and you don't touch your keyboard while it's happening; the clipboard implementation is fallback for a reason as it's the least reliable. The first two work 'well enough' though, and are fairly tunable.
The other thing hyprvoice does in probably the most linux-friendly and universal way is the 'hotkey handling'. The server creates a socket in /tmp that the cli can then ping when the user triggers the start/stop/cancel, and they do this by binding whatever their DE's keyboard shortcut mapping mechanism is to trigger `hyprvoice toggle` as a background shell command. This works extremely well and is much cheaper than you'd intuitively think coming from Windows. This way you don't have to interface with DE-specific global keyboard listeners etc, but leave that to the WM (that's not to say that your installer couldn't prompt the user to configure the keyboard shortcut for them with their detected WM, you just wouldn't do it in the software itself).
I haven't actually looked at your project in too much depth yet as I have a solution for this already, so apologies if none of the above is news to you. Hope it helps though - happy to poke around and contribute something if the gap's still there.
we have wtype and ydotool support as well in Handy, but it's not exactly a great out of the box experience always, but some of it is on me for poor linux packaging
TensorRT and CUDA is effectively the same speed as CPU for the speech to text models I was testing via ONNX at a huge binary bloat penalty. WGPU is hard to ship and also equivalent speed or slower. This may not be the case for LLM or other models but the runtimes did not seem well supported for what I needed to do. ONNX is incredibly well optimized for CPU, best in class even, but the other execution providers at least for STT seemed lacking.
I did this investigation before creating transcribe.cpp it would have been much more convenient and save me literal months of work. Happy to share the repo and binaries produced as well, but it was mostly throw away work to profile how to ship accelerated ONNX in Handy.
While not exactly an “SSD”, High Bandwidth Flash will make it to market and make big sparse MOE models even more accessible. Everyone knows bandwidth is the game right now and you think hardware vendors are going to sit on their heels and not take part of the market? Absolutely silly to discount progress both on the software and hardware sides.
reply