I've recently played with the Korg Monologue, it's a solid machine, analog synthesizer and step sequencer, enough standard ports to extend it eventually - a criticism to the device mentioned in the OP is that it's pretty nonstandard in a bigger workflow. It's < $300, still not cheap but a pretty good package overall.
A lower price is the Korg Volca, at $100 - $150 per unit. They're less versatile though and you'll probably end up having to buy / link up multiple.
Even lower and you're looking at stuff like the Teenage Engineering Pocket Operator series, single card devices in the $70 range. There's a wide range of variations, each of which (correct me if I'm wrong) has 16 instruments and a programmable step sequencer, 16 patterns preloaded. The preloaded patterns have only 16 steps, but you can program it to have more than that and switch between patterns every 16 beats. You can link up multiple together as well, they can synchronize with one another (iirc you have to hook up your phone as the first step which generates a synchronization signal).
Arturia microbrute is a great first analog monosynth.
The Arturia microfreak is an interesting new digital synth in a similar form factor.
The Yamaha Reface CS is an 8-voice virtual analog synth with a very understandable layout. Basically a juno 106 in an apartment-sized form factor.
As someone else mentioned, the volcas are a great starting point, and the korg monologue has an oscilloscope built in that makes learning sound design surprisingly understandable.
Just my opinion, but I think you should stay away from the Roland Boutique synths. I found them very unpleasant to use and not inspiring at all. I have not used the 808/909 drum machine boutiques - they may be better.
You can always go vintage. E.g. Some of the 2000s grooveboxes (electribes, various roland mc-xxx things) can be had for $100-200 on craigslist, and will be a ton of fun.
Korg KO2S. It's a synthesizer, but in a rather unusual package (so if you're looking for something with a piano keyboard, look elsewhere). It's a huge amount of fun and you can at least to some extent write proper music on it. It's also for what it is capable of very inexpensive.
Same as sibling comments, and I would add that if you’re interested in modular fun, there is now a Volca Modular and in a slightly more expensive bracket, the Behringer Neutron has a ton of features for the money.
