Can confirm, it is quite painful.

The bpftools maintainers tell you to learn the bytecode format when you ask them what the errors mean, because they expect you to understand what the verifier means when it tells you "unknown scalar" on every single goddamn line of code.

Something like "ebpf coding rules, what to use and what not" would be very helpful.

All ebpf examples that are older than say, 3 months, already don't work anymore. Not even the official ones from the XDP tutorial project (and the libxdp maintainers because the kinda are splitting off a lot of headers into a separate xdp library as it seems).

Most userspace code still relies on the 5 years old bpf-helpers.h, which meanwhile is not supported anymore because it doesn't use the __helper methods from the kernel (they also refactored the kernel in the meantime, and force you to use e.g. __u128 instead of native data types).

Oh boi, did I underestimate what "bytecode vm" means when kernel developers talk about it.

Also, always use llvm, and remember to build two bpf files for each endianness, and use -g for debug symbols. Otherwise you will try to find out what the bpftool errors mean for days, because of shitty mailing list answers.

I mean, the flip side of this is: do you really expect people on a mailing list to debug your custom XDP code for you? You get what you pay for, and you can pay people to help you with this stuff, or not.

It would be much easier to write a kernel module than an eBPF program. But the eBPF program is unlikely to panic your machine, and the kernel module is almost certain to.

Also In cloud environments, I would much rather trust an eBPF program, than a kernel module.

Why in cloud environments?

You don’t want kernel panic affecting other users

Why would your cloud instance panicking affect other users of the cloud provider?

Or do you mean something else?

Anecdotically, I had the case on VMWare 4 (that was in 2012 or 2013) that a Solaris 11 VM managed to reboot the entire ESX it was hosted on. Very weird bug where ESX passed through some interrupt or something.

But in this case I think they mean on the same machine. "In production" would be more accurate than "in a cloud environment". And yeah I wouldn't load custom kernel modules in production just to do observability.

Cloud goes beyond rented VMs. Fully managed cloud services have thousands or millions of production customers on the same node. They have to be very careful about what they run as root.

I understand your point, but millions sounds an exaggeration- I have a hard time believing a single node can handle millions of concurrent users

I didn't mean to imply concurrent. A large fraction of the user base is very sporadic in its usage!

Thanks for clarifying :)

Reading more about that… How does the verifier detect infinite loops anyway? Halting problem and all. It must use some rather crude heuristics, no?

It doesn't. Flip things around and you get something tractable, though incomplete.

Unsolvable problem: reject any loop that is provably infinite.

Solvable problem: reject any loop that isn't provably finite.

The trade off is that there will always be some loops that in fact always do terminate, but that the verifier can't prove do.

You need to slightly modify your code. Rather than:

while (condition) { … }

Do: #define MAX 1000 for n = 0; n < MAX; n++ { if !condition break; … }

Unroll all loops, don’t allow any backward jumps and limit to (say) 1m instructions.

Incidentally, iteration limits are a good idea for production code anyway. If you don't imagine any input needing more than 50 k iterations, throw a user-friendly exception after something like 10 M iterations. Prevents much more annoying problems than it causes.

> If you don't imagine any input needing more than 50 k iterations

What could possibly go wrong.

You get an error is the worst that happens.

Way better than running a denial of service attack on your own systems or those of your customer's.

> You get an error is the worst that happens.

That certainly depends on what eBPF is used for. If your load balancer errors out at [greatest number of connections envisioned] and an adversary manages to establish [greatest number of connections envisioned] then the result is a denial of service.

Not every operator is confident in making code changes in 3rd party software or might even be allowed to make such changes. Increasing resources o.t.o.h., e.g. adding RAM, is rarely banned. I sure would want a system to make best use of available resources.

I still think a denial of service due to tripping some sort of circuit breaker is preferable to one due to resource exhaustion.

If the code is intended to use as a library or the binary distributed to third parties one will have to handle it differently. For libraries taking a parameter indicating the maximum expected is common, for example. See e.g. man 3 read.

Nice. Be good if the language had an easy way to handle it, eg.:

while(condition)[1000 label]{…}