I could be wrong here, but I believe this (ceph, et al) is the answer to the question:
> """But what if I don't have a JBOD of 6x18TB hard drives with good amount of ECC RAM for ZFS?
What if I have 3 raspberry pi 4's, at different houses with 3x 12TB externals on them, and 2 other computers with 2x 4TB externals on them, and I want to use that all together with some redundancy/error checking?"
That would give (3x3x12)+(2x2x4)=124 TB of storage, vs 108TB in the ZFS single box case (of raw storage).
If you could figure out the distributed part (and inconsistency in disk size and such), then this is a very nice system to have.
"What if I have 3 raspberry pi 4's [...] with no significant performance requirement?"
Ceph is nice, but performance is lackluster on anything but a proper cluster (pun intended).
It's also somewhat heavyweight. I ran ZFS over iSCSI with four RPis serving the iSCSI targets via SATA-USB. It was network limited mostly. The advantage of that is that you can take the same disks and plug them all into a single host and import the pool directly (ie not via iSCSI), if needed.
The core concept is rather simple though: iSCSI allows for an iSCSI server to expose raw block devices (called targets[1]), so do that and use them from a client machine as-if they were locally connected disks.
So I used LIO[2] as the iSCSI server on each of the Pi's, making sure to use the "by-id" to reference the block device so it would work fine across reboots (skipped changing "directories"):
sudo targetcli
/backstores/block> create name=block0 dev=/dev/disk/by-id/usb-SAMSUNG_MZ7PD256HAFV-000_0123456789000000005-0:0
/iscsi> create
/iscsi> cd iqn.2003-01.org.linux-iscsi.rpi4.armv7l:sn.fc1b1c9879a1/
/iscsi/iqn.20....fc1b1c9879a1> cd tpg1/luns
/iscsi/iqn.20...9a1/tpg1/luns> create /backstores/iblock/block0
Note you might have to use /block/ instead of /iblock/ in the last command there[3], depending on targetcli version.
Then you create the portal which exposes the target (backed by the raw disk) to the network as normal (see Debian guide fex). I did it like this, you have to adjust for the name of your device:
/iscsi/iqn.20...1c9879a1/tpg1> cd portals
/iscsi/iqn.20.../tpg1/portals> delete 0.0.0.0 3260
/iscsi/iqn.20.../tpg1/portals> create 10.1.1.101
/iscsi/iqn.20.../tpg1/portals> cd ..
/iscsi/iqn.20...1c9879a1/tpg1> set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1
This creates a portal which exposes a single LUN, backed by the raw disk, on the IP 10.1.1.101, and disables authentication and demo mode (write protection).
Next you install the iSCSI client (called initiator)[4], I did this on a separate machine. There I connected to each of the target on each Pi. Once successful you should get a number of disks under /dev/disk/by-path/ip-*, which you can then specify when importing the pool on the iSCSI client machine, say
Since you're exposing the raw block device, there's no difference in that regard to having the disk plugged into the client machine directly.
Btw, I recommend not messing with authentication at first, though it's not terribly difficult to set up.
edit: As mentioned the nice thing about this approach is that it's effectively "harmless". If it doesn't work out for you, you can always put all the disks in a single machine and import the pool as normal.
Also, iSCSI has a lot of robustness built in. For example the iSCSI client will temporarily store writes and re-issue once the target (server) is back online. I safely rebooted one of the Pis while copying data to the pool, for example.
If you could figure out the distributed part (and inconsistency in disk size and such), then this is a very nice system to have.