"external drive" is not what a smart card is. You can not read the key from the smart card. The actual crypto operations involved in performing the authentication are handled on the card it's self.
If your key is on a smart card it can not be stolen, even if the machine you're ssh'ing from has been compromised. This has real value and is not "insane"
In your setup, if their machine is compromised, it doesn't matter if they password protect the key, the attacker can key log the key password or replace the ssh executable with one which leaks the key, then ssh into your servers at will.
"If your key is on a smart card it can not be stolen, even if the machine you're ssh'ing from has been compromised."
If the machine you're ssh'ing from has been compromised by a halfway competent attacker, there's no need for them to get the key from the smart card. They can log in to the remote machine by simply piggybacking on to your ssh session (with a compromised ssh client).
If your key is on a smart card it can not be stolen, even if the machine you're ssh'ing from has been compromised. This has real value and is not "insane"
In your setup, if their machine is compromised, it doesn't matter if they password protect the key, the attacker can key log the key password or replace the ssh executable with one which leaks the key, then ssh into your servers at will.