After the initial load from the CDN the tax is 10ms (Chrome), 20ms (IE9) or 80ms (Firefox). So for modern browsers it's really a tiny amount you save. For IE7 users the tax is more than 100ms but it seems a waste of time to optimize for them given their market share.

I'd say saving 20ms is premature unless you are already doing everything else you can do.