I find that -ffast-math is not so bad, so long as I develop and test with it from the beginning. It's much like any of the other more aggressive optimizations in that sense.

Plus, comparing against strict math as I go tends to highlight where I might have been about to do something dodgy anyway.