Hawaii is already heavily agricultural. Most of the non-agricultural land is reserved for conservation. Banana production would likely replace other production, and then we've got less of that stuff.

Also, shipping to continental US is limited by the Jones Act and the lack of capacity in US built, owned, and crewed shipping lines. Assuming a desire to produce things in the US, I don't think it's sensible to tarrif bananas to grow them in Hawaii, and then relax the Jones act so they can be shipped to the continental US on foreign carriers.