"Most southerners didn't own slaves" is another popular argument from Confederate apologists.
SA is in a hard place, with the US having used all the good rationalizations over the last 150 years, as well as the example of how Germany dealt with its Nazis.