There's some evidence that he gave U2 spy plane secrets to the USSR. They didn't trust him and put him to work in a toy factory. He later came back to the US with a Russian bride that was related to a KGB officer. Unlike everyone else who came back, he was not debriefed by the CIA. They just welcomed him back.

Compare that with how they treat Chelsea Manning.