For athletes who are afraid of losing their titles in 2020, this could give a good excuse. I remember a few using the air quality of Beijing for the same purpose :-p
I was in the organizing committee of a big conference (200 attendees) in 2012, so I have faced this problem.
Ground radiation levels in Tokyo are not any different from most major cities in other countries Some cities have higher levels if the ground is granite based). Even if people visit the border of the evacuation zone for an hour, they won't absorb more radiation than they do in a flight from US to Japan.
The food, those who are scared can worry. I am sure that the athletes will have all ingredients well-checked, but no one can guarantee that 0% of the food comes from regions close to Fukushima and are completely free of radiation. The evacuation was a good choice to prevent any food being produced there, though. Even then I doubt that eating in these areas for a few weeks could pose any risk.
People have a right to worry about their safety, and some of them are paranoid. So, I guess there will be some who won't go see Olympics citing radiation fears. I can also see some Chinese media plotting to spread a few stories when the event is near. All in all, I think the event will go on with no problem. The event that I was involved was much less important, but we had a full house event even in 2012.