if you are talking about rent and general cost of living. we have discussed it recently http://akihabara.tokyo/question/12379/whats-the-cost-of-living-in-japan/
as for the difference between big and small cities... There is a huge difference between Tokyo (and maybe Kyoto) and the rest of the country. The same apartment/house will cost twice as much in Tokyo as in a smaller city like Toyama (my only reference city, sorry :)).