When The Block first landed in Daylesford, some questioned the impact the hit television show would have on the small town.