'Unspoilt market town' has all the charm of very famous neighbour without crowds