Why do we say “Talk of the town”?

People have been the talk of the town since ancient Rome, where the poets Horace and Ovid wrote of being the common talk of the city. In 1661 the London diarist Samuel Pepys wrote about learning “what the common talk of the town is.” The New Yorker still runs a column by that very name.