The town was founded as Thomlinson, but renaming rights were auctioned in 1791. The high bidder, who reportedly offered "five dollars and a jug of rum," changed the name to Grafton after his home town of Grafton, Massachusetts. Possibly as a result of having celebrated a bit too much with the rum (some say it was hard cider), the money was never collected.
In the early 19th century, sheep raising became popular and multiple woolen mills sprang up along the branches of the Saxtons River. Soapstone was quarried on nearby Bear Mountain. The town became a notable stagecoach hub for traffic across the Green Mountains into Albany, New York. One inn from that era, "the Old Tavern," was founded in 1801. It remains one of the oldest continually operating hotels in the United States.
Grafton had a population of almost 1,500 just before the American Civil War. The town suffered severe losses during the Civil War. Local cemeteries in the village hold many tombstones of casualties from the Battle of Gettysburg. After the war, the community declined in population. The soapstone quarry was depleted and closed late in the 19th century. Around the time of the Great Depression, the town's population was less than 400 (Info from wikipedia)