The site dates back as far as the Iron Age, with Walton Castle set upon a hill fort. Reference to it is also made in the Doomsday Book as belonging to “Gunni The Dane”. However the construction as we now see it took place, according to Pevsner Encyclopedia, between 1615 and 1620. It was designed as a hunting lodge for Lord John Poulett, a wealthy and renowned Somerset land owner and Member of Parliament.
The land surrounding Walton Castle had been acquired by marriage. The English Civil War marked a decline in such enormous and valuable holdings by absentee landlords and so by 1791, Walton Castle lay abandoned and derelict. The roof and floors fell in and parts of the building became a convenient home for dairy cattle. Despite its exposed position, the structure remained, meriting Pevsner`s description in 1957 as “remarkable as a piece of ornamental planning”. Planned restoration by the largest landowners in the area was abandoned, but was later taken up by a city financier, who returned Walton Castle to its former glory.