The land east of the Palladian Bridge and rising to the Gothic Temple was part of the manor of Lamport, owned by the Dayrell family. It was purchased in 1826 and the manor house demolished by Richard Temple. His son, landscaped the area from 1840; Mr Ferguson was the gardener and Edward Blore the architect.

From the Stowe 1848 Sale Catalogue:

On that side of the gardens adjoining [the Palladian] Bridge, formerly stood an ancient manor-house belonging to the Dayrell family: it was pulled down about ten years ago, and on the site of it and its gardens and ponds a large enclosure has been made, for an ornamental plantation of rare and curious shrubs and trees; and a menagerie, for the reception of rare animals and aquatic fowls—this party of the grounds being supplied with water. This beautiful spot was converted into its present picturesque appearance of hill rock and dale, rock-work and waterfall, by the scientific genius of Mr Ferguson, the present gardener at Stowe, whose taste in landscape gardening is worthy of his predecessor, “Capability Brown.” Within the present enclose also stood the Vicarage House.

Near yonder copse, where once the garden smiled,
And still where many a garden flower grows wild;
There, where a few torn shrubs the place disclose,
The village preacher's modest mansion rose.[1]

And immediately adjoining, the school-house of Stowe, which was also pulled down when the pleasure grounds were enlarged.

Beside yon straggling fence that skirts the way,
With blossomed furze unprofitably gay,
There, in his noisy mansion, skilled to rule,
The village master taught his little school.

[1] Oliver Goldsmith (1730—1774), “The Deserted Village.”