Buckingham House was at 91 Pall Mall. It was one of the great Pall Mall mansions and was named after the 1st Marquess of Buckingham (Nugent Buckingham), who inherited it from his uncle. It previously belonged to Thomas Pitt, grandfather of William Pitt the elder. 

Shortly after Nugent Buckingham inherited the property, he purchased the house next door. Thomas Grenville lived there for a while but after the Crown lease was renewed in 1783, Buckingham began to create a grander house, initially retaining R F Brettingham, who had worked at Grenville's previous residence of 9 Berkeley Square. From 1790, he retained Sir John Soane to create a house commensurate with Buckingham's social and political pretensions. The two houses were knocked into one and Soane gave the house a three-storey, seven-window facade of Portland Stone. It was rusticated on the ground floor and had a stone balustrade. Within the building, Stone built the main staircase and probably several reception rooms (see Saving Wotton, page 23). Work was complete in 1795 at a cost of £11,000. 

While Soane's work was in progress, the Buckinghams lived in Paddington while in town (HEH STG Correspondence Box 74 (01); Box 74 (35))

Buckingham House was sold in 1847 by the 2nd Duke of Buckingham and the effects auctioned by Christies. The Carlton Club occupied the house in 1854–5 while its clubhouse was being rebuilt. Thereafter it was occupied by the War Office until 1906. It was demolished in 1908 and the Royal Automobile Club was built on the site.

Sources

Weinreb & Hibbert. Beckett; Saving Wotton. 

For a full description and architectural history of the house see: 'Pall Mall, South Side, Past Buildings: No 91 Pall Mall: Buckingham House', Survey of London: volumes 29 and 30: St James Westminster, Part 1 (1960), pp. 360-363.

Buckingham Palace

The Buckingham House above should not be confused with the house now known as Buckingham Palace. That Buckingham House was built in 1702 for John Sheffield, Duke of Buckingham and was purchased by George III in 1761. It was then known as the Queen's House and more recently as Buckingham House or Palace.