Whitehall is a monument to British constitutional history, a curious, unsatisfactory muddle of royalty, fleeting republicanism, imperialism and bureaucratic compromise. No wonder Prince Charles is a defender of architectural tradition. Like love and marriage, power and architecture go together, only much, much more. When Henry VIII took over his old tutor Thomas Wolsey’s mighty York Palace, which extended from Charing Cross to modern Downing Street, it was to make clear that he was now in charge. The rebranded Palace of Whitehall became the centre of the Tudor kingdom. Other palaces were mere satellites of the mighty labyrinth of chambers and walks and preaching courts and tennis courts that ran along the north bank of the Thames and either side of Whitehall. It was gated with fantastical archways that allowed the king to pass unseen from privy chamber to his new hunting ground in St James’s Park.

The decoration was so rich, the extent of the palace so awesome, that when its owners were absent it was thrown open to the public as a silent manifestation of their majesty. Elizabeth I added a banqueting house, for masques and entertainments such as Shakespeare’s The Merry Wives of Windsor. After her banqueting house burned down in 1619, it was replaced by the first major Whitehall commission for Inigo Jones. Later, months before his execution, Charles I also called on Jones to draw up plans for a new Whitehall Palace. It was to rival that byword for monarchical splendour, Versailles, a mighty reassertion of divine and royal authority. There was to be a long Palladian waterfront, behind which would lie a series of courts and crescents giving finally to the king’s private quarters overlooking St James’s Park.

But the central plan is not the British way. Long before a slab of Portland stone could be laid, Charles was taken from the upper floor of the Banqueting House, the only Jones building completed in Whitehall, to his death on the scaffold. All that survives of his plans is the circular court in the modern Treasury. After the Restoration, Charles II took up the ambition of monumentalising political London. He, too, was doomed to fail. Rebuilding London after the fire of 1666 fully absorbed his architect, Christopher Wren. Besides, the hostility of the London mob persuaded the king first of the need for tactical retreat and then of the wisdom of a more modest palace. Whitehall baroque was to be restricted to office systems. Little survives of the Tudor palace that was ravaged again by fire, except – astonishingly – Wolsey’s wine cellar, which was moved and lowered to fit into the basement of the 1930s Ministry of Defence.

There was no new masterplan for this prime central London site for nearly 350 years. Instead, as royal political influence waned, so royalty retreated from Whitehall and the machinery of government slowly expanded to fill the space. At first, politics was conducted in the salons and dining rooms of the mini-palaces built by the great political magnates in the ruins of the old palace. Then bureaucracy began to edge out from the wily turncoat George Downing’s eponymous (and jerry-built) terrace to consume the magnates’ houses. This was a government of the elite, and its new architecture reflected it.

So when, in the mid-19th century, a Foreign Office building was planned, Gilbert Scott’s proposal of a great Gothic revival building (think Midlands Hotel at St Pancras) with a piazza that opened on to Whitehall was rejected in favour of the closed, inward-looking Italianate building, a deliberate equating of the British with the Roman empire.

It was not until the 1960s that a plan to rival Charles I’s emerged. The Conservative minister for public building, Geoffrey Rippon, commissioned the Festival Hall architect Leslie Martin to rebuild Whitehall as a “national and government centre”. Unveiled with the glorious ambition of “ziggurats for bureaucrats”, it was howled down by those who preferred the accretions of tradition over the radicalism of the new age of bureaucracy. It was a harbinger of the death of the wartime supremacy of Whitehall, a hint of what was to come. The man in Whitehall did not, after all, know best.

Today a degree of interdepartmental coherence has been created behind the old façades along the west side of Whitehall. Although, to outward appearances, nothing has changed, there is now an almost continuous chain of government offices from Admiralty Arch to Downing Street broken only by Horse Guards: modern values in a traditional setting.

Colin Brown is a cheerful guide through this maze, full of anecdote and observation. But keeping up with his tour is at times a daunting challenge, because the book has one great weakness. It is woefully under-mapped and under-illustrated.