But in the last few years it's gained a new lease of life thanks to the young royals, and it now seems to be an appealing place for many of them to call home.
The Cambridge gang now live in a lavish home on the site, and Prince Harry and Meghan Markle will be moving in next door after their wedding.
The couple lived in the small two-bedroom Nottingham Cottage, but they are due to be upgraded to Apartment 1 after their wedding.
It's the second biggest on the site, and work is already underway to get the lavish 21-room wing all ready for them. But the Cambridges won't be their only royal neighbours, as Kensington Palace is full of members of the royal family - 15 to be exact.
Take a look at all Harry and Meghan's neighbours.
The Cambridges - Apartment 1A
Wills, Kate, George, Charlotte, baby Louis and Lupo the dog now live in the biggest home on the site - Apartment 1A.
It boasts six-bedrooms and was once home to Princess Margaret.
After their wedding Kate and Wills spent £4.5million doing it up.
Princess Eugenie and Jack Brooksbank - Ivy Cottage
The engaged couple are the newbies in KP, having moved in just last month.
They live in Ivy Cottage, which is just around the back from Harry and Meghan's current home.
Eugenie is Prince Harry's cousin, and they're said to be good pals who enjoyed partying together.
The couple will marry in St George's Chapel in Windsor Castle, the same venue as Harry and Meghan, in October.
The Duke and Duchess of Gloucester - Apartment 1 (for now)
Prince Richard, the Queen's cousin, currently lives in Apartment 1 however he's about to be booted out to make room for Harry and Meghan.
The couple's children have now all grown up and flown the nest, so they've agreed to move out so the new couple can have more space.
According to the Daily Mail, they're moving into the Queen's private secretary Sir Christopher Geidt's old apartment.
It's a lot smaller, but apparently they don't mind...
The Duke and Duchess of Kent - Wren House
Prince Edward is also the Queen's cousin and he lives in an apartment with his wife Katharine.
Prince and Princess Michael of Kent - Apartment 10
Michael is another of the Queen's cousins, but their living situation caused a bit of controversy back in 2001 when it was discovered the couple were paying peanuts to live in KP.
They had a seven-bedroom home with nine-reception rooms but were paying just £67-a-week in rent, reports the Telegraph .
After the scandal the Queen decided to start charging her family the market rate for their palace homes.
// <\/s'+'cript>'); // --> //]]> From ICE
Tags:
news