The old castle was constructed around 1100 by the cadet branch of the House of Wittelsbach, but demolished between 1398 and 1403. William IV of Bavaria and his son Albert V ordered the construction of a Renaissance style four-wing palace with a court garden on the site of the old castle. The new building was designed by Heinrich Schöttl. Construction began in 1546 and was completed in 1577. It later became the favoured residence of the rulers of Bavaria. In 1715, Maximilian II Emanuel commissioned a redesign in Baroque style by Joseph Effner.