Estate of Vrbovec dominion enters history by a document dating from 1140 in which Peregrin, patriarch of Aquilea, announces that noble Dietbald de Chagere and his wife Truta relinquished their allodial in Gornji grad along with the castle and other estates to a church of Aquilea. It is possible that already at the time the old castle (Alterburg) was standing, of which location is not yet completely clear, but D.Kos presumes that the castle wasn't yet built at the time. The castle is explicitly mentiones in 1248 as Castrum Altemburch. It must to be mentioned that also Slovene name Vrbovec - castle built among willow trees (Slovene vrba=willow tree) - is very old even if in medieval documents its German designation is consistently used: in 1361 vest Altenburch, in 1369 Altenburch die vest and Altemburch castrum, in 1434 haws Altenburch, in 1447 "new Vrbovec castle" - newe vest Altenburch and in 1457 Altemburch ob Presperch gesloss.
First lords of the castle were patriarch's stewards among which Wulfing of Letuš or Luče is mentioned in 1248 and magister Gregorij around 1270-80. Patriarch's stewards are last mentioned in 1281-82 and in 1286 first mentioning of knight of Vrbovec occurs. They are not anymore directly subordinate to patriarch but are liegemen of Heunburg, count from Carinthia which evidents that patriarch has given his shares of estates in Gornji Grad to lords of Heunburg. Knights of Vrbovec can be found in many documents as companions of lords of Heunburg and when they died out in 1322, lords of Vrbovec took control of Vrbovec estate into their hands. As liegemen they were in constant disputes with monastery of Gornji Grad and these disputes went over the edge in the litigation for Rudenek castle near Rečica. It seems that knights of Vrbovec later became poor because already in 1349 or 1350 they left Gornji Grad castle to lords of Celje and also Vrbovec castle in 1360 (actually just half of the castle) which was ratified by patriarch of Aquilea the same year. In 1362 they pawned the other half of the castle to monastery of Gornji Grad. After a few years, more precisely in 1369, Counts of Celje received Vrbovec castle formally as a fief and kept it until they died out. After that the estate went into property of provincial prince who then gave the estate to steward. In 1458 it was leased by Tomaž Pfafoitscher and in 1472 its steward was Benko noble Lustthal. In 1478 Jurij Obračan got the estate as a fief, Tomaž Gradeneški in 1505, baron Ivan Kacijanar and his wife Elizabeta of Strmol before 1527 followed by their daughter Sholastika, wife of Ulrik baron noble Eitzingam, who sold Vrbovec to Hans Regall from Rače. In years 1566 and 1575 the castle is owned by lords Hurnuss who are followed by Boltežar Wagen. His son Hans Žiga leased the estate in 1602, baron in Šoštanj and Prebold. In 1615 Tomaž hren, bishop of Ljubljana, bought the estate for 4000 goldinars and consecrated a chapel in the castle two years later. With minor interruptions the castle remained as diocese's property until the last war and it was the administration of diocese of Ljubljana's landed property. Between the war occupators post was in the castle and thus it was burned down by partisans in 1944. After the war the castle was only partly restored and was rescued of total ruin by Gozdno gospodarstvo Nazarje (Forestry department of Nazarje), which decided on complete renovation between 1988 and 1992. Administration offices of municipality of Nazarje, many private companies, forestry institutions of the area, wedding hall, inn and museum of forestry and timber industry nowadays have offices in the castle.

Not a lot ov events are known from the history of the castle. In 1440 it was under siege by Celje's army leader Vitovec but this is not evident in sources. In 1528 a protestant priest lived on the castle with his wife and also in 1615 the castle was ruled by "heretic" lords. In 1635 at the time of steward Janez Pečoher the castle was handed over and at least party destroyed by rebelled peasants.

Outer wings of castle complex aren't preserved as a whole anymore. Only the northwestern is undamaged with barrel vaulted basement and outer wall of northeaster wing is also preserved, southeastern wing is old only in its basement part where similar vaults can be found as in western wing but its floor is completely rebuilt. In their mass and proportions are three corner towers also authentic with partly vaulted ground floors. Portals and windows are all modernized.
Castle nucleus built on rocky rick in the center of complex was built at least in first half of 12^th century but could be even older. Between 1437 and 1447 a round tower was probably added to the castle. We can presume that at the time the original castle design was surrounded by defence wall ring which was removed during reconstructions in the Renaissance era. Nowadays the only visible medieval element is surrounding walls in the nucleus. Medieval castle nucleus was toren down sometime in 18^th century (not visible in depictions from 19^th century) and between the last war the fire destroyed also a part of newer mansion, mostly the chapel and its northeastern wing. It was rebuilt on old foundations and in modern form in 1922 by plans of architect Sonja Sekavčnik.