History of the manor is not completely clear. Some connect its existence to so called castrum minus which is mentioned from 1231 and is referred to in a document from 1300 as Wenichhaus. It is more likely that Mali grad (Small castle) was never a fortified castle building which is what both of upper names suggest. It is probably a continuity of not fortified immunity house which is in 1376 mentioned as free court of lords of Ptuj near Dominican monastery and where the designation »court« excludes any association with castle as a fortified feudal outpost.

Among former owners that are known before 1740 are counts noble Wildenstein, then until 1773 barons Moscon, followed by counts Attems'. In the second half of 19th century offices of Emperor's and kings district chief officer were in the manor. Nowadays apartments and a library are inside the manor.