The name suggests some older fortification. First indirect reference to the castle is from 1326 when some seller of the smaller estate is mentioned Herbort von Graecz, ministerial of Spanheim and again later in 1344 and this time as a witness. Castle was supposedly built shortly before the year 1300 by a knight Majnhard of Turjak. First explicit mentioning is from 1393 when the castle is refered to as Grecz, schloß and according to other sources even earlier in 1338 as veste Gretz. Later, in 1466 it is mentioned as castrum Gretz, in 1476 as schloss Grecz and again in 1488 as geschloss Graecz. In 1475 the castle and the estate were both bought from last representative of lords of Gradac by main branch of lords of Turjak. After them other owners are encountered: in 1601 Jurij Lell followed by Janez Ludvik count Thurn-Valsassina and after him Oton Henrik Valsassina, Janez Baltazar Purgstall in 1629 and family Gusič in mid 17^th century. First among them is Andrej noble Gusič, who was acording to Majda Smole, probably only a tenant of the estate because no real estate is mentioned in his probate inventory. But in it we can find Derimatschka, some Turk imprisoned in 1650 in Gradac.
However the ownership of Gradac was at the time the dominion was at least after 1661 owned by Gusič family who were also awners between 1763 and 1787. At the time the castle is in accompanying text to Austrian military Ordnance Survey map, marked on section 237, mentioned as noble residence with a chapel - Der Edelhof nebst einer Capelle.
Gusič family was the owner of the dominion until 1846 when it was bought by founder of Gradac iron foundry Franc knight Friedau. He was followed by some other owners and the last among them was Šutej, merchant from Zagreb in 1939. During WWII the castle was a place for VDV officer's school and even transitional hospital during the last months. Inscription plate located left to the main entrance into the castle is the reminder of those times.
After the war nationalized building became a place for school for difficult adolescents and after a place for farming and house keeping school and since 1956 it is a place for apartments. Nowadays the castle is deserted and some elements can be found in rooms (mostly calendars and posters, also furniture) from 1980s. It looks like at the time residents moved out and nobody has yet cleaned the rooms.

In 1999 Gradac castle was proclaimed as a cultural monument of national importance. In the future it is planned to make it a tourist attraction.

In the book listed in the sources a desolated but once imposing park is mentioned. Unfortunately at the time I do not yet have photos of that part of dominion.