The first known lord is Etienne who paid tribute in 1293 to Jean de Chalon, Count of Auxerre.
The Toulongeon family is one of the great noble families of the county of Burgundy. It includes in particular :
- André de Toulongeon (c. 1390-1432), great squire of France, adviser and ambassador to the Duke of Burgundy;
- Jean II de Toulongeon (1381-1427), his brother, Marshal of Burgundy;
- Antoine de Toulongeon (c. 1385-1432), their brother, Marshal of Burgundy, ambassador and man of war, Knight of the Golden Fleece;
- François-Emmanuel d'Emskerque de Toulongeon (1748-1812), deputy to the States General of 1789, politician and man of letters;
- Lionel de Toulongeon (1820-1868), French politician and former deputy of the Jura.
The de Toulongeon family had a chapel built on the south side of the church of Germagnat, where their coat of arms can be seen sculpted on the ribs of the vault.
The vault on the side, with its arch decorated with phylacteries bearing the motto "A tout", may have received the body of John II in 1427.
The castle and the fiefdom left the family in 1588 to the lord of Lomont and Rosy (Gaspard de Meyriat) for the price of 3333 ECU 1/3. At that time, the castle was already in ruins.