Ferenc Baróczky decided in 1761, that his seminary in Eger will be developed into a similar university as the University of Nagyszombat established by Péter Pázmány which is a three faculty (theology, liberal arts, law) university. The blueprints of the building were made by Josef Ignaz Gerl (1734–1798).

His Successor Károly Eszterházy, added a medical faculty to this concept, the building was supposed to be the first 4 faculty university in Hungary. To found the university the permission of Maria Theresa was required, though she rejected it because of the negative responses from his predecessor Ferenc Baróczky. In hope of success Eszterházy continued the building according to the original plans, and in 1769 the fourth faculty was established and it was the first medical studies institution in Hungary. Even though Maria Theresa made the University of Nagyszombat into the base of medical studies (the city didn't even have a hospital to begin with, and in Eger the hospital was directed by Ferenc Markhot), the Scola Medicianis of Eger's right to doctoral degree was revoked, which caused the closure of the institution in 1775.

In 1774 teaching resumed in the new building. However, in 1777 Ratio Educationis put the establishment in a difficult situation: Maria Theresa moved the University of Nagyszombat to Buda, and declared that there can be only one university in Hungary. From this point on the institution was called a lyceum.

Philosophy and law education was discontinued in 1784 and seminary was moved to Pest in 1786. However, Eszterhazy never gave up on his plans. After the death of Joseph II, in 1790 philosophy was reinstated, along with law and seminary. In 1828 archbishop Johann Ladislaus Pyrker established the first Hungarian teacher education. Later, in 1852 it was moved to the Lyceum and continued until 1948. From 1921 on the initiative of Archbishop Louis Szmrecsányi Roman Catholic Boys Higher Commercial School also functioned in the building.

The college of education has moved to the Lyceum in 1949, when the institution, founded by the Hungarian Parliament in 1948, has moved from Debrecen to Eger. The college's philosophy was based on the academic tradition and ideals of renowned founder professors who valued spiritual guidance, openness to the future, and created a high-quality establishment.

The teacher training college became a national college-level teacher training institutions in half a century. Teacher training programs took on almost every versions and also launched some of the first multi-disciplinary program for the national teacher training. During its operation more than thirty thousand college-educated professionals were trained for the Hungarian education and society. Students enrol from all counties in the country and also from abroad.