The Vatican is a de facto successor to the former Papal States (752 - 1870).
The reasons of existence for a secular Papal authority reach back to the 6th century. At the start of said century, the Italian peninsula was administrated by the Ostrogothic Kingdom. The Kings accepted the nominal authority of the Eastern Roman Emperors (Byzantines) over them, but effectively took their own decisions. Theodoric the Great (reigned 493 - 526) was a remarkably effective ruler, achieving some balance between the Roman and Gothic populations of his realm. Amalasuntha, his daughter, ruled as regent from 526 to 534. She was considered to favor the Romans over the Goths, resulting in her deposition and execution. Justinian I , the Emperor, used the deposition of his ally as pretext to invade Italy and reassert his authority over it.
The so-called Gothic War lasted from 535 to 554. The Byzantines won but actually achieved the annexation of a wasteland. The constant movement of armies had managed to depopulate the peninsula. In 568, Alboin led a coalition of Lombards, Bavarians, Gepidae, Saxons and Bulgars in an invasion of Byzantine Italy. Within a few years Italy had been divided in several areas of Byzantine and Lombard authority.
Rome was among the Byzantine-held territories. In practice the Popes led the local government , only reporting to the Exarch of Ravenna and the Emperor. A number of political and theological arguments between Popes and their overlords took place, but initially left little lasting results. This changed with the Iconoclasm dispute. From 726 onwards the Emperors banned the use of religious images ("icons"), considering it a form of idolatry. With Rome being a center for Iconolatry (veneration or worship of icons), the Popes actively opposed the imperial decisions, resulting in decades of strained relations.
In 751, the Lombards were attempting to expand their realm and managed to capture Ravenna. The Byzantines had secured their hold over Southern Italy and Sicily, but had just lost Northern Italy and were abandoning Rome to its fate. Unwilling to see his realm fall to the Lombards, Pope Zachary had to seek another patron. He found one in the person of Pepin the Short, then de facto ruler of the Franks. For over a century the Kings of the Merovingian dynasty were mere puppet rulers, authority actually held by their Majordomos (Mayors of the Palace). Peppin, the current Majordomo, was offered coronation as King in exchange for military assistance. The decision was easy.
Peppin's campaigns in the 750s allowed him to take over the former areas of the Exarchate. He granted to the Popes secular authority over Ravenna, Rimini, Pesaro, Fano, Sinigaglia and Ancona. Two decades later, Desiderius, King of the Lombards managed to capture much of the Papal States, including Rome itself. This promptly backfired when Pope Adrian I summoned his patron Charlemagne for military help. Charlemagne conquered the entire realm of the Lombards and added the title "King of the Lombards" to his titles. In 781, the Papal States were expanded to include areas of the Duchy of Benevento, Tuscany, Corsica and Lombardy.
Actually the area of the Papal States now included realms administrated by local Roman nobility and various Lombard or Frankish aristocrats. Said nobility often held de facto authority over the Popes and/or was able to have its own candidates elected to the Papal throne. For example in the 10th century, Senatrix Theodora and her daughters Theodora the younger and Marozia managed to have a series of their lovers and male descendants elected Popes. Also the Holy Roman Emperors were able to ascertain their authority over Popes between the 10th and the 13th century, although seldom without a war involved.
From 1305 to 1378, the Popes resided in Avignon, France under the political influence of the French Kings. The city would remain part of the Papal States to the French Revolution. In their absence from Italy, various subordinate nobles were able to rule as sovereign rulers. This ended in 1353-1359, with the military campaigns of Cardinal Gil Alvarez Carrilo de Albornoz. Appointed a Papal legate (representative), Alboroz managed to both re-establish the Papal States and introduce a new code of civil law which would stay in effect to the early 19th century.
The territory of the Papal States was expanded through the military campaigns and diplomatic efforts of Pope Alexander VI (Rodrigo Borgia, reigned 1492 - 1503) and Pope Julius II (Giuliano della Rovere, reigned 1503 - 1513). However the Popes were often subject to the influence of the leading rulers of Christian Europe, the Kings of France (first Valois, later Bourbons) and the Habsurg rulers of Spain and Austria. In any case the Papal States extended to Latium, Umbria, Marche, Ravenna, Ferrara, Bologna, Romagna, Benevento, Pontecorvo and the Comtat Venaissin of southern France. Meaning they controlled most of Central Italy and minor territories of Northern and Southern Italy.
From 1791 to 1798, the Papal States were in conflict with Revolutionary France. France abolished the States in 1798. They were restored in 1800 but again annexed by the French empire in 1808. They were restored again in 1814, following the defeat of Napoleon I. The Popes which followed the restoration were extremely conservative. The education system, court system and all charitable institutions were fully placed under the authority of the church. Bible societies which translated and distributed the Bible were condemned, Jews were placed under harsh legal restrictions (many choosing to migrate), Carbonari and Freemasons (liberal political societies) were persecuted. Pope Gregory XVI (Bartolomeo Alberto Cappellari, reigned 1831 - 1846) outperformed his predecessors by opposing basic technological innovations, such as gas lighting and railways.
With the increasing unpopularity of these Popes among the populace of their own States, secular revolts were becoming common. The election of Pope Pius IX (Giovanni Maria Mastai-Ferretti, reigned 1846 - 1878) was hoped to lead to more liberal policies and hopefully a more popular Papacy. He reversed most of the policies of his predecessors, spreading initial enthusiasm across Italy. However his policies alienated the conservatives both within the States and in other European states, particularly the Austrian Empire. What Pius failed to take into account was that for many Italian liberals and revolutionaries anticlericalism and secularization was a major part of their ideology. A liberal Pope was not enough. Meanwhile Italian nationalists were struggling for a unified Italy, They hoped Pius would lead the fight against the Austrian occupation of Northern Italian provinces. When he did not, he lost the chance to gain their support.
In 1848, a revolutionary wave spread in Europe. Starting from France and moving to many other European states, there was no real universal cause. People from almost every social class were protesting the status quo established since the fall of Napoleon. The Papal States were no exception. Pius attempted to stay in place by offering a Constitution. what many revolutionaries across Europe were demanding. It was not enough. In 1849, Pius was deposed and a Roman Republic declared. But the revolutionaries soon lost control of the situation and the Papal States fell to anarchy.
Pius managed to find military support from Louis-Napoleon Bonaparte, President of France (later Napoleon III, Emperor of France). The French Army crushed the Republic and restored the Pope. From 1850 to 1870, the Papal States relied on French military protection. Pius still attempted reforms but was seen as a puppet ruler. In the Second Italian War of Independence (1859), Napoleon III allied himself with Victor Emmanuel II, King of Sardinia against Franz Josef I, Emperor of Austria. The result was a victory, ending Austrian territorial control of parts of Italy. France annexed Savoy and Nice, Sardinia annexed the Duchy of Parma, Duchy of Modena, Grand Duchy of Tuscany. In 1860, Napoleon allowed Sardinia to also annex most of the Papal States. Only Latium and Rome itself remained outside Sardinian control.
By 1870, Sardinia existed no more. Having expanded to almost the entire Italian Peninsula, it had evolved to a "Kingdom of Italy". Rome remained a coveted prize for it. But they would not risk a war with France. The solution came with the Franco-Prussian War (1870 - 1871). The French Army could no longer protect Rome. It had to be recalled to France and prepare to protect Paris instead. After a month of observation, Italy moved against the Papal States. It took the Italians about ten days to capture Rome and annex what was left of the Papal States.
The war left the Popes with no secular authority and no state to rule.In protest, Pius refused to leave his residence at the Vatican under any circumstances. He would make no public appearances, not accept Italian authority. He declared himself a "Prisoner in the Vatican" and expected the moral outrage of Catholics. A rather futile hope. The population of Rome approved the annexation in a plebiscite, the Italians took pride in their new capital and Catholic European powers were not sufficiently disturbed to declare war on Italy. Nevertheless all Popes until 1929 continued the "Prisoner" stance.
From 1926 to 1929, negotiations between Benito Mussolini, Prime Minister of Italy and his counterpart Pietro Gasparri, Cardinal Secretary of State resolved the "Roman Question". The Popes would realign their claims over the former Papal States in exchange for a fully sovereign state including only the Vatican, though having to declare neutrality in International relations. Italy would accept Roman Catholicism as its state religion and actively promote it. Since then the Vatican has maintained its independence as Europe's only absolute monarchy and electoral monarchy.