A passport does not of itself entitle the passport holder entry into another country, nor to consular protection while abroad or any other privileges. It does, however, normally entitle the passport holder to return to the country that issued the passport. Rights to consular protection arise from international agreements, and the right to return arises from the laws of the issuing country. A passport does not represent the right or the place of residence of the passport holder in the country that issued the passport.

One of the earliest known reference to what served the major role of a passport is found in the Hebrew Bible. In Nehemiah 2:7-9, attributed to the time of the Persian Empire in about 450 BC, it is said that Nehemiah, an official serving King Artaxerxes I of Persia, asked leave to travel to Judea, and the king granted leave and gave him a letter "to the governors beyond the river" requesting safe passage for him as he travelled through their lands.

In the medieval Islamic Caliphate, a form of passport was used in the form of a bara'a, a receipt for taxes paid. Only citizens who paid their zakah (for Muslims) or jizya (for Dhimmis) taxes were permitted to travel to different regions of the Caliphate, thus the bara'a receipt was a "traveller's basic passport."

It is considered unlikely that the term "passport" is derived from sea ports, but rather from a medieval document required to pass through the gate ("porte") of a city wall.[citation needed] In medieval Europe, such documents were issued to travellers by local authorities, and generally contained a list of towns and cities into which a document holder was permitted to pass. On the whole, documents were not required for travel to sea ports, which were considered open trading points, but documents were required to travel inland from sea ports.

King Henry V of England is credited with having invented what some consider the first true passport, notwithstanding the earlier examples cited, as a means of helping his subjects prove who they were in foreign lands.

The rapid expansion of rail travel in Europe from the mid-nineteenth century led to a breakdown of the European passport system of the early part of the nineteenth century. The speed of trains, as well as the numbers of passengers that crossed many borders, made enforcement of passport laws difficult. The general reaction was the relaxation of passport requirements. In the later part of the nineteenth century and up to World War I, passports were not required, on the whole, for travel within Europe, and crossing a border was straightforward. Consequently, comparatively few people had passports. The Ottoman Empire and the Russian Empire maintained passport requirements for international travel, in addition to an internal-passport system to control travel within their borders.

Early passports included a description of the passport holder. Photographs began to be attached to passports in the early decades of the twentieth century, when photography became widespread.

During World War I, European governments introduced border passport requirements for security reasons (to keep out spies) and to control the emigration of citizens with useful skills, retaining potential manpower. These controls remained in place after the war, and became standard procedure, though not without controversy. British tourists of the 1920s complained, especially about attached photographs and physical descriptions, which they considered led to a "nasty dehumanisation".

In 1920, the League of Nations held a conference on passports and through tickets. Passport guidelines and a general booklet design resulted from the conference, which was followed up by conferences in 1926 and 1927.[citation needed]

The United Nations held a travel conference in 1963, but passport guidelines did not result from it. Passport standardisation came about in 1980, under the auspices of the International Civil Aviation Organisation (ICAO).