The long term relative motion of continents has been charted using several different approaches. Continental fit was used to invoke a mobile Earth ("continental drift") by many people long before they formalised the plate tectonics hypothesis. However, Wegener added extra information - matching geology between the continents. The offset of geological markers of known age can be compared with the present position of the same features to determine long-time averaged displacement rates.
The most striking information comes from palaeomagnetism. Rocks can record the orientation of the earth's magnetic field at various times in their history (e.g. when molten rocks crystallise and cool). This site gives some insight on using palaeomagnetic orientations to determine rotations and changes in palaeolatitude with time. These may be used to reconstruct the position of the magnetic poles relative to a particular site. Perhaps confusingly, pioneers of this approach in the mid 20th century reported their results in terms of Apparent Polar Wander Paths (APWs), in which the relative movement between continent and pole was reported as though the continent was fixed! The terminology has remained so people talk of APWs of individual continents. By comparing APWs for different places there relative movement (using magnetic poles that are fixed) can be charted.