The teaching that those who perform the Hajj in Mecca are forgiven all their previous sins, is not found in the Qur’an. However, there are sayings of the Islamic prophet Muhammad about the pilgrimage, which have been recorded by several people. Some examples are:

“Allah’s Apostle said, “Whoever performs hajj to this House (Ka’ba) and does not approach his wife for sexual relations nor commits sins (while performing hajj), he will come out as sinless as a newly-born child. (Just delivered by his mother).” (Bukhari Vol3, Book 28, Nr 45)

“Perform Hajj and ‘Umrah consecutively; for they remove poverty and sin as the bellows removes impurity from iron.” (Sunan an-Nasa’i 2630, 2631)

According to several Hadith, the Hajj gives entrance to paradise, which means the performer has received complete forgiveness. For example:

“The reward for a Hajj mabroor* is nothing but Paradise” (Al-Bukhari 1773, Muslim, 1349, Sunan an-Nasa’i 2631).
* Mabroor means perfectly performed.

There are two pilgrimages, the Hajj and the Umrah. The Hajj is the bigger one and takes place in the month Dhu-al-Hajj. The Feast of Sacrifice coincides with this pilgrimage. The Umrah is the small one and can be performed all year round. According to some Hadith this one is enough to obtain forgiveness of sins, but not sufficient to enter paradise.

By the way, not every Muslim believes that one makes a complete new start, after perfectly performing the Hajj.
Someone asked a Muslim scholar whether he ought to fast for the Ramadans he had neglected during his youth. Based on other traditions, the scholar replied that, even though his sins were forgiven through the Hajj, he still was obliged to fast more so as to make up for those Ramadhans.