Li Yuan had served three terms as provincial governor during the reign of Emperor Wen of Sui.

Early in the reign of Emperor Wen's son Emperor Yang, Li Yuan had served as prefecture governor (as Emperor Yang converted provinces into prefectures), but had later been recalled to serve as a junior minister within Emperor Yang's administration.

When Emperor Yang carried out his second campaign against Goguryeo in 613, Li Yuan was in charge of part of the logistics operation.

When the general Yang Xuangan rebelled near the eastern capital Luoyang, Emperor Yang had commissioned Li Yuan as a general and placed in charge of the operations west of the Tong Pass, although Yang Xuangan's rebellion eventually did not involve that region.

Li Yuan took the opportunity to recruit talented people to his staff.

Later that year, when Emperor Yang summoned him to his presence, he had declined, citing ill health—an excuse that Emperor Yang did not believe, as he questioned Li Yuan's niece, a Consort Wang (Emperor Yang's concubine), "Will he die?".

In fear, Li Yuan took up drinking and receiving bribes to try to show Emperor Yang that he did not have great ambitions.

In 615, Emperor Yang had placed him in charge of the operations against agrarian rebels in the Hedong region (roughly modern Shanxi), but recalls him in 616.

Later this year, Emperor Yang puts him in charge of the key city of Taiyuan (in modern Taiyuan, Shanxi).

Li Yuan begins to gather forces from the region, claiming that they are necessary to defend against the Turks, which draws suspicions from his deputies Wang Wei and Gao Junya.

Li Yuan, afraid that Wang and Gao will act against him first, then uses a Turkish attack as an excuse to falsely claim that Wang and Gao are working in concert with the Turkish khagan, Shibi Khan (Ashina Duojishi), and has them executed.

He sends secret messengers to Hedong to recall his sons Li Jiancheng, Li Yuanji (both by Duchess Dou) and Li Zhiyun (by his concubine Lady Wan), whom he had left there to watch over his household, and the capital Chang'an to recall his daughter (the future Princess Pingyang) and her husband Chai Shao.

Li Jiancheng and Li Yuanji, leaving Li Zhiyun at Hedong, soon meet with Chai, and they arrive together at Taiyuan.

Li Yuan's daughter, believing it would be difficult for her to flee with Chai, chooses to hide instead.

Once Li Jiancheng, Li Yuanji, and Chai arrive at Taiyuan, Li Yuan formally declares his rebellion, but maintains the guise of a Sui loyalist and declares that his intention is simply to install on the throne Emperor Yang's grandson Yang You, the Prince of Dai, who is at this time at Chang'an, and honor Emperor Yang as Taishang Huang (retired emperor).

Li Yuan first secures his northern flank by contacting Shibi Khan, offering tribute, and receives men and horses in exchange.

He puts Li Jiancheng and Li Shimin in charge of his army and, leaving Li Yuanji in charge at Taiyuan, advances south.

Meanwhile, the Sui officials at Hedong arrest Li Zhiyun and deliver him to Chang'an, where he is executed.

His daughter Pingyang sells her possessions to raise an army for him.

She persuades several other leaders to fight under her banner.

They take several towns and her army swells until she has seventy thousand troops under her command.