Jankovic was cruising at 6-1, 2-1 against Belarus qualifier Anastasiya Yakimova then stepped awkwardly when chasing a ball on the baseline and twisted her ankle. Jankovic continued to play after getting medical help, but lost that set 6-3 and eventually retired after losing the first game of the third.
It was her first tournament since Wimbledon, where she retired in the fourth round with a back injury.
In her third career WTA quarterfinals and first this year, Yakimova will next face Ksenia Pervak, who beat qualifier Alexandra Panova 6-2, 7-5 in an all-Russian match.
Pervak jumped into the draw to replace French Open champion Francesca Schiavone, who was to be second-seeded but withdrew due to a wrist injury.
Third-seeded Anastasia Pavlyuchenkova of Russia beat Rossana De Los Rios of Paraguay 6-3, 6-0 to face Johanna Larsson of Sweden, who reached her first WTA quarterfinals by defeating Maria Elena Camerin of Italy 7-5, 6-1.