Jankovic, a former world number one now ranked ninth in the world, has struggled to get her 2010 season rolling but survived a roller-coaster third-round battle with Italian Sara Errani 2-6, 7-6 (8/6), 6-4.
Clijsters, who won the title in 2003 and 2005, fell 6-4, 1-6, 7-6 (7/4) to 27th-ranked Russian Alisa Kleybanova, who won her first WTA title at Kuala Lumpur last month.
After storming back to take the second set, Clijsters appeared to be on her way against 20-year-old Kleybanova when she won the first three games of the third.
But Kleybanova responded to level the set at 3-3 and it went with serve until the tiebreaker.
Once again Clijsters raced ahead, taking a 4-0 lead, but Kleybanova won the last seven points to seize the victory over the former world number one.
"I definitely think I let it slip away a little bit," Clijsters said. "What got me to be ahead was I was playing aggressive tennis and being dominant. I think she, by putting me under pressure during those long rallies, I think that, I kind of just backed off a little bit...
"This is a match where I just have to have that little bit of killer instinct back and try to just finish it off when I have my chances," added Clijsters, who won the US Open last September in just her third tournament since coming out of retirement.
She won again in Brisbane to launch her 2010 season, but slumped to a 6-0, 6-1 loss to Nadia Petrova in the third round of the Australian Open.
Clijsters said she didn't believe her lighter schedule - as she balances motherhood with a tennis career - was a factor in the defeat.
"I feel really happy with the way that I've been improving since the Australian Open. I lost. It was a good match, it was close."
Third-seeded Victoria Azarenka of Belarus was another third-round casualty, falling 7-6 (7/4), 6-2 to Spain's Maria Jose Martinez Sanchez.
Jankovic managed to avoid the upset bug, but it was close.
"I never thought I would lose the match," said Jankovic - even though she was cutting it close when she was one point away from going 5-2 down in the third set against Errani, the world number 49 who was seeking her first win against a top-10 player.
"I knew I could still win it, I was still there, and I just fought very well. That's what got me through the match."
But Jankovic said it was exhausting, especially since she was frustrated by her own inconsistency.
"I would play a couple of points well but then would give away a lot of points, too," said Jankovic. "When I was down 4-2 in the third, and 0-40 to go down 5-2, it was really a battle out there," she said.
Jankovic set up a fourth-round meeting with Israel's Shahar Peer, a 6-4, 6-7 (4/7), 6-1 winner over ninth-seeded Italian Flavia Pennetta.
Those making it safely through included eighth-seeded Australian Samantha Stosur, whose 6-3, 6-0 win over Anastasia Pavlyuchenkova set up a clash with defending champion Vera Zvonareva, a 6-2, 6-3 winner over Anastasija Sevastova of Latvia.