Jankovic came back from a set down to win a tense and controversial third set tie-break and book a title match against either Ana Ivanovic or Maria Martinez-Sanchez after a 4-6, 6-3, 7-6 triumph.
In doing so the Serbian, who is aiming for her third Rome crown having won here in 2007 and 2008, became only the seventh player to beat both Williams sisters in the same tournament having thrashed Venus 6-1, 6-0 in the quarters.
"To beat two Williams sisters in two days is amazing so I'm really proud of myself," she said.
This match was an absolute thriller that swung one way and the next and kept fans on the edge of their seats throughout.
Williams had a match point on her own serve and led by a mini-break in the third set tie-break and afterwards she said she was struggling to believe that she lost.
"I almost feel like I won that match, I might turn up (on Saturday) and play because I don't feel like I lost," she joked.
"I wasn't playing the points right. I can't be really upset, but I am.
"I feel I should have won, I could have won but I can't beat myself up over it, I've just come back and hadn't played a match since January."
In the final set tie-break Williams upset Jankovic by holding her hand up and asking her opponent to wait after the Serbian had thrown the ball up to serve on a crucial point with the American leading by a mini-break.
"I was just getting ready to serve and I was serving and all of a sudden she wasn't ready.
"For me when the server comes to play the receiver has to be ready -- that's the rules.
"But unfortunately I had to hit another serve, regroup and refocus because I didn't want to waste my energy on that."
Jankovic did hold her nerve to win that point and broke back when Williams, the 2002 winner here, netted a backhand.
At 5-5 the top seed then went long with a backhand from inside the service box to hand Jankovic match-point which she secured when Williams netted a forehand.
It was a nervous opening from both players as the first four games went against the server.
Williams held to steady the flow of breaks and went on to take the set when some positive attacking play ended with a drive volley winner to break for 6-4.
The second set was going on serve until Williams played some sloppy games at the end, dropping her last two service games to love, even handing Jankovic the second set on a double fault.
The Australian Open champion broke at the start of the decider and crucially held in the second game despite facing several break points.
When she then went up 0-40 on the Jankovic serve, this war of attrition appeared to be taking a decisive turn, but the Serbian hit back to win five straight points and stay in the match.
She then broke Williams after the American failed to deal with a deep forehand to her backhand side and it was very much game back on.
Williams broke at 3-3 after a brilliant succession of shots in which she forced the seventh seed wider and wider out to the backhand before hitting a winner into the empty court the other side.
Jankovic saved a match point at 5-4 after a lucky return off the frame landed in and Williams served again for the match at 6-5 but still could not take advantage.