The seven-time Grand Slam winner screamed in pain and stopped playing while receiving at 0-1 down, after being broken in the first game. She limped to her chair and quickly conceded defeat after seeing a trainer.
Petkovic was stunned at the retirement by Williams, her first in 251 Grand Slam matches. Williams had needed treatment to a psoas muscle, which runs from the abdomen to the groin, in her previous match.
"I am very surprised she retired," said the German. "I saw she got injured in her last game but she finished that match so I figured she was going to play tonight."
The retirement continues a miserable run for the older Williams sister in Melbourne and marks a bleak day for US tennis, as it is the first time in the Open era that no American woman has reached the fourth round of a Grand Slam.
While sister Serena is a five-time champion, the best Venus has managed is a runner-up finish to Serena in 2003. She has only reached one semi-final, back in 2001.
On a court that should suit her she has often been bundled out by players ranked well below her.
In 2009 she lost to Spaniard Carla Suarez Navarro in the second round and last year she was beaten by Li Na in the quarter-finals.
Petkovic, 23, has now reached the last 16 at a Grand Slam for the second time in succession. She was beaten at the same stage of the 2010 US Open by eventual runner up Vera Zvonareva.
She had a good start to the year when she made the final of the Brisbane International, going down to Petra Kvitova.
She will take on former champion Maria Sharapova in the fourth round following Sharapova's three-set win over Germany's Julia Goerges earlier in the day.