Anti-racism campaigners on Friday urged England's Football Association to take action after some fans taunted Rio and Anton Ferdinand with abusive chants during last week's World Cup qualifier in San Marino.
Rio Ferdinand was targeted after he withdrew from the England squad to play in the San Marino game because it did not fit in with his pre-planned training programme and then travelling to Qatar to commentate on England's 8-0 victory.
The Manchester United defender had been named in the squad for the first time in nearly two years after being left out following a racial abuse row between then-England captain John Terry and Anton Ferdinand.
Anti-racism group FARE reported England to FIFA on Thursday and the world governing body confirmed it was looking at the allegations, which centred on a chant suggesting the brothers should be burned on a bonfire.
The chairman of the Kick It Out anti-racism group, Herman Ouseley, told BBC radio: "Whether it's racist or not, it's certainly unacceptable. It's vile and it shouldn't be part of sport. Something needs to be done about it.
"These are the supporters of the England national team who are travelling abroad and singing songs like that. What messages does it send out about the type of people we are and who we represent?
"The Football Association should be taking a stance on this about the people it wants supporting the England team, the image it wants to send abroad."
Ouseley, a former head of the Commission for Racial Equality watchdog, said he had already contacted the Football Association and called for an investigation, arguing that the side's reputation abroad was at stake.
"FIFA will determine (whether it is racist) but clearly we can take a stance on that. We have policies on anti-racism, homophobia and all other forms of unacceptable behaviour. Why are we so quiet about it?" he added.
Rio Ferdinand admitted on his Twitter account that he was shocked to hear that England supporters may have used racist abuse to taunt him.
"You expect+accept banter from fans on the terraces as its part of what makes the game great,but racism is not banter,and from ya own fans. WOW," he tweeted.
"Always a small minority who ruin it for others.
"Let's not jump to conclusions + assume though as it might just have been banter. We'll see after the investigation."