The firebrand Italian, who replaced Martin O'Neill towards the end of last season, kept the Black Cats in the Premier League but faced criticism for his no-nonsense handling of his players and his strict training regime.
His former boss at third-tier Swindon Town famously said Di Canio managed "by hand grenade".
"The players cannot fear the manager, it's not possible," Di Canio told a press conference in Hong Kong, where his side take on Manchester City on Saturday in the final of the Barclays Asia Trophy.
"I don't kill people, I let them train."
He added: "Our team is a very good team, but not a top team. To reduce the gap between us and the top clubs it's obvious that we need a strategy and tactics and technical quality.
"But physical condition is also one of the main parts, otherwise it is difficult to reduce this gap."
And he joked: "No, they are not worried (about me)... obviously they don't love me. I told them: they don't need to love me, they have to follow me, and then they're going to be ok.
"Sometimes we have a fight and sometimes they drive me crazy, and then I'm going to get angry, but to be honest it is more like a family, with respect."
Di Canio's shock appointment at the Stadium of Light in March stirred controversy due to his far right-wing politics.
Sunderland beat Spurs 3-1 in torrential rain in Hong Kong on Wednesday to book a final date with City.
I do not kill people, 'clarifies' Sunderland coach Paolo Di Canio
Advertisement