"Rape and gang rape continue to be perpetrated by armed elements in Darfur, some of whom are members of law enforcement agencies and the armed forces, and the government appears either unable or unwilling to hold them accountable," the report said.
"In the vast majority of cases where perpetrators have been identified, they were either members of the government armed forces, law enforcement agencies or pro-government militia," it added.
An international inquiry last year concluded that rape by all parties involved in the conflict in Darfur was "widespread and systematic".
Most of the victims are among the 1.88 million internally displaced in Darfur -- nearly one third of the population in the region -- many of whom are in camps, according to the report by the High Commissioner for Human Rights.
The report was based on a detailed follow-up of some reported cases.
Between August 2004 and June 2005, UN human rights officers in West, South and North Dargur managed to follow up 86 incidents of rape or attempted rape, involving 230 victims. They were aged from just five to 60 years old.
Many women were raped or gang raped when they went to collect firewood or grass, by groups of armed men arriving on camels or horses.
"Collective rapes of a number of women and girls together were also common," the report noted.
Victims were insulted and humiliated, threatened with death, beaten and sometimes killed.
Those who survived normally did not approach authorities, either for fear of reprisals or because they felt complaints would be futile because nothing would be done.
Some police stations refused to register or investigate complaints of sexual violence, according to the report.
Although there were signs the Sudanese government was "starting to address the issue", most of the rapists and attackers have not been brought to justice, the High Commissioner for Human Rights added.
Arbour called on the Sudanese government to "fully acknowledge the scope of the problem and take concrete action" to end impunity in Darfur.
"Only timely and credible investigations and prosecutions of sexual violence will make it clear to the perpetrators of sexual violence, who include members of the law enforcement, security forces and pro-government militia, that sexual violence will no longer be tolerated," the report said.
In some instances, action by Sudanese authorities "aggravated the situation and revictimized the rape survivor through humiliation and insensitive treatment," it added.
In one harrowing account, a rape victim was forcibly taken from a clinic to be repeatedly examined against her will by government doctors.