The state minister of humanitarian affairs, Mohamed Yousif Abdalla, publicly assured them of their safety.
"Against that promise a Sudanese translator has been arrested," U.N. envoy Jan Pronk told reporters in Khartoum. "And I am asking them (the government) to keep their word in particular if that word is a public word to the secretary-general of the United Nations," he added.
Annan had entered a reed hut to talk with rape victims, one of whom was pre-pubescent, aid workers said. The translator accompanied him and was later arrested.
The issue of rape is sensitive in Muslim Darfur and the government denies allegations by rights groups, aid agencies and a U.N.-appointed commission of inquiry that there is widespread rape in Darfur.
Two aid workers from the Dutch Medecins Sans Frontieres (MSF) have been arrested for publishing a report based on medical evidence in the agency's hospitals in Darfur documenting about 500 rape cases over 4 1/2 months in the troubled region.
Pronk also accused the Sudanese media of conducting a "smear campaign" against non-governmental organisations (NGOs) and doctors and nurses.
The pro-government Sudan Vision daily published on Monday a full-page report accusing aid agency Medecins du Monde of falsely issuing a report on rape and called for the immediate deportation of the aid workers involved and the organisation to be expelled.
The English-language Sudan Vision article published the names of the rape victim and the two MDM workers involved in the report. The alleged victim was 17 years old.
Pronk said he deplored the Sudanese media for not believing the victims of rape.
"I consider (these) statements in the press ... as a smear campaign against nurses, doctors and NGOs who are helping victims," Pronk said.
He also criticised the media for not writing about the issue of rape despite his lengthy statements on the subject.