It is alleged the holy man took the woman to a nearby beach to “pray” for her.
Neighbours say the physically and mentally disabled woman was found sitting on a street corner crying.
The neighbours alerted the woman’s family who later reported the incident to the police.
The pastor was arrested on Tuesday 16 May at his home and appeared in the Mitchells Plain Magistrates' Court last Thursday.
The victim is said to be in her 30s and is regularly seen walking around in the neighbourhood.
A 26-year-old resident, who does not want to be named, said the accused is an ex-convict who was trusted by his community.
“He is a man who helped and motivated the young people to live a righteous life, but now he turns around and does this,” she says.
“The woman said he took her to the beach to pray for her and then raped her.
“They must lock him up and not give him bail.”
The victim’s family refused to comment on the case or the incident.
Mitchells Plain police spokesperson Constable Samantha Adonis confirmed the arrest.
“This office confirms a rape case was reported at Mitchell’s Plain SAPS where a 56-year-old male was arrested and appeared in court on 18 May on the said charges.
“The victim has a slight mental disability and the investigation continues.”