The frightened people sought refuge at a church in Cape Town on Wednesday night.
One of the men, who cannot be identified, says he was forced to flee with just the clothes on his back after he was accused of harbouring criminals.
“They said I am keeping two gangsters in the house but I am a community leader, why would I be doing that? Where do we go from here? We only have 48 hours to stay here.”
Another woman, 57, says she and her husband, their 12-year-old daughter and her four-year-old grandson were also threatened.
“Rumours spread that we are keeping gangsters,” she says.
“I have been living in Blikkiesdorp for 10 years. Even the people whose homes were burnt at the weekend are not being allowed to rebuild.”
On Saturday night eight homes were destroyed after vigilantes set alight the house of an alleged drug mert.
A 20-year-old man was also beaten to death after he was accused of drug dealing.
Vigilantes previously said they are targeting skollies who have been terrorising the community for years.
Ward councillor, Roxanne Moses, condemned the violence and told the Daily Voice they are trying to move people out of the temporary relocation area as fast as possible.
“We are ensuring that the numbers are becoming less and not more. Just yesterday three families were moved to new housing. We do have Law Enforcement in the area and police 24 hours.”
Police warned people from taking the law into their own hands and rather report crime.