The residents have lodged a Judicial Services Commission complaint and recusal application against Acting Judge Leslie Weinkove.
Residents are accusing Weinkove of impugning their dignity and violating their right to a fair hearing.
The Ndifuna Ukwazi Centre, which is assisting the 27 residents, said yesterday the recusal application would be argued on May 17 and 18.
The families had been living in Bromwell Street for generations when they were served with eviction notices in March 2016.
The City of Cape Town’s plan to relocate them after the houses were sold for development was met with strong resistance and accusations of apartheid-style forced removals.
The residents turned to the Western Cape High Court to stay the eviction and have rejected the City’s proposal to move them to Wolwerivier, about 29km north of Woodstock.
Weinkove made a number of “disparaging remarks” during their hearing.
Ndifuna Ukwazi said: “These included insinuations that unemployed people do not need to be close to opportunities and services.”
He also insulted spokesperson for the group, Charnell Commando, referring to her as a kitchen hand and questioned her understanding of the eviction and relocation processes.
“Where judges make comments which degrade a person because of their socio-economic and employment status this adds a more insidious barrier to access to the courts.”
Weinkove has reserved judgment.