Residents fear city will take their homes