Residents refuse to move off railway line