Eminent lawyer and Jesuit Priest Frank Brennan says it is time to look again at the idea of federal legislation to protect religious freedom in Australia.