Does an employer have to pay holiday pay to its casual workers?

Yes. Every worker (whether an employee or otherwise) is entitled to 5.6 weeks' paid annual holiday, and (as is more likely, given the short-term nature of casual workers' engagements) accrued holiday pay on termination, calculated from day one of their contract.