Yes, Mathematics Students can apply for the UGC NET exam in Population Studies subject. To apply for the exam, you need to fulfill all the eligibility criteria of the exam.
You should have minimum 55% (For General) aggregate marks in master’s to apply for UGC NET exam or if you belong to OBC/SC/ST/PwD category then you will get the 5% relaxation in aggregate marks.

There is no restriction on the number of attempts.

But to apply for NET JRF, your age must not exceed to 30 years, on the other side for Assistant Professor, no age criteria is applicable. You can apply whenever you want.