First of all, congrats.
Assuming you are on tier 2/skilled visa, you need to apply for a dependent visa for your new born baby if you are planning to travel in near future using the same process you used to apply for your spouse/partner. You will need to pay IHS and visa fees.
If you have no plans to travel out of UK in near future then you can delay that visa application for a later time but the only issue is that your newborn won’t be covered by NHS for any hospital level care(hopefully your baby won’t need that) after three months as it’s only free for first three months and then you may be asked to pay for any treatment. GP and health worker care is always free so it’s just hospital treatment.
We had a baby girl at end of last year and since we had no plans to travel anywhere this year plus I was due a visa extension at end of July, I didn’t apply for her visa before and only recently applied for dependent visa for her along with our visa extensions.