spending time outside the UK on a skilled worker visa

Hello,

I've been reading the rules regarding the skilled worker visa, and I could not find any restrictions about the maximum time that the visa holder is allowed to stay outside of the UK.

There is a rule about not spending more than 180 days abroad on any 360-day-window in order to be able to apply for indefinite leave to remain after the fifth year. But, assuming that the visa holder does not care about applying to the indefinite leave to remain, are there any other rules or consequences of spending too much time abroad or travelling frequently in and out of the UK?