Question for UK Sponsors: OK to be outside UK for a while?

I have sponsored my husband for a settlement visa. I am a British Citizen but have not lived in Britain for many many years. My husband and I lived in other countries for many years together and had no trouble getting him a settlement visa when we applied for it overseas. After receiving it we traveled to the UK together. He had a job he had to begin immediately so he stayed in UK. i had to travel abroad again for an extended period of time to finish up my own job overseas and wrap up other issues. We are worried that the fact I am not actually living permanently with him yet in the UK is a violation of the terms of his settlement visa. He travels in and out of the UK frequently and last time he re-entered they asked him at immigration where his wife was and if i was in the UK. he replied no and they asked him for how long had i been outside. it seemed tense and he is worried that if he leaves UK again and re-enters without me again while I'm still abroad it could create real problems. is there any law or guidelines on whether the UK sponsor must be present all or most of the time in the UK? does anyone have advice or experience?