Entering the UK with eea dependant stamp

Hi,

I am an israeli citizen married to a french citizen. We have entered the uk on the beginning of August (a month and a half ago) and asked for a eea dependant stamp (Israeli citizens do not require a visa to enter and I didn't have time to get a family permit). After a while and some questions I received my stamp and we were let in. A few days ago I returned from a US work trip alone (without my wife), and the immigration officer gave me hell in the entrance even though I had all the documents required- raising his voice, accusing, taking my passport and putting me in detention for an hour or so. After which he let me in, but said that I may not be granted access again, if I don't get a visa before.

That would have been ok, unless I had a vacation planned for this weekend in spain...
My question is - Can they actually deny me access if I leave again, despite the fact that I have all necessary docs and supposedly have the right to enter and receive the stamp? Does it worth the risk?


Thanks!!

11(4).