I think ordinarily they shouldn't refuse you. here are their own guidelines, and i have posted it in full...make your own interpretation:
EUN2.14 Can family members of British citizens qualify for an EEA family permit? ('Surinder Singh' cases)
A British national and his/ her non-EEA national family members can only benefit from free movement rights if they meet the criteria established in the ECJ case of Surinder Singh. The case stated that nationals of a Member State who are exercising an economic Treaty right (i.e. as a worker or self-employed person) in another Member State will, on return to their home state, be entitled to bring their non-EEA family members to join them under EC law.
Example: A British national is exercising an economic Treaty right in Germany and living with his non-EEA national spouse and children. On the British national's return to the UK, his non-EEA national family members can apply for an EEA family permit to join him under EC law.
The Surinder Singh judgement is incorporated into the EEA Regulations in Regulation 9. Family members of British nationals who meet the requirements of Regulation 9 are treated as family members of EEA nationals for the purposes of the EEA Regulations.
Applications for EEA family permits must meet the following criteria:
• The British citizen is residing in an EEA Member State as a worker or self-employed person or was doing so before returning to the UK.
• If the family member of the British citizen is their spouse or civil partner, they are living together in the EEA country or they entered into the marriage or civil partnership and were living together in that EEA country before returning to the UK.
Because EEA nationals have an initial three months right of residence in the UK, there is no requirement for the British national to be a qualified person on arrival. Therefore, an EEA family permit can be issued to the non-EEA national family member of a British national even if they are only visiting the UK with the British national before returning to the Member State where they are resident.
It does not matter if the only reason the British national went to another Member State was to exercise an economic Treaty right was so that he/ she could come back to the UK with his/ her family members under EC law.
They go on to give guidelines on refusals, which is where Rebel82 was affected negatively:
If, after consulting this guidance, the ECO is still unsure as to the appropriate course of action to take when assessing an EEA family permit application, the ECO should contact ECO Support for advice.
Applications should be referred to the Home Office (European Casework Group) in the following circumstances:
• Where a refusal is considered on the grounds of public policy, public security or public health in accordance with Regulation 21. The provision for refusal of an EEA family permit on public grounds is distinct from the provision in the Immigration Rules for refusal on non-conducive, criminal conviction and medical grounds. The level of evidence required for refusals in these cases is high. A person's previous criminal convictions alone do not justify a refusal.
• Where an applicant is the subject of a deportation or exclusion order. The applicant must apply first for the order to be revoked. If successful, upon application for the family permit, the case must be referred to European Casework Group.
