If you’ve been travelling alone to the UK with the EUFAM Stamp and you’re from a visa requiring country originally then you’ve been very lucky indeed. Firstly being able to cross into the UK (unless of course you’re in possession of a valid appropriate visa) and secondly not asked at the Irish border for your passport.
See the Romania example (different countries similar scenario) which would apply to your case (Romania not part of Schengen Area currently)
I am a non-EU national living in Romania with my Latvian husband. What documents do I need to travel to Portugal?
If you hold a residence card issued in Romania under EU law and you plan to travel to Portugal with or to join your husband, or any other EU family member, you can enter Portugal simply on presentation of your passport and your residence card.
However, if you travel to Portugal on your own and do not plan to join EU family members there, you will require a visa even if you hold a valid residence card
https://www.google.ie/amp/s/europa.eu/y ... amp_en.htm
To my understanding EUFAM Stamp isn’t an EU ID National Card hence one should always have the passport with them. So the officer [would] arguably in this instance be crossed and annoyed with you for not producing the passport when asked.
As for the stamp, If we were so lucky as to have all border officers extremely well informed on the regulations of Freedom of Movement then stamping a passport isn’t required.
All the best
