We just got asked to pay back 2 years worth of child benefit

I am myself a Birtish/French citizen but according to the Transitional provisions stated in schedule 3 (part 2, subparagraph 3 and 5) of the Immigration Regulations of 2012 (European Economic Area) (Amendment) I am considered to be an EEA nationals for my wife EEA Family Member status since we were living in the UK in July 2012 with my wife having a valid residence card. It was confirmed since they sent the new residence card in September 2016.
My wife, an indian national has claimed child benefit since our first son was born in January 2012. At that time she had a residence card apposed to her passport (the card was to expire in November 2015). In August 2012 we left the Uk to come back in July 2013. She applied anew for Child Benefit and got it back. Same for our next son born in April 2014. Same for our daugther born in February 2016 (her residence card was then expired).
My wife applied for a new residence card in March 2016 and we received it in September 2016.
We did not realise until a lot later but they stopped giving Child Benefit from November 2016. In January 2017 we received a letter which was not so clear but was actually telling us that they stopped paying until we give them back proofs of her status. At that time we were actually just leaving for India for holidays so we could not give any proof as it was required for travel. I sent them a letter stating our impossibility to comply right away.
As soon as we got back in May, I realised they stopped paying us child benefit and I sent the documents including the new valid residence card. After 12 weeks of wait we were expecting to get our child benefit back from November 2016.
As it turns out they claim that she was not entitled to Child Benefit for the time she did not have a valid residence card. They are asking us to pay back 2 years of child benefit, arround £2100.
Note that in the letter they do not mention the residence card but state "because you are subject to immigration control your status as an EEA resident changed and you did not satisfy the rules and regulations"
I called them right away and they granted me a mandatory reconsideration. I stated that the residence card does not give any right, it is just an evidence to facilitate border control among other things. As such their claim is wrong and she was entitled to child benefit all along.
On the phone she kept asking me if my wife has access to public funds. I kept replying that she is allowed Child Benefit, not knowing anything about the term "public funds". I am not sure anymore about Child Benefit and since then, I looked up on the net to find where they state that EEA family member is allowed to get child benefit but I cannot find a clear answer. Since they gave Child Benefit before I guess it is fine but would like confirmation if someone can provide me with the info? Also if someone can confirm that having a residence card or not should not change anything, that would be great.
Also I saw many mention of the "no recourse to public funds" statement online and I am not sure an EEA family member has such restriction and if it applies to Child Benefit in our case. I guess she has no such restriction on Child Benefit since they granted her in the past.
Also, note that the situation is a bit stupid since I could have applied for the child benefit myself as a british citizen. The reason we did not do that was we felt she was more rightfully entitled as she is the one taking care of the children during the day. And if I remember correctly it can give some pension to my wife later.
I guess their point is that since she did not have a residence card as evidence of her status as EEA family member she was considered as subject to immigration control.
I think they are wrong an EEA family member is not obliged to have a residence card as it is just an evidence. am I right ?
let me know if I am wrong and if we should be able to not pay them back.And if I should do something in the mean time while we wait (up to 8 weeks)
Any help appreciated
Thank you,