Tier 2 General Visa - Tax refund as a expat

Hello experts,

I am on Tier 2 general visa and 6 months left for my ILR. As I was preparing for my ILR application, I realized that I have done a blunder by listening to one of my colleagues by opting for a tax refund service for expats.

During AY 2015-16 and 2016-17, I had applied for a tax refund through a middle man and I received tax refunds. I discontinued applying for a tax refund for AY 17-18, AY18-19 thinking that this is not right and also now got to know through some of the discussions in this forum that being on Tier 2 general visa I cant apply for expat tax refund.

Could anyone please let me know whether this blunder will affect my ILR application and also can I pay back to HMRC now to resolve this issue? Please confirm.

Thanks,
Vij