EU National -UK citizenship

Hello,

I am an Polish/EU citizen, and have been living (and working continuously) in the UK for over 7 years now. I moved here in 2002 with my wife, an EEA citizen and was granted a 5-year Leave to Remain (issued in 2002). When Poland joined EU in May 2004 I assumed I did not have to register under the WRS as we had already been living in the UK before that date. I was issued with a Permanent Residence card only when dealing with Home Office with an entirely different issue in 2006. I would like to apply for Naturalization, but I am told by various sources (HO info line including) that I do not qualify till May next year, as I need 5 years of PR to be settled and then maintain the settled status for 1 year in order to qualify.
The question is: does my nearly 3 years in the UK before 1 may 2004 not count at all towards that period (as it was only a temporary Leave to Remain)?
Are there potentially any other grounds I could be applying on, do you think?

Many thanks in advance for your help.

Piotr