This is a late reply and you've probably already found your answer but here goes.
There are a lot of misconceptions about permanent residency. The key point to note is that your PR status never expires automatically. The PR card however, which provides you the ability to re-enter Canada, does have a 5 year term. So the short answer to your question is that there is no time limitation for renewing your PR card.
In order to renew your PR card, you will need to meet your residency obligation (RO) which is 730 days of physical presence in Canada of the previous 5 years. Failing to prove that you met your RO will cause you to lose your permanent residency status.
However, as you already know, any days spent outside the country with a Canadian citizen spouse is not counted as an absence. Hence, you continue to meet your residency obligation even if your card is expired.
For all you care, you could be living outside Canada with your wife for 50 years, and you will still have met your residency obligation when you apply to renew your PR card, or when you apply for a permanent resident travel document (PRTD).