As early as you want. At the return request website it merely says that if they've only just received your application, the normal 10 working days to return your documents may have to be extended to 20 working days. Return of your passport (whether you or the non-EEA FM or both) will have no effect on your application.
My experience: I've requested return of travel documents for every single EEA application I've submitted (5 so far) and never had a problem. In every case I've waited until the day after they signed for receipt of the application package at Durham, to include for my PR application which they just received last Thursday. In every case I've had the passports back in at or under two weeks. Once in five days. Having said that, some people have waited much longer than I have to get theirs back, for no obvious reason. So, while your results may vary, what have you got to lose?
Caveats: I may have been lucky on the timing (although 4 times is less luck and more trend); the request I sent Friday is the first one since biometrics have come in; and the back-log of post-referendum applications could be a complicating factor this time around.
https://eforms.homeoffice.gov.uk/outrea ... ments.ofml