Normally, when setting the fees and charges for users of a large national service, the cost of maintaining the presence of the infrastructure is spread across all users of that infrastructure, even if they do not use a significant part of that service.
Part of my transport fares almost certainly funds transport in remote parts of Scotland. The immigration fees paid by me likely also funds a part of the immigration appeals process, which I have never had to use. The profits that the Royal Mail gets in London (where it is cheaper to deliver letters) are used to cross-subsidise services in the Outer Hebrides, where it is more expensive.
So, your argument about the cost of individual visa applications and of EEA citizens not having to pay for a service is not entirely correct. I can appreciate that EEA citizens do not utilise a service, but their non-EEA family members do and they should be treated as being on par with the same application for the family member of a British citizen.
Another picture of the situation comes from the
Home Office Annual Report and Accounts 2014-15 (the most recent I could find). Page 114 suggests that in financial year 2014-15, UKV&I makes a nett income (monopoly profit perhaps

) of about £182 million and the HMPO of £104 million, both of which were however, completely subsumed by an expenditure of £436 million on Immigration Enforcement.
Now, to you, it may seem that the cost of immigration enforcement should not be applied to you as you have not broken any laws (presumably), but to an external analyst, the cost of the system should be borne by all users, not just ones benefiting (or not) from it.
PS: There are some really interesting stats in that report. Page 120 states that there are 6,315 people in UKV&I and 4811 staff in Immigration Enforcement. So, almost 80% as much staff in Immigration Enforcement as in the whole of UKV&I. And the HMPO get by on 3352 staff, just half of the UKV&I.
I am not a lawyer or immigration advisor. My statements/comments do not constitute legal advice. E&OE. Please do not PM me for advice.