It depends whether the institution which conferred your degree is accredited by UK NARIC, as not all colleges are regarded as of sufficient academic standard that their degrees are worth very much. Certainly in Thailand, where I hang out, only one or two universities are recognised.
You are justified in not accepting the judgement of VFS, who are merely a contracting company employed by UKBA to take in visa applications and do the biometrics etc before passing them on to the Entry Clearance Office where the application is considered. They have frequently been known to have given wrong information to applicants. You should try and contact the ECO direct by phone or email and ask them whether your degree is recognised by NARIC. Alternatively, you could apply to NARIC, but as you see from the link below, this involves cost and delay, and you might find it quicker and nearly as cheap to take the test, which judging from your post you would pass without difficulty.
http://www.ecctis.co.uk/NARIC/individuals/