The US Embassy in the UK's website
specifically excludes even British non-citizen nationals (BOTCs, BOCs, BNOs, British subjects and BPPs) with British passports from applying for ESTA. So a Refugee Travel Document issued by the UK will also likely not count.
As an aside, the document issued to refugees for the purpose of international travel are not passports, but travel documents
. A passport is a form of travel document, but not all travel documents are passports (i.e. passports are a subset of travel documents). Other forms of travel documents that are not passports are UN issued laissez-passers
, for instance.
The US ESTA seems to specifically require passports and I am not sure whether that includes other travel documents or not.
Even after one becomes a British citizen and acquire a British passport, their past travel history and your citizenships may also be relevant. Since 2016, those who have previously traveled to Iran, Iraq, Libya, North Korea, Somalia, Sudan, Syria or Yemen on or after March 1, 2011, or who are dual nationals of Iran, Iraq, North Korea, Sudan or Syria, are not eligible to travel under the VWP.
I am not a lawyer or immigration advisor. My statements/comments do not constitute legal advice. E&OE. Please do not PM me for advice.