Councillors have said they are concerned a hospital in Cholsey, near Wallingford, may become a prison or centre for asylum seekers.
Fairmile Hospital is due to close in March, and more than 200 patients with a range of mental disorders will move to the Prospect Hospital in Reading, Berkshire.
Wallingford town councillors want to see the hospital site developed to provide affordable homes for key workers and first time buyers.
A draft planning and development document has been produced to encourage proposals from potential developers. Luxury housing has been suggested as one option but the document also suggests the 130-year-old listed building could be designated for "institutional use".
Town council member Betty Atkins told colleagues: "Institutional use could mean a prison or centre for asylum seekers."
Pat Granados said it would be a perfect site for first-time buyer housing and key worker housing, but not for social housing.
Former mayor Theresa Jordan said: "Whatever housing is put on that site will have an impact on Wallingford. People will come into Wallingford for our shops, restaurants, pubs and other services and that is bound to have an effect here.
"We should be very careful and study all the possible implications."
An exhibition about possible uses of the site has been held in Cholsey, and comments from the exhibition will be passed to South Oxfordshire District Council.
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules hereComments are closed on this article