Architect Sir Jeremy Dixon is to meet project managers of the Oxford Prison site development after his building design was described as "appalling".
Sir Jeremy, who also designed the new Said Business School near Oxford railway station, has been asked to meet the developers, Osborne Properties, and city council planning officers on June 7.
City council planning committee rejected outright the designs for a key building on the site off New Road, housing a restaurant, hotel rooms and an underground car park.
The rest of the £18m scheme to turn A-wing of the former Victorian jail into one of Britain's most unusual hotels was approved.
City councillors were unanimous in their condemnation of the two-storey New Road building design because it would overshadow the ancient Mound, one of the few remaining parts of Oxford Castle, which dates from 1071.
They are calling for a "fundamental rethink".
Councillors want a heritage centre based on St George's Tower, another prominent feature of the castle, to be given equal significance on the site and made fully accessible to the public, as well guests at the hotel. Planning committee chairman John Goddard said: "We are all of one mind on this. It is the building that has attracted most attention during public consultation."
He said that, as the design stood, people would regard the development simply as a hotel.
He said: "The Jeremy Dixon building is essential to how the whole site appears. I am upset about this building.
"I think the Mound will remind people that there is a wonderful history within there and they would be tempted to go in."
Green city councillor Dr Mike Woodin described the design as "appalling".
Former Lord Mayor Maureen Christian attacked the design as giving too much importance to commercial activity at the cost of the historic significance of the area.
Osborne Properties' project manager Robert Carter said the New Road building was vital to the commercial viability of the entire scheme.
He said: "If that building was completely rejected and the city council decided to have no building there at all, that would be the end of it."
Mr Carter said: "We have set up a meeting with Sir Jeremy Dixon, English Heritage and the city council. We are quite happy to look at it again."
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules hereComments are closed on this article