Councillors have refused to grant permission for more than 100 new homes to be built on the site of a former prisoner of war camp in Friday Bridge.