Council land for around 1,000 houses is waiting to be developed - 15 years after it was set aside for housing.
And Kirklees Council says it needs to be more ambitious to bring forward plans to develop it.
The sites were allocated for development in the UDP - the 1999 planning and land policy.
A debate about housing needs took place at Kirklees Council and Kim Brear, assistant director of Place, said the council was not “punching at its weight” in strategic housing terms.
Mrs Brear said: “There is an opportunity to focus on a more ambitious and new building programmes.”
Clr Cathy Scott, Cabinet member for housing, said: “We have a housing crisis in Kirklees, we have a duty to address it.
“Nothing irritates people more than seeing houses built in their area that they can not afford to buy.
“We all live in space that was once somebody’s field.”
Clr Peter McBride said they needed to build more affordable houses, those less than £100,000 to buy or £120 a week or less to rent.
Clr Andrew Marchington, Lib Dem deputy leader, said: “Houses need to be in appropriate places and done with appropriate consultation with the public and members.”
Clr Robert Light, Conservative leader, said it was a ‘scandal’ that sites for 1,000 houses were available “and this administration is doing nothing about it”.
Clr Andrew, Cooper, Green Party leader, is unhappy that a Government scheme which allows people to buy council houses at discounted rate has depleted the council housing stock, adding: “As a council we should be standing up against the Right to Buy”.
Despite all agreeing more houses where needed – there were some disagreements.
Clr Bill Armer, Conservative, said that no-one in his Kirkburton ward “ever complains of a crisis in four or five bed luxury detached houses - but that’s all we seem to have built in the ward”.
Clr Kath Pinnock, Lib Dem, called for a list of the UDP sites adding: “Lets rally around and present these for development.”
Clr David Hall, Conservative, highlighted a debate in July – backed by Full Council – that called on the authority to do more to tackle the shortage of housing. Clr Hall asked: “Four months later, this Cabinet hasn’t done anything about it.”
Clr Charles Greaves, Independent, said more retirement living schemes were needed, saying they provided for an aging population, one which would likely leave family homes, plus they were exempt from the Right to Buy.
Clr Molly Walton, Labour, slammed some of her fellow councillors on planning committees for “making stupid decisions” asking: “I can’t understand why houses have become a negative concept. Fancy somebody wanting to have a house to live in!”
She said some councillors “played to the gallery... but it’s costing Kirklees hundreds of thousands”.
Clr Ken Sims, Conservative, responded to Clr Walton saying: “I have a duty of care when I sit on the planning committee”.
“The UDP was 20 years ago, things have altered, highways have altered.”
Labour and Council leader Clr David Sheard put politics to one side as he rounded on the national Labour Party for not blocking or abolishing the Right to Buy – which he has previously dubbed the ‘right to steal’.
Kirklees has seen recent successes - 1,789 new houses, 513 of them affordable; 466 new council houses; 35 wheelchair accessible flats; 140 extra care homes – ten of them suitable for people with dementia.
But there remains a massive challenge – Kirklees says demand is outstripping supply and affordable housing is needed as wages fail to keep up with house prices.
And it’s making it difficult first time buyers and middle income households struggling with housing costs.
A relaunched Local Plan will soon emerge but it will be late 2016 when it’s submitted to the Secretary of State to be examined and 2017 when it’s adopted.
It will set out how many new homes and jobs are needed and allocate land to meet those needs. It will also state which land needs protection from development.
Ideas put forward to increase house numbers include working with the private building sector to get sites already with planning permission for housing started and Kirklees says small builders will be play a big role in making that happen.
A target to provide 1,200 new homes – 400 of them affordable – has been backed, council land suitable for housing will be put forward, plus they’ll tackle owners of empty homes.
Weekly private rental in Kirklees is £122,77 - below the England average of £174.69.
Council housing rent is £69,64 - £80,96 for social landlord rental averages.
The council house waiting list for housing is 7,932 - with around 634 new applications each month.
The average age of a first time buyer is 29 - but only if they have parental help with a deposit. The average age for a first time buyer without financial help from parents is 36.
A study has found that by 2040 half of 20-34-year-olds will still be living with their parents.
The average house price in Kirklees is £149,974 - which needs an annual gross income of £38,564 and a 10% deposit. But 62% of households do not earn enough to meet market prices, either buying or renting.
Kirklees needs 1,457 affordable homes every year - just 513 were build between 2012 and 2014.
All figures from Kirklees Council