Around 13% of all prisoners in Britain are foreign nationals
More foreign prisoners should be deported to help tackle prison overcrowding, the chairman of a House of Commons committee has said.
Tory MP Edward Leigh, head of the Public Accounts Committee, said it was "astonishing" 1,019 people were freed without deportation being considered.
The figures emerged in April, and Home Secretary Charles Clarke was sacked in a Cabinet reshuffle in May.
Prison numbers in England and Wales may top 80,000, the committee warned.
It was of the "highest importance" that the Home Office kept track of all of the 10,000-plus foreign prisoners and consider deporting them during or at the end of their sentence, said Mr Leigh.
Overcrowding is undermining many key objectives of prisons, increasing tensions between staff and inmates, driving up suicide rates, providing a "fertile environment" for unrest and disrupting education and welfare programmes, he added.
The committee's report urged the National Offender Management Service (NOMS) to plan ahead for further increases in numbers.
The prison population in England and Wales rose above 77,000 last year.
The PAC said numbers could be cut by using electronic tagging to reduce the 13,000 people being held on remand before trial, as well as considering alternatives to prison for mentally-ill offenders and children.
Mr Leigh added: "Around 13% of all prisoners are foreign nationals.
"It is now of the highest importance that the Home Office knows exactly what foreign criminals it has under lock and key and what scope there is for deporting them - either during or at the end of their sentences."
'Years' to fix immigration system