A government adviser says agencies should make sure people they recruit as carers can speak English before placing them in vulnerable people's homes.
Dr Shereen Hussein, scientific adviser to the Department of Health, told BBC Radio 5 live that poor language skills could lead to bad care and abuse.
King's College London says that 20% of carers are migrant workers.
Care minister Norman Lamb said communication skills would be required for the new Care Certificate.
The brother of one dementia sufferer says his carers struggled to communicate.
Phil, whose full name we have withheld, has a 62-year-old brother with a severe form of dementia. He employed carers for 18 months to oversee the care of his brother, but grew frustrated with the quality of the staff.
He told 5 live Breakfast the situation grew so problematic he took the decision to put his brother in a care home: "The dynamic between the carers and a frightened man who was suffering with dementia didn't work most of the time.
"Almost all the carers were from overseas, mainly from Poland. I found some of them exceptionally difficult to understand, which meant my brother, in his condition, was never going to understand them and the reaction from him was to hide or get slightly violent," said Phil.
Dr Hussein, of King's College London's Social Care Workforce Research Unit, advises the Department of Health in England and says that changes to immigration policy and the relaxation of EU labour rules has altered the profile of migrants who work as carers in the UK.
She said: "Migrants from outside the European Union have a long history of working in the UK's care sector, and have always had to prove their efficiency in the English language before securing jobs in the industry, but this is not the case with new arrivals from EU countries.
"This means new migrants can be vulnerable when they're placed in people's homes - and carers have reported instances of racism and discrimination that stem from communication problems."
Dr Hussein said: "It would be really beneficial to have a standard interview process to establish English language proficiency, communication skills and softer skills of all care workers aiming to work in the sector.
"At the moment, vulnerable workers are placed in the homes of vulnerable adults with complex needs, and sometimes communication problems can result in bad treatment for both parties."
In some UK cities it is thought around half of care workers are foreign nationals.
BBC Radio 5 live Breakfast spoke to some former care workers from Hungary, who gave examples of times where they struggled to communicate with clients.
Hajnalka Deak, who came to the UK to work as a carer in 2008, said: "To do the job you have to understand the doctor's instructions, report to relatives and speak to the agencies.
"You have to help clients do things they can't do for themselves, and it's essential to understand their instructions right away.
"I remember once I worked with a wheelchair-user.
"I had to strap him in the the car, unfortunately I couldn't understand the exact instructions and we had to stay at home and couldn't go anywhere.
"I felt partly embarrassed and partly frustrated."
But Edna Maharig, who has joint Hungarian-Jordanian nationality and worked as a carer from 2011 until last week, told 5 live having a grasp on the English language is not the most important thing when it comes to being a good carer.
"There are certain terms and technical words that we don't understand and that's difficult... but it's more important to have empathy and understand your role as a carer," she said.
Dr Hussein told 5 live Breakfast the onus should be on the care agencies to prove their recruits can speak the English required for the job, and not the workers themselves: "We know that migrants form a very considerable part of social care workforce and their work is really valuable and important to the system.
"But we also know the work is reliant on very good communication skills. This standardised interview would not be a deterrent for migrant workers, but it will mean care agencies identify areas where induction and training is needed before they go to people's own homes and provide intimate care to them."
Care agencies are contracted by local authorities to supply home care to elderly and vulnerable adults.
Councils in England have been hit by more than £2bn in budget cuts for adult social care since 2011.
Colin Angel, of the UK Home Care Association, which represents the interests of care agencies, said: "Dr Hussein has identified an issue at a time when public spending on social care is extremely constrained and with the low rates that local authorities are currently paying for home care it's extremely difficult for agencies to increase the amount of paid training.
"Terms and conditions of the work-force are being pushed closer to the national minimum wage under the strain of spending cuts."
At the weekend the BBC learned the government is planning a scheme meaning new care workers will have to earn a training certificate in topics such as dementia care and patient dignity within 12 months of starting a job from March 2015, but English language proficiency is not included in this training remit, according to Dr Hussein.
Care minister Norman Lamb said: "For people to get the care they want they need to be able to communicate with their care workers. Employers are responsible for recruiting staff that can competently speak and read English but communication will be a key part of the new Care Certificate.
"This certificate will show the people who use care services that the person in front of them has been trained to a specific set of standards and knows how to act with compassion and respect."
The Labour Party has launched its own review into exploitation in the care sector, overseen by Baroness Kingsmill, which is due to be published later this month.