Carr is on tour with his Terribly Funny show which critics claim contained the gag.
Little People UK – co-founded and headed by Star Wars actor Davis and his wife Samantha – was incensed by Carr and contacted his agents to complain.
The charity, which supports people with dwarfism, said it took action after staff received messages from people with the condition who were offended by the jibe.
In a statement, it said: “People with dwarfism simply want to live their lives just like everyone else – going to school, shopping, walking down the street and going to work, sometimes in highly respected professions.
“To potentially hear a member of the public repeat such a comment, after seeing a stand-up show, can affect them for their entire life.
“This causes unnecessary distress and anxiety through no fault of the person with dwarfism, simply because they have a genetic difference.
“Additionally, we consider the parents of people with dwarfism – with 80 per cent of people with dwarfism born to average height parents as the result of a spontaneous genetic mutation.
“Anyone can have a child with dwarfism.
“For parents, particularly new parents, to hear such jokes about their child can have long lasting effects.
“We appreciate comedy is a matter of personal opinion.
“However, in 2019, people with dwarfism should not have to live with the consequences of being the subject of someone else’s humour.”
Some Twitter users took offence, while others thought it was funny.
One said: “I’ve only ever seen @jimmycarr on TV and hadn’t realised his humour was so smutty and demeaning.
“Note to self: Don’t ever go to one of his shows. (What he said about dwarfs is disgusting. Sorry you had to hear that)”.
@samcarlisle tweeted: “Is a dwarf an abortion that made it?” I challenge Jimmy Carr to come to my home and say that to my daughter’s face.”
In 2016 watchdog Ofcom said broadcasting rules were broken after Carr made comments about dwarfs when he appeared on the One Show on BBC One.
“I tried to write the shortest joke possible,” he said.
“So, I wrote a two-word joke which was: ‘Dwarf shortage’. It’s just so I could pack more jokes into the show.”
The comedian has been contacted for comment.
