The exodus began after Mr Musk expressed his agreement to an antisemitic post on X that accused Jewish people of hating white people.
Below this post, Mr Musk wrote: “You have said the actual truth.”
Following the reply, over 150 rabbis called for companies to stop advertising on the site.
Speaking to the Guardian following their decision to stop advertising on the platform, IBM said: “IBM has zero tolerance for hate speech and discrimination and we have immediately suspended all advertising on X while we investigate this entirely unacceptable situation.”
As the controversy unfolded, both X’s CEO Linda Yaccarino and the White House stepped in to comment on the unfolding situation.
Ms Yaccarino said: “X’s point of view has always been very clear that discrimination by everyone should STOP across the board.”
In a statement, White House spokesperson Andrew Bates condemned Mr Musk’s comments and the impact of promoting antisemitism.
Mr Bates said: “We condemn this abhorrent promotion of antisemitic and racist hate in the strongest terms, which runs against our core values as Americans.
“We all have a responsibility to bring people together against hate, and an obligation to speak out against anyone who attacks the dignity of their fellow Americans and compromises the safety of our communities.”
While Mr Musk is facing fallout from his reply to the antisemitic post, it isn’t the only controversy facing the troubled social media platform after watchdog Media Matters for America published a report about the site.
Reuters reported that the report found corporate adverts by the likes of Apple, IBM and Oracle next to antisemitic content on the site.
Media Matters has also said that Comcast/NBC, Paramount and Warner Brothers Discovery were pulling their adverts from X.
Following the report, Mr Musk announced the company would be filing a lawsuit against the watchdog next week.
He wrote: “The split-second court opens on Monday, X Corp will be filing a thermonuclear lawsuit against Media Matters and ALL those who colluded in this fraudulent attack on our company.”
In the past, Mr Musk has claimed Media Matters is trying to “undermine freedom of speech and mislead advertisers”.