The International Olympic Committee (IOC) said on Tuesday that comments about women made by Tokyo 2020 Olympic Games chief Yoshiro Mori were "absolutely inappropriate".
Mori has come under fire for saying last week that meetings with female participants take a long time, adding that they "get competitive" with each other.
He later apologised and retracted his comments while also attempting to justify them, inviting further criticism.
"The recent comments of Tokyo 2020 President Mori were absolutely inappropriate and in contradiction to the IOC's commitments and the reforms of its Olympic Agenda 2020," the Olympic body said in a statement.
The Tokyo Games organising committee (TOCOG) will convene a special board meeting as early as Friday as a result of the outcry, broadcaster NHK reported on Tuesday.
Following Mori's remarks, about 440 Games volunteers have quit and local organisers have received more than 5,500 complaints, according to local media.
The Tokyo Olympics was postponed last year due to the COVID-19 pandemic.
The IOC, working with local organisers to host the Games in July, did not mention any further action against Mori but said TOCOG also considered its own chief's comments as inappropriate.
"Besides Mr Mori's apology, the Tokyo 2020 Organising Committee also considers his comment to be inappropriate and has reaffirmed its commitment to gender equality," the IOC said.
"As the leader of the Olympic Movement, we are committed to our mission to encourage and support the promotion of women in sport at all levels and in all structures, as stated in the Olympic Charter."
Sponsors have also distanced themselves. Nippon Life Insurance Company said it was "disappointed" with the remarks and had made that clear to the organising committee.
"That was definitely a gaffe, but Mori is old ... and I think that people of that age have a tendency to look down on women, it's a factor of their age," said Misako Yoshizawa, 70, who teaches English part-time in Saitama prefecture and whose plans to volunteer haven't changed despite the outcry.
"Mori is Mori, I'm not his volunteer. I'm a volunteer working to carry out the Tokyo Olympics," she said.
Still, an online petition seeking action against Mori has attracted 140,000 signatures so far, and an editorial on Tuesday in the daily Mainichi said he should resign.