Former Liberal staffer Brittany Higgins says she was raped by a colleague in Senator Reynolds' ministerial office in 2019.

The prime minister said Senator Reynolds was frustrated over suggestions she did not give enough support to Ms Higgins at the time of the incident.

He said the remark was made in a private office and she later apologised to staff for the outburst.

"She deeply regrets it. They were offensive remarks. She should never have made them. I don't condone them," the prime minister told reporters in NSW on Thursday.

Ms Higgins says the comments were "incredibly hurtful", and has threatened defamation action unless the defence minister apologises.

"I appreciate that it has been a stressful time but that sort of behaviour and language is never excusable," she said in a statement.

"It's just further evidence of the toxic workplace culture that occurs behind closed doors in Parliament House."

Senator Reynolds, who is on medical leave following heavy criticism of her handling of the allegation, is under increasing pressure to resign.

Ms Higgins didn't proceed with the police complaint in 2019 out of fear she would lose her job as a media adviser.

Independent senator Jacqui Lambie is calling for the minister to quit over the remark.

"She's gone this morning. She needs to resign immediately," Senator Lambie told Sky News.

"If she's not coming out denying that, if she's trying to smooth it over and throw it somewhere else then I think she's finished. Her career's done."

Senior Labor frontbencher Penny Wong said Senator Reynolds needed to explain herself.

"It's obviously not an acceptable comment and I think the minister needs to respond publicly," she told ABC radio.

Independent MP Zali Steggall said she was shocked to hear Senator Reynolds used the derogatory term towards Ms Higgins.

"That's incredibly disappointing and inappropriate of Linda Reynolds and the minister should know better," she told the Nine Network.

"It makes her position all the more untenable."

Senator Reynolds issued a statement saying she had never questioned Ms Higgins' account.

"I did, however, comment on news reports regarding surrounding circumstances that I felt had been misrepresented," the minister said.

Ms Higgins has recently reinstated her official complaint with the Australian Federal Police.

Three other women have since alleged they were assaulted by the same man.

He was sacked as a ministerial adviser because of a security breach on the night of Ms Higgins' alleged rape.

Australian Associated Press