Vincent Burroughs, of Fall Creek, Ore., says the harassing relationship began in August of 2011 when Dora Abrahamson, an agent with the Internal Revenue Service, called him and said he would be audited, CBS affiliate KVAL reports.
Burroughs says he didn't know Abrahamson, and that he hadn't met her before those calls - nor had he heard that he was being audited by the IRS.
"She was sending me texts that she wanted to come out, give me massages because she needed to help me relax," Burroughs said in a phone interview with KVAL News.
Over the next two months, Burroughs alleges that Abrahamson sent him several flirtatious text messages - offering to give massages, asking to meet him, and sending racy photos of herself to his cell phone.
"She said she knew more than my mother knew about me," said Burroughs.
In the lawsuit, Burroughs says in September 2011 Abrahamson came to his home wearing provocative attire.
"Next thing I know, she's at my gate, honking...so I opened my gate, she came into my property dressed exactly like [when] she texted me," Burroughs said.
The lawsuit states: "She said that she could impose no penalty, or a 40% penalty, and that if he would give her what she wanted, she would give him what she needed."
As a result, Burroughs says he had sex with Abrahamson in his home that night.
In the lawsuit, he says he was "sexually harassed and intimidated."
The law firm representing Burroughs filed the suit with the U.S. District Court in Eugene on Jan. 25.
Officials with the IRS were not available for comment when KVAL called them on Friday.
Burroughs is seeking restitution from the U.S. governemt for the IRS's part in employing Abrahamson and not properly enforcing their rules involving the privacy act.