The chairman of the House Judiciary Committee says Congress should “go to court very quickly” to stop President Obama if he decides to act alone on immigration by deferring deportations and offering work permits to millions of people in the U.S. illegally.

“If the president were to take further action, I believe it would be very important for Congress to undertake a challenge to that,” said Rep. Bob Goodlatte, R-Va., chairman of the House Judiciary Committee, at a Heritage Foundation event about Obama’s use of executive action.

“I would hope we would go to court very quickly and seek an injunction restraining the administration from granting those kind of work authorizations that I don’t think the law in any way provides for,” Goodlatte said.

Obama said last month that he plans to put off executive action on immigration until after the November midterm elections, citing the surge of unaccompanied children who flooded the U.S. border with Mexico this summer—and political concerns—as the reasons for the delay.

President Obama said he will announce executive action on immigration after the November midterm elections.

Even though the House has rebuffed attempts to revive comprehensive immigration reform during the border crisis this summer, Goodlatte said Obama should work with Congress to make an changes to current policy.

Last year, the Senate passed a bipartisan overhaul of immigration law that included a path to legal status and citizenship for many illegal immigrants.

Goodlatte argues that because Obama announced he would act alone, it discouraged Democrats from undertaking serious negotiations in Congress.

“Most people feel immigration reform is needed, but there’s disagreement on what it should be,” Goodlatte said.

But when you try to bring legislative bodies together and you have to work out the differences and in the middle of that the president says, ‘Here is my list of things … and if you don’t do it I will,’ those who agree with the president’s policies can sit back and say, ‘Well I don’t need to enter into tough negotiations about what needs to be done to enforce the law or reform the law and instead, I will just wait for the president to act.’

Besides effectively expanding his Deferred Action for Childhood Arrivals (DACA) program, which delays deportation for certain immigrants who came to the United States illegally as children, Obama is also expected to pursue changes that would produce more legal visas for people wanting to work in the United States.

No matter the form of action, Goodlatte vowed to fight it.

“The president does not have the authority to act,” Goodlatte said. “We should challenge.”