In a potentially troubling sign for the First Amendment, President Barack Obama told a town hall meeting in Cleveland Wednesday that it would be “transformative” if we had “mandatory voting” to ensure that “everybody voted.” Apparently, he wants to follow the example of a very small number of other countries and implement a reverse poll tax that would fine you if you failed to vote.

The 24th Amendment to the Constitution was ratified in 1964 to prohibit making voting contingent on paying a “poll tax or other tax.” Yet Obama is now proposing what would in essence be a poll tax on American citizens for not voting.

Make no mistake about it:  the right to speak freely, particularly about political matters, which is guaranteed by the First Amendment, includes the right not to be forced to speak by the government. And when we vote, we are speaking with the choices we make in the ballot box. When we do not vote, we are again making a choice, but this time it is a choice not to speak.

Anyone who doubts that the government lacks the power to compel speech—or to compel voting—should read Supreme Court Justice Robert Jackson’s famous and eloquent opinion in West Virginia State Board of Education v. Barnette (1943), in which the Court held that forcing school children to recite the Pledge of Allegiance violated the First and Fourteenth Amendments.

Jackson, who was also the chief prosecutor at the Nuremberg trials after World War II, said that “we set up government by consent of the governed, and the Bill of Rights denies those in power any legal opportunity to coerce that consent.” The federal government has no power to force Americans into the ballot booth and “coerce” them to cast a ballot. In fact, such a scheme would threaten one of American’s most cherished liberties: The freedom to be left alone by the government.

The right to speak freely, particularly about political matters, includes the right not to be forced to speak by the government.

The president claims that young, lower-income and minority Americans don’t vote because “some folks try to keep them away from the polls.” Really? The Justice Department hasn’t filed a single lawsuit since 2009 under Section 11(b) of the Voting Rights Act, which prohibits “folks” from intimidating, threatening or coercing “any person for voting or attempting to vote.”

And the lawsuit filed in 2009 against the New Black Panther Party for intimidating voters and poll watchers in Philadelphia in 2008 “to keep them away from the polls”—in that case, the “them” being white voters—was dismissed without explanation by the Holder Justice Department. Moreover, the president’s erroneous claim notwithstanding, in the 2012 election, black Americans turned out to vote at a rate that was two percentage points higher than white Americans, according to the U.S. Census Bureau.

Census Bureau surveys also show that the president is wrong in his assessment of why people don’t vote. As I pointed out before, Census surveys of nonvoters show that the vast majority of them don’t vote because they don’t like “the candidates or the campaign issues or [are] simply not interested” in the election process. The choice not to vote sends a message to candidates and political parties about their relevance or irrelevance to the lives of those nonvoters. As my coauthor John Fund says, “Do we want to deny such persons their freedom by forcing them to make a choice they view as either unpalatable or meaningless?”

There is no doubt that candidates, the political parties, advocacy groups and the media should do everything they can to encourage people to vote. But coercing Americans who, for whatever reason, aren’t interested in voting, into the ballot booth is not only beyond the constitutional power of the government, it is bad public policy. Such a provision could be enforced only by imposing fines or jail time on nonvoters, an untenable and discriminatory practice just as bad as the untenable and discriminatory poll taxes that were outlawed by the 24th Amendment.

The fact that the president believes such a reverse poll tax would “completely change the political map in this country” is no justification for violating the Bill of Rights by coercing certain behavior and speech from American citizens. As Justice Jackson concluded, such coercion “transcends constitutional limitations on [the government’s] power, and invades the sphere of intellect and spirit which it is the purpose of the First Amendment to our Constitution to reserve from all official control.”

Barack Obama may have been a constitutional law instructor at one point, but on mandatory voting, as John Fund says, “he gets a failing grade when it comes to understanding the notion of liberty.”