Are Voting Laws Not Subject to Judicial Review?

Ian Millhiser notes a radical assertion from Justices Gorsuch and Kavanaugh in the recent Wisconsin voting case:
As Gorsuch notes in his concurring opinion, which is joined by Kavanaugh, the Constitution provides that “the Times, Places and Manner of holding Elections for Senators and Representatives, shall be prescribed in each State by the Legislature thereof.” A separate constitutional provision provides that “each State shall appoint” members of the Electoral College “in such Manner as the Legislature thereof may direct.”
According to Gorsuch, the key word in these constitutional provisions is “Legislature.” He claims that the word “Legislature” must be read in a hyper-literal way. “The Constitution provides that state legislatures — not federal judges, not state judges, not state governors, not other state officials — bear primary responsibility for setting election rules,” he writes.
Old timers will remember this argument from the 2000 Florida recount debacle. The idea is that the Constitution doesn’t allow state courts to intervene in election rules. What the legislature says, goes.
I’ve never understood this. IANAL, but it’s implicit in the Constitution that all laws are passed by a legislature of one kind or another, and those laws are all subject to judicial review. So why would voting laws be any different? Just because the Constitution actually mentions the word legislature there? This doesn’t make any sense to me. Still, nobody else joined this concurrence, so presumably it’s not likely to affect things much going forward.
I should mention, by the way, that I’m less concerned about court-mandated voting rules being overturned than a lot of people. There are two reasons for this. First, some courts really do seem to be overstepping their bounds on election law this year. Kavanaugh and Gorsuch may be wrong about legislative supremacy, but modifying election law in response to COVID-19 really does seem like it’s a legislative responsibility. Mandating that ballots have to be received by Election Day, for example, is hardly a new or unusual requirement, even if many of us don’t like it.
The other reason is that this year’s setbacks on extended voting pale in comparison to both the advances in many states and the long-term trend:
This is from the MIT Election Data Science Lab, and it shows a very steady increase in early voting since 1992. Modest ups and downs along the way haven’t changed this and aren’t likely to. Early voting is obviously going to skyrocket this year, and it strikes me that the American public has spoken pretty loudly and clearly about its preference for having a choice of when to vote.
In any case, what I’d really like to see is a federal law that sets voting standards across the country. It’s kind of ridiculous for every state to have wildly different laws, often set at the last minute by partisan hacks seeking some tiny electoral advantage. Why not one uniform standard for everyone?