Yeah it should be a pretty simple system. Have the 4th ref look at all potentially reviewable plays before timeouts or stoppages in play to where the refs calling the game can just walk to the scorers table, consult the ref at the table who tells them what the call should be, and they call it.
And as you say, review any potential close play without having to make a call in order to review it. That's such backward thinking.
But the length of time that these reviews take is so awful. It needs to be fixed right after the season ends, and I'm sure it will with it already been talked about.