it's because having "freedom" and "lots of choice" are great for marketing and getting people hyped, but with ballooning development costs from giant open worlds and a slavish devotion to realistic 3D graphics, actually implementing a lot of choices that have rippling effects and each require fully voiced dialogue is easier said than done. so they posture at giving you options, only to railroad you. and while this has always been done and is often an accepted reality of game development, it's becoming less and less subtle, because hiding that railroading is itself something that takes skill and time.
Back in the 90's, most cutting-edge RPGs had text boxes. Some had voice acting, at least for key moments, but most just had text. Look at Planescape: Torment (1999).
It has a lot of dialogue choices, in part because each dialogue option (and NPC responses) were mostly just text strings that may or may not trigger certain scripts when the dialogue was over. They could include as many as the writer could think of, and if they realized something was off, they could just edit the text.
You didn't need to pay animators and voice actors and maybe motion-capture artists to re-perform every minute variation of a scene, and you didn't need to redo all that work to account for minor changes.
It used to be practical for RPGs to be developed with open-ended dialogue webs and forests of flags. If you can write it, it can go in the game. And when they're written well, that's extremely appealing! Modern WRPG* developers want that kind of appeal! But AAA games "need" loads of bells and whistles to look Modern™, and that gets expensive (especially if you need to change things. So keep it simple, keep it lean, while giving the player the best sense of agency you can fake within this rigid structure.
It's no accident that the best-written RPGs of the last decade have been throwbacks to older styles of RPG. Disco Elysium cut the special effects down to text and voice acting, which gave them the same freedom Planescape: Torment had back in the day.
AAA WRPGs* with a clever script can mitigate the overt lack of player agency imposed by these production limitations. Cyberpunk 2077 is, by all accounts, not particularly clever.
*Western RPG. In this context, it's meant to distinguish video RPGs in the lineage of Might & Magic or Ultima from JRPGs like Dragon Quest. JRPGs have their own parallel history of grappling with the limitations of 80's and 90's technology, and then the burdens of 21st-century budgets, but the genre norms were different, which comes with different expectations and burdens.




