The Author's Death for Agency

It seems to me that Barthes wants to take the author out of the work in order for the work to have agency in itself. The presence of the author steals the agency away from the work. According to Barthes, "to give a text an Author is to impose a limit on that text" (877). Therefore the Author holds the agency until it is removed. With the author present, the book, text, or other work is largely defined by that author. The ultimate critical interpretation lies within the author and not the work itself. With the removal of the author the text then then contains a multitude of critical interpretations within itself. Without the author the text then has the agency.

Or does it? I don't see how the text or the author have any meaning or interpretation outside of the agency held by the reader. Wouldn't the reader's own perception shape the meaning of the texts? I'm not sure how Barthes can justify saying "the reader is without history, biography, psychology; he is simply that someone who holds together in a single field all the traces by which the written text is constituted" (877). I don't see how this is at all true. The text is constituted of language which ultimately contains the meaning of the text. But doesn't the  reader have agency of language as well precisely due to his history, biography and psychology? I might say "I have to get up early tomorrow at 8:30" whereas you might say "I get to sleep in till 8:30." I might say it's cold out when it's 45 degrees and you might say it's warm out when it's 45 degrees. These factors depend usually on what temperature you are most comfortable with (unless you're just trying to sound tough) and what kind of schedule you keep in your day-to-day life. In other words the variation in interpretation of words like "early" and "cold" depend entirely on the reader's history, biology, and psychology. So why wouldn't the reader still have agency over language, the author, and the text? And why does Barthes say the reader is without those traits? Why does the fate of the author and the text matter at all outside the reader?