1) Libraries have protective charms.
When Muggles get close to Hogwarts (which they can't see -- because magic) they suddenly remember something very important they have to do, so they rush off. Hogwarts' protective charms help keep Muggles, as well as Voldemort's followers, away from the school so the students inside are safe. Like Hogwarts, libraries have protective charms too and are seen as a safe space where all people (both magical and Muggle) can go. Remember when the Ferguson Library stayed open for the community when even the schools were closed? In addition to bringing spellbinding stories to the public, the library is a safe place and that's pretty magical.
2) Librarians are the ultimate defense against the Dark Arts.
They banish ignorance faster than an expelliarmus charm. A common boggart among librarians is ignorance, so they're quick to protect themselves and their surroundings against it. Riddikulus? Yes. Ridiculous? No.
3) Every time you check out a book it's like getting 10 points to Gryffindor.
Or 10 points to Slytherin if you're checking out books on potions or just being evil. Or you can totally claim 10 points for yourself because you were awesome enough to support your local library by getting those circulating numbers up (which helps libraries make a case for increased funding).
4) Have you seen the hallowed halls of some of the world's most gorgeous libraries?!
Some of them are practically castles. I'm talking the New York Public Library's central branch with the stately stone lions out front or even the Linn-Henley Research Library in Birmingham. Not to mention the libraries in Europe that are older than the whole U S of A and others all over the world. Between the long wooden tables, the marble staircases, and the high, arched ceilings, the hallowed halls are about as close to Hogwarts as we Muggles will ever see.
Not to mention that if the NYPL was under attack, all you'd need is McGonagall with her piertotum locomotor charm and we'd be saved! If it's good enough for the Battle of Hogwarts, it's good enough for me.
5) Every bookshelf is its own Chamber of Secrets.
And thanks to Harry and his merry gang, you won't find any angry basilisks in there! You don't need an education from a school of wizardry to know that books are magical -- after all, without books we'd never know about The Boy Who Lived.