I've been working with a plugin that embeds kb attachments, and it breaks as soon as I log out too, think me there's a misunderstanding about "public" somewhere.
Kb attachments redirect to the login page, I don't use login for guests, so none of the users have accounts.
Looks like all attachments are stored the same way, and are accessed the same way, so it restricts who can access them.. well, except for the logo, and the pages.. hmm, might be something there.