Click a collected book icon again and you’ll get a prompt confirming you want to remove that item from your collected.
If this isn't happening, what happens when you try clicking again to remove it? Are you only having this problem on Trakt.tv or are you connected to some apps that are showing your collection incorrectly?
Also, try https://trakt.tv/settings/data. This can fix a lot of issues on Trakt.tv.
Please open a ticket if you need help investigating the issue.
I have the same problem and it doesn't look like anybody is getting any answers.