The first thing to say is that your son and daughter and your ex wife would not have needed any authorisation to use your card - all they would need is to have the card itself. It would not have been necessary for the card to have previously been 'registered' with Amazon.
Think about this - how could Amazon possibly have your daughter's email address if someone hadn't provided it to them?
How could Amazon have taken money from your bank account if someone hadn't provided them with your card details?
Based on what you have written here, no blame lies with Amazon. What do your ex-wife and your daughter and son have to say about it all? You say that Amazon emailed your daughter, and two days later the money was refunded - did she take any action when she got the email?