I have the same problem. It is tough. It really isn't about growing a backbone, so don't let anyone tell you that. Its just that you are a sensitive person and a pleaser. What is wrong with that?? Well, besides the fact that you get walked all over
I am working on my ability to charge people what my cakes are worth and you are absolutely right, they have no idea what they actually cost. When I first started my sister was like, "Well, just figure out your costs and then what your time is worth. The ingredients are probably, what 2...3 dollars?" I almost peed my pants! Don't I wish! I wish I didn't have to charge people what I do, but you just can't do it for less. Even for friends and family. If you do, you will get burnt out and quit.
A little tip that helps me. I do email really well. I can tell people whatever I need to about price and everything over email. If you have the opportunity to do it that way, try it. If nothing else, its good for practice.
I have to raise my prices again and its going to hurt telling people. One of my first cakes I actually said "Its $30. Is that okay?" Idiot, of course its okay, b/c that is what the price is. I can't believe I said that.
You're not alone. Keep it up, you are really good!