You made the right choice. Family comes first. You can still makes if you wish for family celebrations. After the kids are older, you can still get back into it. Maybe not as a business but as a hobby. Cake decorating is very time consuming. Only other decorators can appreciate that.
Best of luck to you and your challenge. I do agree with another response that customers do not fully appreciate all the time, effort, and money involved in decorating cakes, especially in your situation.