If you are in business, it makes a difference. They will be a little richer and fuller bodied, more tender and it helps browning. If you are making them for fun, probably not if you still like the recipe. When I'm experimenting, I don't use the expensive local dairy stuff. I use the ultra pasteurized carton of whole milk. The expiration is over a month. This may work for you. I like the fat in my recipes. And in some cases, the fat can come from other sources. I don't know, but does carton milk in the little boxes come in whole?
If you try to sub, for example, butter, you have to account for the water too. A big pain.
I just googled it. One site said to make sure you don't overmix if you use skim. Another said that there is a powder that is whole. It's called Nido. Hope this helps.