You shouldn't be frilling with a ball tool, that won't frill the petal the way you need it for a hibiscus. You want to use this type of a frilling tool (or some people just use a cocktalk/tooth pick)
Lay the petal on your pad, have the tool laying down with the thinner tip over the edge of the petal (have a look at my really bad picture attached), and roll the tool back and forth across about a 1cm area. You will need to practice the weight you put on the tool and how far into the petal you place the tip. When that area is frilled, lift your tool and move it up 1cm and work on that little bit and continue around the edge you want frilled.
Really hard to explain in writing - watch some tutorials, perhaps on frilling because from your pictures you do seem to have the idea for putting the flowers together.