Hey, welcome to the community
If you show what you're doing in the shader, I can help figure out how to fix it. Usually, however, you only mask the movement by the channel you want to affect.
So set up your vertex moves and multiply them by the green channel of your vertex colors to mask them.
You may also want to have a little leverage on the branches and thinner areas of the tree. So you can mix it into your blade movement with a multiplier or bias to control it.
I'm assuming you are using Unity and Shadergraph.