When I decided to procedurally generate bushes for Hedge Maze (back when it had the inspiring title "Maze Game"), the idea seemed daunting, to say the least. I had never done procedural geometry before, and I couldn't find my copy of How to Scatter Leaves on Bushes for Dummies anywhere on my bookshelf. Nevertheless, I decided I was up for the challenge.
Base mesh
I knew I was going to want to create the base mesh procedurally in order to have full control of the resolution and vertex placement. Unfortunately, I had no idea where to start with the procedural geometry.
Lucky for me, Catlike Coding had a tutorial on the exact thing I was looking to make. I loved this tutorial: it was simple to follow and explained the thought process every step of the way. It demystified the process of procedural geometry for me, and I was able to use it to create the vines for the game as well.
Adding leaves
Once the base geometry was finished, it was time to work on the leaves. I was quite proud of the leaf scattering. I was out taking a walk when I came up with the idea for how to do it. As soon as I got back home, I sat down to test the idea, and to my surprise it was more or less perfect on the first try. (That doesn't happen to me very often!)
Here's a breakdown of how the leaves are scattered. On the left you'll see the actual leaf, and on the right is a visualization of the position and angle.
First, a leaf is placed on each vertex of the bush mesh, aligned with the normals:
Next, the position is offset a little bit to break up the uniformity:
Finally, the leaves are rotated to lie almost flat against the surface:
Building a maze
I created a "bush" object by combining the procedural mesh and the leaf scattering system. With just a few variables, I was able to have a high degree of control over the look of the bush, which I spent a lot of time fine-tuning.
Compared to all the work that went into creating an individual bush, using it to make the maze walls was quite simple. The maze generation already took care of defining the size of the walls and spacing between cells, so I used those values to drive the size of the bushes, and created one for each wall of the maze. As far as intersections go, I just decided to let the corners overlap a bit. With all the visual noise from the leaves, you really don't notice the overlap.
GPU instancing
Another thing that I want to mention before I wrap up is that my initial approach to creating the leaves was very naive, and did not scale well. I started by just creating a new leaf object for each leaf in the scene. That worked fine in testing, but it got really laggy when drawing larger mazes. I ended up using GPU instancing to draw the leaves in batches, which made a huge difference. I was able to draw millions of leaves, all with vertex displacement animations, with almost zero cost to performance.