(I originally wrote this for https://www.reddit.com/r/gamedev/comments/1nfps3.... The comments are all there.)
Almost a year ago, I made a post on r/gamedev, asking how to improve the pitiful state of the 2D smoke hazards in my game. A short summary of my previous post: I have a mobile game using OpenGL ES, with geometry-based levels (loaded from SVG files), I needed a "smoke area" hazard, where the area is defined as an arbitrary polygon, and I didn't quite know how to do it.
I got a few great tips, but I haven't had much time to work on the game, so it kind of fell by the wayside. Recently I've been able to squeeze a few improvements in, so I thought I'd share the solution I eventually used for this problem.
It still doesn't look as good as I'd like, but it's a notable improvement over the old version. The solution I ended up using is pretty simple and hopefully someone will find it useful.
Part 1: the smoke itself
So the first thing I looked into was just using smoke particles floating inside the polygon area, bouncing off the inside edges (also tried it without the bouncing) and eventually disappearing as new ones appear - this is how trails of smoke and similar effects are often done in simple 2D games. Except I needed to fill a polygonal area with it.
It did't turn out very well - it was too sparse, you would have to have many layers of particles, spread evenly to make it look good. That would mean drawing 3-4 times over a lot the area. Because the game performance is already bound by pixel operations, this would be totally unfeasible. Another disadvantage is that with larger particles, it's not possible to handle very acute angles - it just doesn't fit there.
Another suggestion was perlin noise procedurally generated in the fragment shader. Looking around the web, I found several implementations (because why reinvent the wheel when I'm not particularly interested in this area), the most popular being the one at https://github.com/ashima/webgl-noise (though that's actually simplex noise). It claims to be ok performance-wise, so I downloaded the shader and put it into the game.
In the desktop version, it looked fine:
and ran fine, so I put it into the mobile version. Not only did it not look right there (I didn't investigate why, but the shader compiled ok, I suppose it wouldn't be very hard to fix), it was way too slow, even on relatively powerful hardware (Galaxy S3 - Mali 400 GPU).
In the end I decided to just use a plain-old texture - more precisely a series of textures representing 2D slices of a 3D noise texture. OpenGL ES 2.0 doesn't support 3D textures, but since I don't need that many frames, I just decided to use a bunch of 2D textures - 64 to be precise. With steps large enough so that it's realtively smooth, it's enough frames that people won't notice it's looping when they're not specifically looking for it.
As for the textures themselves, they're 128x128 8-bit textures (so all the frames in total should take 1 MB of memory), stretched so that the tiling isn't super-obvious but so they still don't look too pixelated either. They way I currently use it, the noise function values just represent the color, but they could easily be used for alpha as well, or anything else.
To generate the texture, I needed 3D noise tileable in all dimensions (so the animation would be seamless). Luckily, I found this implementation http://code.j3d.org/javadoc/org/j3d/texture/procedural/PerlinNoiseGenerator.html in the j3d Java library and used it to generate all the texture files.
After creating the slices and a few tweaks, it was done.
Part 2: fading out along the edges
Second part of the problem was to have the noise fade out along the edges, so it wouldn't look like this.
This one is solved pretty easily, by creating a buffer around the polygon (also known as polygon offsetting, basically the same problem this person had http://stackoverflow.com/questions/1109536/an-algorithm-for-inflating-deflating-offsetting-buffering-polygons) like this
and setting 0 alpha to the outside vertices and 1 to the inside of the buffer. Just scaling the polygon doesn't work, for this reason:
A ready-made solution is available in the Clipper library - http://www.angusj.com/delphi/clipper.php. There are a few things about it though, that are bit inconvenient: I used the jtMiter join type.
(picture is from library's website). This works well for all angles, no matter how acute or reflex they are. But in the end I still have to convert the resulting buffer to triangles
, for which I use a very simple algorithm that doesn't handle some of the extreme situations where points would be removed. This is not a problem for my game though, as I won't have any such polygons in there.
An alternative solution would be to subtract the inner polygon from the outer one, and then break the result into triangles, for example with the poly2tri library from https://github.com/jhasse/poly2tri (that would work for all polygons).
There is no connection (that I know of) in the library between the resulting buffer polygon and the original one. I had to modify the library a bit to return the information about which corners were squared, in order to match them to the original polygon.
The result always starts from the bottom-most vertex and continues in a counter-clockwise direction (the input polygon's winding also has to be counter-clockwise - though it depends on the coordinate system, it's all in the library's documentation)
Fading out linearly didn't look exactly great though, the outer edge was still a bit too sharp (
). Squaring the alpha made the start of the buffer way too pronounced. In the end, I remembered these easing equations http://www.gizma.com/easing/, which are used primarily for nice looking motion, and applied this one (http://www.gizma.com/easing/#sin3). Now it looks a bit better (
). Textured: 
. I'm not sure about the exact performance hit, but overall it seems to work fine.
In the end
The result still looks kinda bad on its own, but when used in the game, it's pretty ok:
(I also added a tiny bit of green tint to it) - it looks like an area that the player would intuitively want to avoid, and it does resemble smoke at least a little. It's definitely much better than the awful first version . And it's really fast.
It could probably be tweaked a bit more (like using larger textures) to achieve nicer results, but for, now, I'm satisfied with the current state.