Bug Fixes and Things

It turns out that using this background was actually masking some nasty bugs I had with not enough knots on my color and skeleton spline. Those issues, which looked pretty bad against a white background, have been fixed. The skeleton spline knot error that I had was causing the piecewise sinusoidal alpha attenuation function not to work properly (because it's defined over [o, 1], and some print statements informed me that I was only going to [0, 0.6], which is close to where alpha is a maximum.)

The blending seems decent enough, although I am sure it's possible to find some really pathological colors to mix together that look bad. Here, I have preserved the alpha the blending; in fact, the resulting alpha value for two splines that collide is equal to the greatest value of the attenuation function. I did this to make sure that colliding splines don't seem to "fade" when the center where one aurora is strongest collides with the edge of another spline, where that spline is supposed to fade away to an alpha of zero.

The next two projects I want to look at are getting some kind of a brightness adjustment, perhaps with a noise map, and looking at some notion of depth so that I can further refine the aurora blending.

To be honest, I think I am close to maximizing the potential of this kind of structure. In combination with different shapes, or small piecewise auroras, it could be effective. Or possibly another approach is needed entirely. One thing that does bother me, for example, is that we have no easy way of having pixels cast rays at the normal to the NURBS curve. It would be really nice if the color spread followed the actual normals instead of some arbitrary normal that I specify. For shapes that loop, fold, or spiral, like many found in nature, my structure is deficient.

Not really sure what to do but keep plugging away.

Blending Colliding Splines

To simulate more complex auroral structures, we will eventually want to be able to blend overlapping auroras. This is not an obvious process, and it is not without its flaws. What I have done is to create a weighted average when multiple splines collide. I let W be equal to the product of the thickness and endpoint interpolation (both are functions with a domain and range of [0, 1]). Then each color is put into the average as

Sum(for all i ) [Wi^2*Ci] / Sum(for all i) [Wi^2]

This removes the hard edges along the collision regions, but it also could make colors blend into one another too fast. I will have to do some experiments to figure out a sweet spot. The bottom picture is of three auroras that collide without any alpha attenuation, and with weights equal to 1. The top picture corresponds to the same splines, squared weights, and both skeleton endpoint and height interpolation.

I will try to do some comparisons tomorrow for reference.

Noise Alpha Attenuation

Something else we can do is use diamond-square noise height maps (due to Lucky) to create an organic kind of randomness to the aurora. Not really sure how much mileage we get out of it though... The alpha channel becomes

Alpha = 255 * Noise(s, t) * EndpointInterp(s) * HeightInterp(t);

with all the noise and interpolation functions having a domain and range of [0, 1]. Here, s represents the parameter moving along the skeleton spline and t represents the value of the distance from the pixel to the skeleton spline divided by maxdist(s).

Here, you can see the difference between having noise and not having noise. I might try doing a brightness adjustment with the noise instead.

Gaussian Alpha Attenuation

As a preface, I fixed the endpoint interpolating issue. The problem was that I was confusing the polynomial order with the degree. As a result of this confusion, I did not use enough redundant knots. Including them has fixed this issue.

I have now turned my attention to the issue of how to round off the boundaries of the spline itself, particularly near the base, left and right sides, where the aurora does not blend convincingly into the background image.

To make the blending more convincing, I have used Gaussian functions to attenuate the alpha channel. Right now, what I am doing is something like

Alpha = 255 * exp(-1 * c^2/(2*q^2) * exp(-16 * t^2)

where c = d / dmax - f, q = .1, f = .4, d is the point distance along the normal direction to the skeleton, and dmax is the result of computing the maximum distance spline function for the parameter value of t (which is also used to compute the spline shape and color).

While these mathematics are crude and convoluted, my point here is to convey the fact that we want the alpha to attenuate according to two factors. First, the alpha should be greatest at some "center" value of t in the range [0, 1]. This will help to make sure the ends of the aurora will not stick out. Second, for the same reason as before, we want the alpha to attenuate from some "center" defined as a fraction of the maximum distance spline function. This way, the aurora maintains the same shape as the skeleton, but eliminates the need to give the base special treatment.

First Attempt at Rendering

For the first time, I am rendering onto something other than a black background, and that reveals a few fundamental flaws. First, as I have previously mentioned, the sides (i.e., the gradient near the ends of the skeleton spline) do not blend well into the background. They look distinctly artificial. I am not sure how to fix this at the moment.

Second, the problem just revealed is that our B-splines are not endpoint interpolating. That means we have gaps at both ends where the curve goes off to zero. This is most obvious in the color spline (so, at the base of the aurora, the color goes to black when it should go to purple), and in the distance spline (so, at the top right end of the picture, the distance goes to zero, and the aurora tapers off to a single point). I am revisiting the code for the B-splines to fix this problem.

Third, although the aurora colors are interpolated in CIE Lab space, the alpha blending occurs in RGB space. One artifact I have noticed it causes is the pink near the base of the skeleton, where we should see a deep purple instead.

A C++ Port for Optimization

I was disappointed with the speed I was getting from C#, so I ported the code over to C++ to take advantage of the SSE instructions and other optimizations. At first, I didn't notice a major improvement, which suggests some slowness in my algorithm. I tracked it down to the BSpline intersection method.

The problem on there is that it involves bisection over the parameter space of the spline, which results in doing a lot of calculating. And in fact, a lot of points, particularly the endpoints, were being calculated once for each pixel in the image, which doesn't really work well, especially because I am going to want to sample with multiple rays later on.

To solve the problem, I noticed that I could precalculate a large table of points on the spline over the parameter space, say with a delta-t of dt=0.0001. Instead of calculating stuff over and over again, I could just compute the table once and simply reference the table in my intersection method. This solution was a huge win, because on a 500x500 image with a single aurora spline, I was able to do the optimized rendering in 0.48 seconds instead of the 6.00 seconds it was running before: the improvement is about 1200% for that situation alone!

Further Comparisons of Color Space Interpolation

Color interpolation works differently in various color spaces. I wanted to get a feel for how spline interpolation would work in the various color spaces, like CIE Lab and CIE Luv in particular. The "Color Transfer Between Images" paper [Reinhard et al. 2001] mentions an L-alpha-beta space, which I do not believe is actually related to the CIE standards. I also experimented with LCH(ab) and LCH(uv), but I am disenchanted with them because the hue coordinates are bent into a cylinder: the interpolation goes through colors we do not want to see.

One thing that is interesting is that YCbCr looks a bit grayed or washed out compared to the other color spaces, so perhaps it would not work well in our application. I have trouble discerning much of a difference between Luv and Lab.

In the first image, from top to bottom, the color spaces are RGB, YCbCr, L-alpha-beta, Luv, Lab.
In the second image, it goes RGB, Luv, Lab, LCH(uv), LCH(ab).

Variable Aurora Thickness

Nothing in nature is truly uniform, so our auroras would be deficient if we did not somehow account for the variability of an aurora's thickness. To do this the quick and dirty way, I used--big surprise--another B-spline, where the control points have dimension one: they are simply distances measured in pixels.

The change to the rendering algorithm was simple. Instead using a maximum distance property for the aurora, I calculate where the ray from the pixel crosses the spline, saving the t-parameter value. That value is in the range [0, 1], so it is fed as an input into the distance spline function. The output of that is the maximum field distance. That distance is compared to the pixel's distance to the aurora skeleton spline, and the color interpolation is done according to that distance. It provides a nice sense that the spline dimensions are changing. In addition, it adds realism because the color bands no longer have a uniform shape as they blend into one another, so the geometric properties of the aurora appear to be more complicated than they actually are.

Colliding Implicits

I was playing around with the problem of what to do when spline skeletons collide and how color should be normalized. Lucky included some field functions in his Starlight Penetrating project, so I gave those a try. It seems the the function F(d;radius) = (1 - d/radius)^2 seems to produce reasonable output. It is clearly better behaved than a simple average of all the colors should multiple auroras collide in image space. I have included an image with three splines on it. The field function defines the weights, and the final color at each pixel is a weighted average according to the specification that Lucky set out in his paper (without the inclusion of ambient color).

I think it's possible to do better. In particular the endpoint boundaries on the spline are too strong, even with the mixing. At this point, I am beginning to think we might benefit from casting multiple jitterred rays at the splines, which might reduce aliasing as well as the sharp discontinuities between the aurora bands.

New Directions...

Overall, I am pretty happy with the YCbCr space in terms of the slightly smoother color blends. I have included a few more pictures that show where things are going with the aurora rendering. In the mid-term, there are three major projects to address before this can be called research, or at least artistic:

1) I have to add support for colliding splines to make sure that the colors can mix in a convincing way, rather than just coming up with a muddy gray as I have been thus far. Lucky has some code for his Starlight project, and I think I might be able to glean a few secrets from that.

2) We need an alpha mask of some kind for blending the aurora itself into a background image, like a mountain scene. This should also serve a dual purpose of mitigating any problems with the edges of the aurora being too sharp (which they are right now).

3) We need a brightness mask so that we can perform an adjustment on the aurora to make it look bright on the bottom, probably along a curve roughly parallel to the aurora's NURB-spline itself.

B-Spline Interpolation in Other Color Space

Well, I've had my fun with RGB, but I thought it would be worth trying other color-spaces to see how the interpolation behaves. I have a sample of RGB vs. YCbCr here. The top bar of each pair is in RGB and the bar below is in YCbCr. The feature that seems to be be most apparent is that the transitions seem slightly smoother, and the colors are more tempered overall. I think the luminance is also constant, but it's hard to tell.

Why Spline Interpolation?

A question that might be asked about this research is why it is so important to use a technique like spline interpolation? Does linear interpolation not suffice? From a mathematical perspective, I am trying to do justice to the fact that aurora colors are produced by the excitation of atmospheric oxygen and nitrogen. Because of the chemical properties of atomic and molecular oxygen and ionized nitrogen, each respond differently to excitation. Not only do these elements produce different colors, but they also have a varying excitation time--anywhere from 0.7s to 110s. This fact dictates that some colors can only have so much influence, because the photons they release do not contribute as much to how we perceive color as the more plentiful photons coming from excitations that last longer.

Since B-splines have local support at more than simply 2 points (or color vectors in our case), which allows us to reflect the contributions from multiple sources in a continuous way. Notice how, in the picture above, the top bar of each pair shows piecewise linear interpolation. We can see a line between the pieces, which is undesirable. In the bottom bar of each pair, we see that B-spline interpolation duplicates the gradient without any deficiency. The discontinuities are a consequence of the fact that linear interpolation has support only between two points and is not C2 continuous.

Preliminary Rendering

At the moment, the algorithm works as follows: The inputs are any number of NURBS, which represent the shape of the aurora; a normal vector for each aurora, which represents the direction of the color spread; and a B-Spline of order 4, which represents the blending of different colors farther from the aurora's NURB-spline.

Basically, the idea is "for each pixel, cast a ray in the direction of each spline (meaning, reflect the spline's normal, and walk toward the spline. For each intersection, if the distance between the pixel and the spline is less than some arbitrary maximum distance, look up the appropriate color by computing the B-spine color interpolation for that aurora's color spline. Perform normalization as necessary, and color the pixel as the final normalized color."

One thing it does not handle particularly well at the moment is multiple auroras that collide. When they collide, the color mixing is not convincing at all. This is probably because I am counting the total intersections and dividing by that number. In reality, it doesn't work that way: we can't just do linear combinations of color.

Another consideration I have at the moment is that we have no antialiasing for the aurora, which we will need to look convincing. To my mind, this can be accomplished in one of two ways: first, we can send multiple rays that are slightly jittered, and average the color after; second, we can apply a Gaussian blur as a post-processing step. The former idea seems okay, but I have my reservations about it, since it will work well around the boundaries of the aurora's NURB-spline but it may cause bad blurring in the interior area of the aurora (since different rays will intersect the spline with different distances, which will result in mixing color from the color spline in ways that defeat the purpose of spline color interpolation). At the moment, I feel like a Guassian blur kernel, sensitively applied, would be the best solution.

On the Colors of the Aurora borealis

Reading through "Simulating the Aurora borealis" by Baranoski et al. provided me with a good grounding for what is physically happening with the Aurora borealis. The causes of the beautiful colors are well known. What interested me is exactly what colors we observe.

The researchers provide light wave lengths that they say are caused by the excitation of atomic and molecular oxygen and nitrogen. They talk about an atomic oxygen "green line" at 557.7 nm, an atomic oxygen "red line" at 630.0 nm, and an ionized nitrogen "blue band" at 427.8 nm. Using an online conversion applet, I was able to obtain approximate RGB colors for experimentation and comparison with actual photographs of Aurora borealis.

Aurora borealis colors seem to follow general patterns. Seeing a green band low in the sky blend into red as one looks up is not uncommon. Green blending into blue is fairly common. Straight blue auroras are also fairly common. I did find one photograph that was mostly red, and I have seen red and blue together (though they appeared to be different bands, rather than one band of color blending with another color).