Today I finished shifting the ray generation part to the GPU, saving another 1-4ms as well as an unnecessary memcopy. Also silhouette-smoothing is working well, together with basic anti-aliasing ( so far only for GTX2xx cards ).
As for the smoothing, I tried two variants (left), and found the one in the middle looks best so far. The unsmoothed original (top) is too edgy and the one on the bottom smoothens too much for the tree-scene which lets near rendered geometry look like a 2D impostor.
The updated demo is here [-download-] (Cuda 2.1)
Also containing softening for the buddha & dragon scenes now
For the experienced ones of you, the shader-folder contains the shader in GLSL (soft.frag). You can experiment a bit by modifying the smoothing.