I finally decided to dive into the new tessellation shaders, and I am quite pleased. Going into it I thought it would be very specific to gaming applications, but as I've found out it is surprisingly general.
New primitive topologies have been added for tessellation. Since the basic unit for the new shaders are patches and control points, the new types allow you to render anywhere from 1 to 32 control points per patch.
Say you're using 16 control point patches. The vertex shader is run per control point, and the output from this stage is passed into the hull shader. The hull shader is really described in two parts; one being a patch-constant function, and the other being the hull program.
The patch-constant function computes user-defined data and is run once per patch. This allows you to compute things that remain constant across the entire patch. The required output from the patch-constant function are tessellation factors: these tell the tessellation hardware how much to tessellate the patch. The hull program is run once per output control point, and both the patch-constant function and hull program can see all control points.
The next step is the actual tessellation, which is performed in a fixed-function, yet configurable stage. The tessellator ONLY looks at the tessellation factors output from your patch-constant function. The user-defined output from the patch-constant function and hull program are provided to the domain shader, which is run after tessellation.
The domain shader is run per-tessellated-vertex and is provided with the tessellated vertex location on the patch. To me it seems that the domain shader can be seen as a post-tessellation vertex shader; this is where you transform the tessellated vertices. The output from the domain shader is provided to the geometry shader (or pixel shader, if not using a geometry shader).
Here are some results from my initial experiments with the new stages:
The toughest part to me is computing the per-patch tessellation factors. But since this is completely programmable, it's a fun problem.
A blog which discusses various GPU applications including visualization, GPGPU and games.