Possible Anomaly Detection Algorithms

Some possibilities include:

Profiling short sequences of system calls (pH, Sana Security's Primary-Response)
System call flow graphs (PAID)
Semanitc constraint violations at the system call level (Systrace)
Various combinations of supervised and unsupervised learning algorithms on network or application input data (i.e. Stefano Zanero's work)
"Loose" signatures (most current IPS products)

Each method has its strengths and weaknesses, but these methods are new enough to most people that the positive and negative attributes are generally mis-understood at best.