User-generated content (UGC) is a big problem for publishers. Right now, most publishers are relying on a combination of keyword filters and human moderators to maintain their editorial standards. Unfortunately, there are problems with both of these approaches. Keyword filters are a notoriously poor defense, as you can beat them by simply replacing a letter with a symbol. Human mods, on the other hand, are expensive and occasionally biased. Given this poor choice of options, many publishers choose to avoid UGC entirely. However, due to the need for publishers to maintain relevance in an increasingly competitive marketplace, this is no longer an option as UGC is quickly becoming a necessity.

We think it's possible to stop comment abuse, and that content-based filters are the way to do it. The idea has worked well for spam; so well in fact, that every major email system includes some kind of Bayesian filtering algorithm. The question now is whether the same algorithms that easily separate spam from legitimate content, can be used to separate abusive content from the non-abusive. Further, can they be used to identify quality contributions? If we apply the spam-filtering methodology at a user level, could we find a community's worst trolls or even its experts? The answer to each of those questions is a surprising, yes.

For a number of reasons, the task of identifying abusiveness is more challenging than a typical spam-filtering problem. The semantics of abusiveness are much more subtle and complex, and this is compounded by the fact that abusive users often obfuscate their comments in order to beat the standard keyword filters.

In order to address these issues we break up the indentification task into several sub-tasks. In practice, this means identifying sub-categories of abusiveness such as "Discriminatory", "Inflammatory", "Violent Threats", etc that are easier for a classifier to handle. We can then combine the input of all the sub-category classifiers and form more reliable and consistent conclusions using a meta-classifier. The same approach can be used to identify quality contributions as well with sub-categories such as "Congenial", "Insightful", and "Informative".

We are using proprietary algorithms that extend far beyond the typical Bayesian spam-filtering. The research behind JuLiA is an outgrowth of the cutting-edge research done by top text-mining experts and the incredible impact of next-generation algorithms on the field of sentiment analysis. These algorithms have been used primarily in academic settings, and we are the first to bring this research to bear directly for publishers of online content.

http://www.cs.cornell.edu/home/llee/papers/sentiment.home.html
http://www.cs.cornell.edu/People/tj/publications/joachims_01a.pdf