How good does Natural Language Processing need to get?

There is a lot of discussion in academic circles about diminishing returns when it comes to machine learning algorithms. Research often runs into a cap on accuracy, usually somewhere high, between 92-100%, at which point tweedy types stop scratching heads and start pounding chalkboards. (No knocks meant at the academic community. Professor brat here.)

At these percentages, the accuracy progress slows to a crawl and you’ve entered the domain of diminishing returns. You don’t have true artificial intelligence, but you have a very smart algorithm. You’re killing yourself for a point or two more.

I’ve spent the last week or so tagging thousands upon thousands of training documents, which we’ve used in turn to re-train our current algorithm. We are seeking our own version of the holy accuracy grail, the difference being, we are not after 100%. That’s just not our focus, and it doesn’t have to be.

At the moment we are focused on using the accuracy we have to do new, interesting, and useful things. The fact is, even without 100% accuracy, you can do just that. For example, sentiment analysis starts to get eerily good at 95%-98% accuracy.

At the Cogito blog, Luca Scagliarini has a semantic dream:

Luca has it right when it comes to gaining traction with enterprise clients, and it’s the same with investors.

You have to build something that is adopted by consumers, which is why a lot of investment groups are watching semantic technologies with intense interest. You don’t need 100% accuracy with these technologies. In the end it’s about another kind of number…the number of people using your app.