Building consistency

I’ve been thinking a lot lately about our JuLiA training process—how to build a clear set of directions for freelance taggers, and how to ensure that tagging results in quality training documents. See, ultimately, JuLiA’s intelligence is entirely dependent upon the quality of human tagged documents. The nature of her intelligence is that of a very consistent categorizer, so it follows inconsistently tagged documents will build an inherent inconsistency. Here are some of the questions that kept me up at night:

How do we build an expert system to perform a task at “better than human” accuracy with human-tagged training documents? Will not the training documents themselves be inconsistent? What’s more, when training documents are submitted by multiple people, are they not likely to be exponentially more inconsistent? Different people have different styles. Will their combined tags create an inconsistent JuLiA?

My solution is to build as much opportunity for consistency into the process as possible. Here's how:

1. Train consistently: everyone gets the same training documents, the same phone call, the same spiel.


2. Audit results: everyone gets the same amount of documents to tag, each tagger stops upon reaching certain milestones, at each interval a consistent amount of documents are audited for accuracy and consistency by the project manager.


3. Re-iterate training: during audits notes are taken and sent to the tagger, taggers correct their own mistakes.


4. Be available: a means of reaching the project manager is always available and taggers are encouraged to contact the PM whenever they become unsure of their decision-making


5. Iterate process: when unforeseen questions are asked or mistakes are made by one tagger, the rest will receive the same feedback regardless of their own performance.


6. Alternate groups: because tagging is tedious and humans are prone to fatigue, boredom, and inconsistency when faced with boring tasks, no tagger will do more than 10,000 documents in a two month period.