Challenge

Developing expert AI systems using Machine Learning (ML) models depends on large amounts of accurate, consistent, and comprehensive training data.​ Raw data is labeled by experts in the domain of interest to create accurate training data. These labeling tasks must be scaled to produce large amounts of consistent and comprehensive training data.

It is challenging, however, to implement a scalable data labeling infrastructure while still enabling rapid experimentation. The key challenges are:

  • Building an in-house data labeling infrastructure requires significant investment
  • Ongoing QA/QC of label quality and labeler performance is necessary to meet project deadlines and to control costs
  • A data labeling infrastructure must integrate easily into the organization’s data storage systems and machine learning pipeline

Successful ML deployments all begin with a custom ontology representing the objects or targets in question. Labelers create training data according to a defined ontology to train a model. A quality labeling interface will accommodate this ontology (as well as iterative changes to it) and present it to labelers through a seamless user interface. Building such custom labeling interfaces can be difficult and expensive.

Solution

NT Concepts needed to label 5–15 target objects in tens of thousands of still images hosted in Google Cloud Platform. Using Labelbox’s interface, NT Concepts defined classes of objects to label in the imagery. As the model training progressed, NT Concepts added new labeling classes by using Labelbox’s interface editor.

“I really love the simplicity of the solution while still providing amazing functionality. The team is easy to get in touch with and are constantly working to improve Labelbox through user feedback and innovation. I highly recommend Labelbox for any size team that is working on a project that requires image segmentation or classification.”

— Zach, Machine Learning Engineer, NT Concepts

With the labeling interface setup, Labelbox was connected to cloud storage and the labelers were set to work. While the labeling process was ongoing, metrics charts provided insight into the accuracy of the labels and productivity of the labelers. This enabled the QA/QC personnel to monitor the labels being generated and promote the most accurate labelers to be project administrators to correct defective or missing labels. The leadership team was able to effectively manage the labeling workflow, labeler tasking, and labeler roles and responsibilities.

Labelbox data labeling performance metrics **This is a demonstrative illustration, underlying data is fictitious and in no way is related to the NT Concepts project.**

After just one day of labeling, NT Concepts was able to begin interim training of the model, interpret the results, adjust parameters for labeling classes, and iterate on the labeling process to obtain initial results. Labelbox’s administrator interface promotes an iterative and real-time monitoring of the labeling process, which is integral to successful model training and useful ML results.

“We looked at several open source labeling solutions for our internal efforts and eventually selected Labelbox for its simplicity, ease of use, cost-effectiveness, and responsiveness of its development team. The choice was a good one and we will be using Labelbox again for our progressive and increasingly demanding AI and ML initiatives.”

— Mel, Program Manager, NT Concepts


About NT Concepts

NT Concepts is a trusted technology-driven solution innovator. We work on the most sensitive and mission-critical programs in National Security. We provide our clients highly differentiated solutions in business operations, AI/ML, geospatial intelligence, digital platforms and services, and security. Founded in 1998 and headquartered in the Washington DC Metro area, our customers span the Federal Civilian, DoD, and Intelligence Community markets.


Get Started with Labelbox

Visit www.labelbox.com to evaluate Labelbox for free or speak to one of our team members to find a right solution for your business.