Enabling Mission-Critical Machine Learning

Case Study

Enabling Mission-Critical Machine Learning

Making machine learning operations a reality in the intelligence community

Machine learning (ML) provides new ways to create actionable insights from existing data. However, developing and operationalizing ML models is challenging, even in commercial environments. Implementing ML can be even more complex in the intelligence community (IC). To accelerate change, one intelligence agency came to 蘑菇视频 Allen for a solution to help develop and deploy ML into operations for mission-critical national systems.

How do you make machine learning a reality across the intelligence community鈥檚 complex operational environment? 蘑菇视频 Allen鈥檚 Clairvoyant program provides a practical answer. The program, which includes three elements鈥擲ageMakerSpace, Integrated Data Platform (IDP), and Framework鈥攊ncorporates DevOps and machine learning operations (MLOps) concepts to develop and deploy mission-critical machine learning across the IC.

The Challenge: Operationalizing ML for Intelligence Missions

Within the IC, ML has the potential to disruptively advance the state of the art. Its intelligent and predictive capabilities can find insights in terabytes of data, recognize patterns in imagery, and expedite tasks that would take human analysts a significant amount of time to unveil on their own.

The rise of new technologies and sensors is causing the amount of data to grow exponentially. To support analysts, the IC needs to leverage ML to augment analyst workloads. Recent studies show, however, that out of all companies using ML, only 22% have actually deployed it in an operational capacity.

IC data scientists face several additional challenges in developing and deploying ML. Here are three of the main roadblocks:

A complex operational environment that makes it hard to use plug-and-play models
Training data that was fraught with issues鈥攎aking it difficult to gather, 聽clean, and match real-world data
A bare-bones development environment that wasn鈥檛 built for machine learning

The Approach: Targeting the Most Common MLOps Challenges

The Clairvoyant program was originally established to address a similar set of issues鈥攖hose faced by software developers in deploying software into intelligence operations.

Software development programs faced three key issues:

All programs, including both large, legacy applications and small, single-page websites, had to have cloud deployment experts on the team
All programs needed to seek formal security accreditation and authority to operate (ATO) before deploying into operations
Maintaining the software, once operational, required additional鈥攁nd often restrictive鈥攁llocations of workforce, time, and resources

Together, these challenges add months to a typical deployment schedule.

A Complete DevOps Tool Suite

In response, Clairvoyant provides a DevOps tool suite that has already significantly improved software development in the intelligence community. 蘑菇视频 Allen instituted:

A deployment framework that includes Kubernetes clusters for hosting containerized software
A continuous integration and continuous deployment (CI/CD) pipeline with security checkpoints to automate security accreditations
Program health monitoring and metrics

These three capabilities made it possible for all teams鈥攆rom massive, mission-critical enterprise programs to the smallest development teams鈥攖o quickly deploy new or modified programs into operations.

The agency鈥檚 second challenge set involved finding, streaming, and using data鈥攁 problem shared by both application developers and data scientists. Across the IC, data is siloed and in myriad schemas鈥攎aking it hard to find and integrate data during development.

蘑菇视频 Allen created an IDP to combine data warehousing, query tools, discovery tools, and streaming services in one solution, reducing barriers for developers and data scientists alike. The Clairvoyant IDP broke barriers across the enterprise by making siloed data discoverable and accessible to developers in a standardized format, making it possible to interface existing data with new, containerized applications.

This DevOps model standardized developers鈥� program interfaces and provided managed services for data, security, cloud infrastructure, and operational support. These services enabled developers to rapidly develop new capabilities and deploy into operations in an agile cadence.聽

The Solution: A Machine Learning Environment Deployable Across the IC

Clairvoyant鈥檚 SageMakerSpace is a purpose-built machine learning environment that can enable the use of machine learning across the IC through MLOps. SageMakerSpace enables data scientists to leverage the power of Amazon SageMaker via a secure portal so they can collaboratively build, train, and deploy machine learning models for their mission use cases. SageMakerSpace is even more powerful when used within the Clairvoyant DevOps ecosystem, which gives users the data they need and the ability to plug-and-play ML into operations.

To make it possible to deploy ML models for use in IC missions, 蘑菇视频 Allen developed a MLOps approach similar to the successful solution created for software developers. The modeling environment solution provides data scientists and ML engineers with industry-standard ML modeling tools and task-ready GPU and CPU hardware, key components for delivering an ML model to operations.

When combined with the other elements of the DevOps stack (i.e., the cloud hosting framework, CI/CD pipeline, and IDP), these capabilities eliminate the most common barriers to operationalizing ML models.

This innovative approach brings the intelligence community closer to exploiting all the advantages that ML models can offer鈥攁pplying the power of AI/ML to support the mission.聽

Article

1 - 4 of 8

蘑菇视频