TRENDING
  • Privacy Policy
  • Write For US
  • Contact Us
  • About Us
No Result
View All Result
  • Home
  • Business
  • Technology
  • Startups
  • Reviews
  • How To
  • Security
  • Devices
    • Smartphones
    • Tablets
    • TV
    • Wear
  • Internet
  • Marketing
  • Services
SUBSCRIBE
  • Home
  • Business
  • Technology
  • Startups
  • Reviews
  • How To
  • Security
  • Devices
    • Smartphones
    • Tablets
    • TV
    • Wear
  • Internet
  • Marketing
  • Services
No Result
View All Result
No Result
View All Result
Home Technology

3 tips for model monitoring for reliable results when deploying AI

by Janice Allen
November 4, 2022
in Technology
0
Wat is AIops?  |  VentureBeat
Share on FacebookShare on Twitter

Couldn’t attend Transform 2022? Check out all the top sessions in our on-demand library now! Look here.


Artificial intelligence (AI) promises to transform almost every business on the planet. That’s why most business leaders are wondering what to do to successfully take AI into production.

Many get stuck trying to decipher which applications are realistic for the business; that holds up over time as the business changes; and that put the least strain on their teams. But during production, one of the leading indicators of an AI project’s success is the ongoing model monitoring practices that are put in place around it.

The best teams use three main strategies for monitoring AI models:

Contents

  • 1 1. Performance Shift Control
    • 1.1 Event
  • 2 2. Outlier Detection
  • 3 3. Track data drift
  • 4 A structured approach to success
    • 4.1 DataDecision makers

1. Performance Shift Control

Measuring shifts in the performance of AI models requires two layers of metric analysis: health and business statistics. Most Machine Learning (ML) teams focus solely on model health metrics. These include metrics used during training, such as precision and recalls, as well as operational metrics, such as CPU usage, memory, and network I/O. While these statistics are necessary, they are insufficient on their own. To ensure AI models have real world impact, ML teams also need to monitor trends and fluctuations in product and business metrics that are directly impacted by AI.

Event

MetaBeat 2022

MetaBeat will bring together thought leaders to offer advice on how metaverse technology will change the way all industries communicate and do business October 4 in San Francisco, CA.

Register here

For example, YouTube uses AI to recommend a personalized set of videos to each user based on several factors: watch history, number of sessions, user engagement, and more. And when these models don’t perform well, users spend less time watching videos in the app.

To increase performance visibility, teams need to build a single, unified dashboard that highlights model health metrics next key product and business statistics. This visibility also helps ML Ops teams effectively resolve issues as they arise.

2. Outlier Detection

Models can sometimes yield an outcome that is significantly outside the normal range of results – we call this an outlier. Outliers can disrupt business results and often have major negative consequences if they go unnoticed.

For example, Uber uses AI to dynamically determine the price of each ride, including price increases. This is based on several factors, such as driver demand or driver availability in an area. Consider a scenario where a concert ends and visitors request a ride at the same time. An increase in demand allows the model to increase the price of a ride by 100 times the normal range. Riders never want to pay 100 times the price to hold a ride, and this can have a significant impact on consumer confidence.

Monitoring can help companies weigh the benefits of AI predictions against their need for predictable outcomes. Automated alerts can help ML operations teams detect outliers in real time by giving them a chance to react before damage occurs. In addition, ML Ops teams must invest in tooling to manually override the output of the model.

In our example above, detecting the outlier in the pricing model can alert the team and help them take corrective action, such as knocking out the spike before riders notice. In addition, it can help the ML team to collect valuable data to retrain the model to avoid this in the future.

3. Track data drift

Drift refers to the performance of a model that deteriorates over time once it is in production. Because AI models are often trained on a small set of data, they perform well initially because the real-world production data is very similar to the training data. But over time, actual production data changes due to a variety of factors, such as user behavior, geography, and time of year.

Consider a conversational AI bot that solves customer support issues. As we launch this bot for different customers, we may find that users can request support in very different ways. For example, a user requesting support from a bank may speak more formally, while a user on a retail website may speak more casually. This change in language patterns compared to the training data can cause bone performance to deteriorate over time.

To ensure that models remain effective, the best ML teams track the drift in feature distribution, that is, embeddings between our training data and production data. A major change in distribution indicates that we need to retrain our models to achieve optimal performance. Ideally, data drift should be checked at least every six months and may even occur every few weeks for high volume applications. Failure to do so may cause significant inaccuracies and impair the overall reliability of the model.

A structured approach to success

AI is not a panacea for business transformation, nor a false promise of improvement. Like any other technology, it holds tremendous promise, given the right strategy.

If AI is developed from scratch, it cannot be deployed and then left on its own without proper attention. Truly transformative AI implementations take a structured approach that includes careful monitoring, testing, and increased improvement over time. Companies that don’t have the time or resources to follow this approach will find themselves caught up in a never-ending catching up process.

Rahul Kayala is chief product manager at Moveworks.

DataDecision makers

Welcome to the VentureBeat Community!

DataDecisionMakers is where experts, including the technical people who do data work, can share data-related insights and innovation.

If you want to read about the very latest ideas and up-to-date information, best practices and the future of data and data technology, join DataDecisionMakers.

You might even consider contributing an article yourself!

Read more from DataDecisionMakers

Janice Allen
Janice Allen

Janice has been with businesskinda for 5 years, writing copy for client websites, blog posts, EDMs and other mediums to engage readers and encourage action. By collaborating with clients, our SEO manager and the wider businesskinda team, Janice seeks to understand an audience before creating memorable, persuasive copy.

ShareTweetPin

Related Posts

Technology

Ensuring Data Security: A Comprehensive Backup for VMware

September 8, 2023
The Benefits of Outdoor Pods
Technology

The Benefits of Outdoor Pods

September 5, 2023
Twitter is renamed X
Technology

Twitter is renamed X

July 24, 2023
Apple is expanding protection against unsolicited nude photos in iOS 17
Technology

Apple’s GPT chatbot is already in use internally

July 23, 2023
Star Trek: Strange New Worlds is getting a musical episode following the announcement of SDCC 2023
Technology

Star Trek: Strange New Worlds is getting a musical episode following the announcement of SDCC 2023

July 23, 2023
The Sony Project Q PlayStation handheld runs Android in a leaked video
Technology

The Sony Project Q PlayStation handheld runs Android in a leaked video

July 22, 2023
Next Post
Daniel Winnik- Wiki, Age, Height, Girlfriend, Net Worth, Ethnicity, Career

Daniel Winnik- Wiki, Age, Height, Girlfriend, Net Worth, Ethnicity, Career

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

  • 129 Followers
  • 952 Subscribers
  • 30 Followers

Recommended

Where is Judie Aronson now?  Net worth, husband, biography

Where is Judie Aronson now? Net worth, husband, biography

August 2, 2022
How option prices can help predict future stock prices

How option prices can help predict future stock prices

August 21, 2022
6 principles • businesskinda.com

6 principles • businesskinda.com

November 25, 2022
The pros and cons of 3 common startup founder personality traits

The pros and cons of 3 common startup founder personality traits

September 28, 2022
Antoine Semenyo – Wiki, age, girlfriend, net worth, height, career

Antoine Semenyo – Wiki, age, girlfriend, net worth, height, career

December 3, 2022
Start your development journey in the world of Web3

Start your development journey in the world of Web3

August 6, 2022
  • Write For US
  • Privacy Policy
  • About Us
  • Contact Us

DISCLAIMER
We are the Simple News website that provide awareness & support to readers. If you find any news or article that belong's to you then contact us anytime, we will remove that things or give you best credit that help for your brand.
© 2022 businesskinda.com.

No Result
View All Result
  • Home
  • Reviews
  • How To
  • Write For US
  • Business
  • Marketing
  • Startups
  • Technology

@ Copyright businesskinda.com