Monitoring Classification Models

Long running and high volume usage of the simMachines server for classification has production-level monitoring of predictions that can be implemented. This feature allows production administrators to monitor and receive email alerts when statistical analysis of classification results varies significantly from typical behavior. This feature applies only for classification models (simClassify and simClassify+).

For each class within a model, a monitor rule can be created. If the percentage for the predicted class changes substantially, according to the parameters set in the monitor page, a notification is generated. There are several parameters that can be input by the user to control how the monitor is created. This functionality can be accessed in the model action: Monitor Model.

Here's the parameters for creating a monitor:

Class Name	The class to monitor.
Packet Size	The calculations for monitoring are performed over packets. A packet is a set of queries. This parameter specifies how many queries are in a packet.
Sample Size	Determines how many predictions for the specified class must occur before monitor activation.
Percentage	The minimum percentage of packets that have to be marked as outliers to trigger a monitor warning.
Revision Frequency	Specifies how often, in milliseconds, the monitoring report will run.
zScore	Specifies how many standard deviations the accepted range is from the mean. The higher it is, the higher is the range where the packets are considered in range.
Email Address	Email to receive the email notification when a monitoring warning is generated.

The current Monitor rules can be visualized and filtered according to the class for which the rule was created.

The monitor will be constantly inspecting queries done to the model. Each "out-of-bound" status per class is stored in the database.