Submitting this abstract for an invited talk. The Laboratory for Physical Sciences has recently been conducting research in ML model uncertainty and confidence, detecting out-of-distribution data, and detecting concept drift. As we deploy ML models into operations, we must be constantly assessing whether the models are still effective and performing as expected in the current data environment. This is relevant in all cases, but especially critical in cybersecurity applications, because the data, technology, actors and behaviors are all evolving so rapidly. This talk will review several algorithmic techniques developed to address this problem.

The proposed transitory cross entropy loss function performs a weighted average of the cross entropy using both the truth labels and the predicted labels; this is a variation of the weighted cross entropy loss function that performs a weighted average using just the truth labels. We tested the transitory cross entropy loss function by training ICNet on the CityScapes dataset and saw an increase in the mean-intersection-over-union relative to the model trained using the standard weighted cross entropy loss function. We further propose modifying the weights based on dynamic performance metrics rather than just static distribution metrics.