Quoc Pham Le

US Patent:

20210383237, Dec 9, 2021

Filed:

Jun 3, 2021

Appl. No.:

17/337812

Inventors:

- Mountain View CA, US
Cihang Xie - Baltimore MD, US
Boqing Gong - Bellevue WA, US
Quoc V. Le - Sunnyvale CA, US

International Classification:

G06N 3/08
G06N 3/04

Abstract:

Generally, the present disclosure is directed to the training of robust neural network models by using smooth activation functions. Systems and methods according to the present disclosure may generate and/or train neural network models with improved robustness without incurring a substantial accuracy penalty and/or increased computational cost, or without any such penalty at all. For instance, in some examples, the accuracy may improve. A smooth activation function may replace an original activation function in a machine-learned model when backpropagating a loss function through the model. Optionally, one activation function may be used in the model at inference time, and a replacement activation function may be used when backpropagating a loss function through the model. The replacement activation function may be used to update learnable parameters of the model and/or to generate adversarial examples for training the model.