(2020 - Now) Overparametrization, generalization and robustness

Along the last decade, machine learning underwent many breakthroughs, usually led by empirical improvements that allowed very large models to perform well and attain state-of-the-art performance on tasks ranging from image classification to playing online games. These models, however, often have millions or, even, billions of parameters. Parsimoniousness is a key principle in the philosophy behind science and engineering and it is often pursued in popular machine learning and statistical methods (ideas about pruning, distilling or building sparse neural networks are popular as it is, in a more classical setting, the idea of selecting a subset of regressors). The traditional bias-variance trade-off can be thought of as a formalization of this principle in probabilistic terms. However, as the bias-variance trade-off gets revisited, with the study of overparametrized models and the observation of a second descent in the risk as we increase the model capacity beyond the point of (almost) perfectly fitting the training data, one might wonder if there are other instances of how the models are evaluated where parsimonious models are still the best alternative. My most recent line of research focus on the interplay between overparametrization and generalization studying how different overparamentrized models generalize. With two special focus: the first is how dynamic models generalize and the second is how robust to out-of-distribution scenarios overparametrized models are. Indeed, robustness is a fundamental goal in machine learning. In many situations, the data distribution observed in using the final model is not exactly the same as the one used to develop the model. Furthermore, the ability to generalize to out-of-distribution scenarios is an important capability for the successful deployment of these models. Indeed, it has recently been observed that the interplay between overparametrization and robustness might be a key question for the deployment of modern machine learning models.

Selected publications:
  • Beyond Occam's Razor in System Identification: Double-Descent when Modeling Dynamics (2021). Proceedings of the 19th IFAC Symposium on System Identification (SYSID) - IFAC-PapersOnLine. Antônio H. Ribeiro, Johannes N. Hendriks, Adrian G. Wills, Thomas B. Schön
  • Overparametrized Regression Under L2 Adversarial Attacks (2021). Workshop on the Theory of Overparameterized Machine Learning (TOPML). Antonio H Ribeiro, Thomas B Schön
  • Beyond Occam's Razor in System Identification: Double-Descent when Modeling Dynamics (2021). Workshop on Nonlinear System Identification. Antônio H. Ribeiro, Johannes N. Hendriks, Adrian G. Wills, Thomas B. Schön

(2018 - Now) Artificial inteligence for ECG automatic analysis

The electrocardiogram is a medical test used to measure cardiac electrical activity, being the most common complementary exam to assess the heart condition. My current research leverages the latest developments in machine learning for extending and improving how ECG is used in clinical practice. My involvement with ECG analysis started in 2018 when I joined the research group CODE (Clinical Outcomes in Electrocardiography) at Universidade Federal de Minas Gerais. Using a dataset with more then 2 million entries collected and annotated between 2010 to 2017 by the Telehealth Network of Minas Gerais we developed a system capable of detecting some ECG abnormalities with greater accuracy then cardiology residents. After this successful proof of concept project, I am now involved on diverse projects using AI for ECG analysis including: the prediction of age from the ECG and its use for improving risk assessment; the creation of tools to detect Atrial Fibrillation before its development; and, tools for detecting non-ST segment elevation myocardial infarction. Now, working not only with data from Brazil, but also, since 2021, with the Clinical Epidemiology Unit in Uppsala University to power new AI applications for ECG analysis using more than one million ECG collected in Stockholm region between 2007 and 2016.

Selected publications:
  • Artificial Intelligence-Based ECG Diagnosis of Myocardial Infarction in High-Risk Emergency Department Patients (2021). SSRN. Stefan Gustafsson, Daniel Gedon, Erik Lampa, Antônio H. Ribeiro, Martin J. Holzmann, Thomas B. Schön, Johan Sundstrom
  • Deep neural network estimated electrocardiographic-age as a mortality predictor (2021). Nature Communications. Emilly M. Lima, Antônio H. Ribeiro, Gabriela MM Paixão, Manoel Horta Ribeiro, Marcelo M. Pinto Filho, Paulo R. Gomes, Derick M. Oliveira, Ester C. Sabino, Bruce B. Duncan, Luana Giatti, Sandhi M. Barreto, Wagner Meira, Thomas B. Schön, Antonio Luiz P. Ribeiro
  • Atrial fibrillation risk prediction from the 12-lead ECG using digital biomarkers and deep representation learning (2021). European Heart Journal - Digital Health. Shany Biton, Sheina Gendelman, Antônio H Ribeiro, Gabriela Miana, Carla Moreira, Antonio Luiz P Ribeiro, Joachim A Behar
  • Contextualized Interpretable Machine Learning for Medical Diagnosis (2020). Communications of the ACM. Wagner Meira Jr, Antonio L. P. Ribeiro, Derick M. Oliveira, Antonio H. Ribeiro
  • Evaluation of Mortality in Atrial Fibrillation: Clinical Outcomes in Digital Electrocardiography (CODE) Study (2020). Global Heart. Gabriela M. M. Paixão, Luis Gustavo S. Silva, Paulo R. Gomes, Emilly M. Lima, Milton P. F. Ferreira, Derick M. Oliveira, Manoel H. Ribeiro, Antonio H. Ribeiro, Jamil S. Nascimento, Jéssica A. Canazart, Leonardo B. Ribeiro, Emelia J. Benjamin, Peter W. Macfarlane, Milena S. Marcolino, Antonio L. Ribeiro
  • Automatic diagnosis of the 12-lead ECG using a deep neural network (2020). Nature Communications. Antônio H. Ribeiro, Manoel Horta Ribeiro, Gabriela M. M. Paixão, Derick M. Oliveira, Paulo R. Gomes, Jéssica A. Canazart, Milton P. S. Ferreira, Carl R. Andersson, Peter W. Macfarlane, Wagner Meira Jr., Thomas B. Schön, Antonio Luiz P. Ribeiro
  • Evaluation of mortality in bundle branch block patients from an electronic cohort: Clinical Outcomes in Digital Electrocardiography (CODE) study (2019). Journal of Electrocardiology. Gabriela M. M. Paixão, Emilly M. Lima, Paulo R. Gomes, Milton P. Ferreira, Derick M. Oliveira, Manoel Horta Ribeiro, Antônio H. Ribeiro, Jamil Nascimento, Jéssica A. Canazart, Gustavo Cardoso, Leonardo B. Ribeiro, Antonio Luiz P. Ribeiro

(2017 - Now) Learning nonlinear models for dynamical systems

Mathematical models of dynamic systems are of fundamental importance for science and engineering. Linear models of dynamic systems are well understood and are the standard choice for most engineering applications. However, pushing the forefront of current engineering technology requires us to better understand, model, and control nonlinear systems. Applications range from using flexible wings for building more efficient airplanes to using amplifiers near the saturation region to meet the needs of the future generation of wireless communication. My research focus on analysing and developing new methods for identifying dynamical systems and learning its behavior from empirical data. This was the theme of my Master's and Ph.D. Thesis and a topic of current research.

Selected publications:
  • Beyond Occam's Razor in System Identification: Double-Descent when Modeling Dynamics (2021). Proceedings of the 19th IFAC Symposium on System Identification (SYSID) - IFAC-PapersOnLine. Antônio H. Ribeiro, Johannes N. Hendriks, Adrian G. Wills, Thomas B. Schön
  • Deep Energy-Based NARX Models (2021). Proceedings of the 19th IFAC Symposium on System Identification (SYSID) - IFAC-PapersOnLine. Johannes N. Hendriks, Fredrik K. Gustafsson, Antônio H. Ribeiro, Adrian G. Wills, Thomas B. Schön
  • Beyond exploding and vanishing gradients: attractors and smoothness in the analysis of recurrent neural network training (2020). Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR. Antônio H. Ribeiro, Koen Tiels, Luis A. Aguirre, Thomas B. Schön
  • Deep Convolutional Networks in System Identification (2019). Proceedings of the 58th IEEE Conference on Decision and Control (CDC). Carl Andersson, Antônio H. Ribeiro, Koen Tiels, Niklas Wahlström, Thomas B. Schön
  • Learning nonlinear differentiable models for signals and systems: with applications (2020). . Antônio H. Ribeiro
  • On the smoothness of nonlinear system identification (2020). Automatica. Antônio H. Ribeiro, Koen Tiels, Jack Umenberger, Thomas B. Schön, Luis A. Aguirre
  • ''Parallel Training Considered Harmful?'': Comparing series-parallel and parallel feedforward network training (2018). Neurocomputing. Antônio H. Ribeiro, Luis A. Aguirre
  • Recurrent Structures in System Identification (2017). . Antônio H. Ribeiro
  • Shooting Methods for Parameter Estimation of Output Error Models (2017). Proceedings of the 20th IFAC World Congress. IFAC-PapersOnLine. Antônio H. Ribeiro, Luis A. Aguirre

(2016 - 2018) Open source algorithms for optimization and signal processing in Scipy

Scipy is a library of numerical routines implemented in the Python programming language. The library contains fundamental blocks for solving scientific problems. Several important projects mention it in their development (from the first image of a black hole to the first helicopter in Mars). My first contribution to SciPy happened while I was taken a master's class in signal processing and needed to implement a filter for removing the powerline interference from the signal of ECG exams (which was the theme of my final project). I integrated the filter design routine I used in a pull request to the SciPy signal processing library, now the function "iirnotch". Latter that same year, I contributed with an implementation of a trust-region optimization method, now the method "trust-exact". Interested in continuing contributing to SciPy, I submitted a proposal to Google Summer of Code to work on algorithms for constrained optimization in SciPy. My proposal was accepted, I was sponsored by Google and mentored by three SciPy developers (Nikolay Mayorov, Matt Haberland and Ralf Gommes) through the development of a interior point methods for solving nonlinear constrained optimization problems. The implementation was based on a similar technique to that of the commercial software KNITRO (that was described in research papers by the authors of the solver). The implementation is now integrated to SciPy trust-constr. In September 2017, because of my contributions to the library, I was invited to join the project in the core developer team and participated on the writting of the paper describing the version 1.0 of the library, published in Nature Methods.

Selected publications:
  • SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python (2020). Nature Methods. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C. J. Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul Mulbregt, SciPy 1.0 Contributors

(2015 - 2016) FPGA-based stereo vision

A stereo camera is a camera that captures two (or more images) simultaneously from different viewpoints for the purpose of estimating depth. The idea mimics our own eyes, which give us a sense of depth precisely because they capture two simultaneous images from different points of view. I worked in all stages of the development of stereo camera based on FPGA technology, from the image acquisition from an image sensor through an FPGA chip to the several steps required to estimate the depth of a scene using a stereo camera: calibration (determination of geometry and parameters of the stereo camera); rectification (alignment of the images); correspondence (determination of the corresponding points in the two images and the distance between them); and, reconstruction (calculating the position of objects by triangulation). The project was develop during my internship at Invent Vision development team. The company develops cameras and provides computer vision solutions for industry. At that point the company was commissioned to make a decoking monitoring system at an oil refinery. For this task, it was interesting to have a cloud of points containing the internal structure of the refinery drum, which sparked the company's interest in stereo vision technology. Working there, I developed a prototype of a stereo camera which is described in a monograph I wrote during my undergraduate course as requirement for obtaing the Bachelor degree in Electrical Engineering.

Selected publications:
  • Implementação de uma Câmera Estéreo (2015). . Antonio H. Ribeiro

(2013 - 2015) Modeling oil wells and Soft-Sensors

In offshore oil extraction, the instruments located at the bottom of the well undergo a rapid wear due to extreme conditions. The maintenance and exchange of such equipment is complicated and expensive. In this context, obtaining mathematical models for variables at the bottom of the oil well from platform variables is a simple and inexpensive solution. These mathematical models are also called virtual sensors. As a undergraduate student, I participated on a research and development project coordinated by professor Luis Antônio Aguirre fromm Universidade Federal de Minas Gerais (Brasil) together with Petrobras Oil Company together for the development of virtual sensors from historical data of the oil well operation. These virtual sensors could replace the measurement instrumented located at the bottom of the production column in offshore oil extraction temporarily in case of malfunctioning.

Selected publications:
  • Selecting transients automatically for the identification of models for an oil well (2015). Proceedings of the 2nd IFAC Workshop on Automatic Control in Offshore Oil and Gas Production. IFAC-PapersOnLine. Antônio H. Ribeiro, Luis A. Aguirre
  • Relaçoes Estáticas de Modelos NARX MISO e sua Representaçao de Hammerstein (2014). XX Congresso Brasileiro de Automática. Antônio H. Ribeiro, Luis A. Aguirre