arXiv subjects

Fernando J. Von Zuben

Publications and source records attributed to Fernando J. Von Zuben.

15 recordsLinked to original sources

arXiv2024

Online Multi-Task Learning with Recursive Least Squares and Recursive Kernel Methods

This paper introduces two novel approaches for Online Multi-Task Learning (MTL) Regression Problems. We employ a high performance graph-based MTL formulation and develop two alternative recursive versions based on the Weighted Recursive Least Squares (WRLS) and the Online Sparse Least Squares Support Vector Regression (OSLSSVR) strategies. Adopting task-stacking transformations, we demonstrate the existence of a single matrix incorporating the relationship of multiple tasks and providing structural information to be embodied by the MT-WRLS method in its initialization procedure and by the MT-OSLSSVR in its multi-task kernel function. Contrasting the existing literature, which is mostly based on Online Gradient Descent (OGD) or cubic inexact approaches, we achieve exact and approximate recursions with quadratic per-instance cost on the dimension of the input space (MT-WRLS) or on the size of the dictionary of instances (MT-OSLSSVR). We compare our online MTL methods to other contenders in a real-world wind speed forecasting case study, evidencing the significant gain in performance of both proposed approaches.

stat.ML↗

arXiv2023

An Extension of the Non-Inferior Set Estimation Algorithm for Many Objectives

This work proposes a novel multi-objective optimization approach that globally finds a representative non-inferior set of solutions, also known as Pareto-optimal solutions, by automatically formulating and solving a sequence of weighted sum method scalarization problems. The approach is called MONISE (Many-Objective NISE) because it represents an extension of the well-known non-inferior set estimation (NISE) algorithm, which was originally conceived to deal with two-dimensional objective spaces. The proposal is endowed with the following characteristics: (1) uses a mixed-integer linear programming formulation to operate in two or more dimensions, thus properly supporting many (i.e., three or more) objectives; (2) relies on an external algorithm to solve the weighted sum method scalarization problem to optimality; and (3) creates a faithful representation of the Pareto frontier in the case of convex problems, and a useful approximation of it in the non-convex case. Moreover, when dealing specifically with two objectives, some additional properties are portrayed for the estimated non-inferior set. Experimental results validate the proposal and indicate that MONISE is competitive, in convex and non-convex (combinatorial) problems, both in terms of computational cost and the overall quality of the non-inferior set, measured by the acquired hypervolume.

math.OC↗

arXiv2023

OFA$^2$: A Multi-Objective Perspective for the Once-for-All Neural Architecture Search

Once-for-All (OFA) is a Neural Architecture Search (NAS) framework designed to address the problem of searching efficient architectures for devices with different resources constraints by decoupling the training and the searching stages. The computationally expensive process of training the OFA neural network is done only once, and then it is possible to perform multiple searches for subnetworks extracted from this trained network according to each deployment scenario. In this work we aim to give one step further in the search for efficiency by explicitly conceiving the search stage as a multi-objective optimization problem. A Pareto frontier is then populated with efficient, and already trained, neural architectures exhibiting distinct trade-offs among the conflicting objectives. This could be achieved by using any multi-objective evolutionary algorithm during the search stage, such as NSGA-II and SMS-EMOA. In other words, the neural network is trained once, the searching for subnetworks considering different hardware constraints is also done one single time, and then the user can choose a suitable neural network according to each deployment scenario. The conjugation of OFA and an explicit algorithm for multi-objective optimization opens the possibility of a posteriori decision-making in NAS, after sampling efficient subnetworks which are a very good approximation of the Pareto frontier, given that those subnetworks are already trained and ready to use. The source code and the final search algorithm will be released at https://github.com/ito-rafael/once-for-all-2

cs.NE↗

arXiv2020

New advances in enumerative biclustering algorithms with online partitioning

This paper further extends RIn-Close_CVC, a biclustering algorithm capable of performing an efficient, complete, correct and non-redundant enumeration of maximal biclusters with constant values on columns in numerical datasets. By avoiding a priori partitioning and itemization of the dataset, RIn-Close_CVC implements an online partitioning, which is demonstrated here to guide to more informative biclustering results. The improved algorithm is called RIn-Close_CVC3, keeps those attractive properties of RIn-Close_CVC, as formally proved here, and is characterized by: a drastic reduction in memory usage; a consistent gain in runtime; additional ability to handle datasets with missing values; and additional ability to operate with attributes characterized by distinct distributions or even mixed data types. The experimental results include synthetic and real-world datasets used to perform scalability and sensitivity analyses. As a practical case study, a parsimonious set of relevant and interpretable mixed-attribute-type rules is obtained in the context of supervised descriptive pattern mining.

cs.LG↗

arXiv2018

RIn-Close_CVC2: an even more efficient enumerative algorithm for biclustering of numerical datasets

RIn-Close_CVC is an efficient (take polynomial time per bicluster), complete (find all maximal biclusters), correct (all biclusters attend the user-defined level of consistency) and non-redundant (all the obtained biclusters are maximal and the same bicluster is not enumerated more than once) enumerative algorithm for mining maximal biclusters with constant values on columns in numerical datasets. Despite RIn-Close_CVC has all these outstanding properties, it has a high computational cost in terms of memory usage because it must keep a symbol table in memory to prevent a maximal bicluster to be found more than once. In this paper, we propose a new version of RIn-Close_CVC, named RIn-Close_CVC2, that does not use a symbol table to prevent redundant biclusters, and keeps all these four properties. We also prove that these algorithms actually possess these properties. Experiments are carried out with synthetic and real-world datasets to compare RIn-Close_CVC and RIn-Close_CVC2 in terms of memory usage and runtime. The experimental results show that RIn-Close_CVC2 brings a large reduction in memory usage and, in average, significant runtime gain when compared to its predecessor.

cs.LG↗

arXiv2017

Efficient mining of maximal biclusters in mixed-attribute datasets

This paper presents a novel enumerative biclustering algorithm to directly mine all maximal biclusters in mixed-attribute datasets (containing both numerical and categorical attributes), with or without missing values. The proposal is an extension of RIn-Close_CVC, which was originally conceived to mine perfect or perturbed biclusters with constant values on columns solely from numerical datasets, and without missing values. Even endowed with additional and more general features, the extended RIn-Close_CVC retains four key properties: (1) efficiency, (2) completeness, (3) correctness, and (4) non-redundancy. Our proposal is the first one to deal with mixed-attribute datasets without requiring any pre-processing step, such as discretization and itemization of real-valued attributes. This is a decisive aspect, because discretization and itemization implies a priori decisions, with information loss and no clear control over the consequences. On the other hand, even having to specify a priori an individual threshold for each numerical attribute, that will be used to indicate internal consistency per attribute, each threshold will be applied during the construction of the biclusters, shaping the peculiarities of the data distribution. We also explore the strong connection between biclustering and frequent pattern mining to (1) provide filters to select a compact bicluster set that exhibits high relevance and low redundancy, and (2) in the case of labeled datasets, automatically present the biclusters in a user-friendly and intuitive form, by means of quantitative class association rules. Our experimental results showed that the biclusters yield a parsimonious set of relevant rules, providing useful and interpretable models for five mixed-attribute labeled datasets.

cs.DB↗

arXiv2017

Spatial Projection of Multiple Climate Variables using Hierarchical Multitask Learning

Future projection of climate is typically obtained by combining outputs from multiple Earth System Models (ESMs) for several climate variables such as temperature and precipitation. While IPCC has traditionally used a simple model output average, recent work has illustrated potential advantages of using a multitask learning (MTL) framework for projections of individual climate variables. In this paper we introduce a framework for hierarchical multitask learning (HMTL) with two levels of tasks such that each super-task, i.e., task at the top level, is itself a multitask learning problem over sub-tasks. For climate projections, each super-task focuses on projections of specific climate variables spatially using an MTL formulation. For the proposed HMTL approach, a group lasso regularization is added to couple parameters across the super-tasks, which in the climate context helps exploit relationships among the behavior of different climate variables at a given spatial location. We show that some recent works on MTL based on learning task dependency structures can be viewed as special cases of HMTL. Experiments on synthetic and real climate data show that HMTL produces better results than decoupled MTL methods applied separately on the super-tasks and HMTL significantly outperforms baselines for climate projection.

cs.LG↗

arXiv2017

An Extremal Optimization approach to parallel resonance constrained capacitor placement problem

Installation of capacitors in distribution networks is one of the most used procedure to compensate reactive power generated by loads and, consequently, to reduce technical losses. So, the problem consists in identifying the optimal placement and sizing of capacitors. This problem is known in the literature as optimal capacitor placement problem. Neverthless, depending on the location and size of the capacitor, it may become a harmonic source, allowing capacitor to enter into resonance with the distribution network, causing several undesired side effects. In this work we propose a parsimonious method to deal with the capacitor placement problem that incorporates resonance constraints, ensuring that every allocated capacitor will not act as a harmonic source. This proposed algorithm is based upon a physical inspired metaheuristic known as Extremal Optimization. The results achieved showed that this proposal has reached significant gains when compared with other proposals that attempt repair, in a post-optimization stage, already obtained solutions which violate resonance constraints.

math.OC↗

arXiv2016

Effects of Social Ties in Knowledge Diffusion: case study on PLOS ONE

In order to capture the effects of social ties in knowledge diffusion, this paper examines the publication network that emerges from the collaboration of researchers, using citation information as means to estimate knowledge flow. For this purpose, we analyzed the papers published in the PLOS ONE journal finding strong evidence to support that the closer two authors are in the co-authorship network, the larger the probability that knowledge flow will occur between them. Moreover, we also found that when it comes to knowledge diffusion, strong co-authorship proximity is more determinant than geographic proximity.

cs.SI↗

arXiv2016

Online Social Network Analysis: A Survey of Research Applications in Computer Science

The emergence and popularization of online social networks suddenly made available a large amount of data from social organization, interaction and human behavior. All this information opens new perspectives and challenges to the study of social systems, being of interest to many fields. Although most online social networks are recent (less than fifteen years old), a vast amount of scientific papers was already published on this topic, dealing with a broad range of analytical methods and applications. This work describes how computational researches have approached this subject and the methods used to analyze such systems. Founded on a wide though non-exaustive review of the literature, a taxonomy is proposed to classify and describe different categories of research. Each research category is described and the main works, discoveries and perspectives are highlighted.

cs.SI↗

arXiv2016

Single-Solution Hypervolume Maximization and its use for Improving Generalization of Neural Networks

This paper introduces the hypervolume maximization with a single solution as an alternative to the mean loss minimization. The relationship between the two problems is proved through bounds on the cost function when an optimal solution to one of the problems is evaluated on the other, with a hyperparameter to control the similarity between the two problems. This same hyperparameter allows higher weight to be placed on samples with higher loss when computing the hypervolume's gradient, whose normalized version can range from the mean loss to the max loss. An experiment on MNIST with a neural network is used to validate the theory developed, showing that the hypervolume maximization can behave similarly to the mean loss minimization and can also provide better performance, resulting on a 20% reduction of the classification error on the test set.

cs.LG↗

arXiv2016

Reducing the Training Time of Neural Networks by Partitioning

This paper presents a new method for pre-training neural networks that can decrease the total training time for a neural network while maintaining the final performance, which motivates its use on deep neural networks. By partitioning the training task in multiple training subtasks with sub-models, which can be performed independently and in parallel, it is shown that the size of the sub-models reduces almost quadratically with the number of subtasks created, quickly scaling down the sub-models used for the pre-training. The sub-models are then merged to provide a pre-trained initial set of weights for the original model. The proposed method is independent of the other aspects of the training, such as architecture of the neural network, training method, and objective, making it compatible with a wide range of existing approaches. The speedup without loss of performance is validated experimentally on MNIST and on CIFAR10 data sets, also showing that even performing the subtasks sequentially can decrease the training time. Moreover, we show that larger models may present higher speedups and conjecture about the benefits of the method in distributed learning systems.

cs.NE↗

arXiv2015

Enumerating all maximal biclusters in numerical datasets

Biclustering has proved to be a powerful data analysis technique due to its wide success in various application domains. However, the existing literature presents efficient solutions only for enumerating maximal biclusters with constant values, or heuristic-based approaches which can not find all biclusters or even support the maximality of the obtained biclusters. Here, we present a general family of biclustering algorithms for enumerating all maximal biclusters with (i) constant values on rows, (ii) constant values on columns, or (iii) coherent values. Versions for perfect and for perturbed biclusters are provided. Our algorithms have four key properties (just the algorithm for perturbed biclusters with coherent values fails to exhibit the first property): they are (1) efficient (take polynomial time per pattern), (2) complete (find all maximal biclusters), (3) correct (all biclusters attend the user-defined measure of similarity), and (4) non-redundant (all the obtained biclusters are maximal and the same bicluster is not enumerated twice). They are based on a generalization of an efficient formal concept analysis algorithm called In-Close2. Experimental results point to the necessity of having efficient enumerative biclustering algorithms and provide a valuable insight into the scalability of our family of algorithms and its sensitivity to user-defined parameters.

cs.DM↗

arXiv2015

Asymmetric Distributions from Constrained Mixtures

This paper introduces constrained mixtures for continuous distributions, characterized by a mixture of distributions where each distribution has a shape similar to the base distribution and disjoint domains. This new concept is used to create generalized asymmetric versions of the Laplace and normal distributions, which are shown to define exponential families, with known conjugate priors, and to have maximum likelihood estimates for the original parameters, with known closed-form expressions. The asymmetric and symmetric normal distributions are compared in a linear regression example, showing that the asymmetric version performs at least as well as the symmetric one, and in a real world time-series problem, where a hidden Markov model is used to fit a stock index, indicating that the asymmetric version provides higher likelihood and may learn distribution models over states and transition distributions with considerably less entropy.

stat.ML↗

arXiv2014

Multi-task Sparse Structure Learning

Multi-task learning (MTL) aims to improve generalization performance by learning multiple related tasks simultaneously. While sometimes the underlying task relationship structure is known, often the structure needs to be estimated from data at hand. In this paper, we present a novel family of models for MTL, applicable to regression and classification problems, capable of learning the structure of task relationships. In particular, we consider a joint estimation problem of the task relationship structure and the individual task parameters, which is solved using alternating minimization. The task relationship structure learning component builds on recent advances in structure learning of Gaussian graphical models based on sparse estimators of the precision (inverse covariance) matrix. We illustrate the effectiveness of the proposed model on a variety of synthetic and benchmark datasets for regression and classification. We also consider the problem of combining climate model outputs for better projections of future climate, with focus on temperature in South America, and show that the proposed model outperforms several existing methods for the problem.

cs.LG↗