arXiv subjects

Ke Yu

Publications and source records attributed to Ke Yu.

At least 55 records · Page 3Linked to original sources

arXiv2022

Anatomy-Guided Weakly-Supervised Abnormality Localization in Chest X-rays

Creating a large-scale dataset of abnormality annotation on medical images is a labor-intensive and costly task. Leveraging weak supervision from readily available data such as radiology reports can compensate lack of large-scale data for anomaly detection methods. However, most of the current methods only use image-level pathological observations, failing to utilize the relevant anatomy mentions in reports. Furthermore, Natural Language Processing (NLP)-mined weak labels are noisy due to label sparsity and linguistic ambiguity. We propose an Anatomy-Guided chest X-ray Network (AGXNet) to address these issues of weak annotation. Our framework consists of a cascade of two networks, one responsible for identifying anatomical abnormalities and the second responsible for pathological observations. The critical component in our framework is an anatomy-guided attention module that aids the downstream observation network in focusing on the relevant anatomical regions generated by the anatomy network. We use Positive Unlabeled (PU) learning to account for the fact that lack of mention does not necessarily mean a negative label. Our quantitative and qualitative results on the MIMIC-CXR dataset demonstrate the effectiveness of AGXNet in disease and anatomical abnormality localization. Experiments on the NIH Chest X-ray dataset show that the learned feature representations are transferable and can achieve the state-of-the-art performances in disease classification and competitive disease localization results. Our code is available at https://github.com/batmanlab/AGXNet

cs.CV↗

arXiv2021

Can contrastive learning avoid shortcut solutions?

The generalization of representations learned via contrastive learning depends crucially on what features of the data are extracted. However, we observe that the contrastive loss does not always sufficiently guide which features are extracted, a behavior that can negatively impact the performance on downstream tasks via "shortcuts", i.e., by inadvertently suppressing important predictive features. We find that feature extraction is influenced by the difficulty of the so-called instance discrimination task (i.e., the task of discriminating pairs of similar points from pairs of dissimilar ones). Although harder pairs improve the representation of some features, the improvement comes at the cost of suppressing previously well represented features. In response, we propose implicit feature modification (IFM), a method for altering positive and negative samples in order to guide contrastive models towards capturing a wider variety of predictive features. Empirically, we observe that IFM reduces feature suppression, and as a result improves performance on vision and medical imaging tasks. The code is available at: \url{https://github.com/joshr17/IFM}.

cs.LG↗

arXiv2021

ReconfigISP: Reconfigurable Camera Image Processing Pipeline

Image Signal Processor (ISP) is a crucial component in digital cameras that transforms sensor signals into images for us to perceive and understand. Existing ISP designs always adopt a fixed architecture, e.g., several sequential modules connected in a rigid order. Such a fixed ISP architecture may be suboptimal for real-world applications, where camera sensors, scenes and tasks are diverse. In this study, we propose a novel Reconfigurable ISP (ReconfigISP) whose architecture and parameters can be automatically tailored to specific data and tasks. In particular, we implement several ISP modules, and enable backpropagation for each module by training a differentiable proxy, hence allowing us to leverage the popular differentiable neural architecture search and effectively search for the optimal ISP architecture. A proxy tuning mechanism is adopted to maintain the accuracy of proxy networks in all cases. Extensive experiments conducted on image restoration and object detection, with different sensors, light conditions and efficiency constraints, validate the effectiveness of ReconfigISP. Only hundreds of parameters need tuning for every task.

eess.IV↗

arXiv2021

Path-Restore: Learning Network Path Selection for Image Restoration

Very deep Convolutional Neural Networks (CNNs) have greatly improved the performance on various image restoration tasks. However, this comes at a price of increasing computational burden, hence limiting their practical usages. We observe that some corrupted image regions are inherently easier to restore than others since the distortion and content vary within an image. To leverage this, we propose Path-Restore, a multi-path CNN with a pathfinder that can dynamically select an appropriate route for each image region. We train the pathfinder using reinforcement learning with a difficulty-regulated reward. This reward is related to the performance, complexity and "the difficulty of restoring a region". A policy mask is further investigated to jointly process all the image regions. We conduct experiments on denoising and mixed restoration tasks. The results show that our method achieves comparable or superior performance to existing approaches with less computational cost. In particular, Path-Restore is effective for real-world denoising, where the noise distribution varies across different regions on a single image. Compared to the state-of-the-art RIDNet, our method achieves comparable performance and runs 2.7x faster on the realistic Darmstadt Noise Dataset.

cs.CV↗

arXiv2021

BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond

Video super-resolution (VSR) approaches tend to have more components than the image counterparts as they need to exploit the additional temporal dimension. Complex designs are not uncommon. In this study, we wish to untangle the knots and reconsider some most essential components for VSR guided by four basic functionalities, i.e., Propagation, Alignment, Aggregation, and Upsampling. By reusing some existing components added with minimal redesigns, we show a succinct pipeline, BasicVSR, that achieves appealing improvements in terms of speed and restoration quality in comparison to many state-of-the-art algorithms. We conduct systematic analysis to explain how such gain can be obtained and discuss the pitfalls. We further show the extensibility of BasicVSR by presenting an information-refill mechanism and a coupled propagation scheme to facilitate information aggregation. The BasicVSR and its extension, IconVSR, can serve as strong baselines for future VSR approaches.

cs.CV↗

arXiv2020

Context Matters: Graph-based Self-supervised Representation Learning for Medical Images

Supervised learning method requires a large volume of annotated datasets. Collecting such datasets is time-consuming and expensive. Until now, very few annotated COVID-19 imaging datasets are available. Although self-supervised learning enables us to bootstrap the training by exploiting unlabeled data, the generic self-supervised methods for natural images do not sufficiently incorporate the context. For medical images, a desirable method should be sensitive enough to detect deviation from normal-appearing tissue of each anatomical region; here, anatomy is the context. We introduce a novel approach with two levels of self-supervised representation learning objectives: one on the regional anatomical level and another on the patient-level. We use graph neural networks to incorporate the relationship between different anatomical regions. The structure of the graph is informed by anatomical correspondences between each patient and an anatomical atlas. In addition, the graph representation has the advantage of handling any arbitrarily sized image in full resolution. Experiments on large-scale Computer Tomography (CT) datasets of lung images show that our approach compares favorably to baseline methods that do not account for the context. We use the learnt embedding to quantify the clinical progression of COVID-19 and show that our method generalizes well to COVID-19 patients from different hospitals. Qualitative results suggest that our model can identify clinically relevant regions in the images.

eess.IV↗

arXiv2020

Multi-resolution lattice Green's function method for incompressible flows

We propose a multi-resolution strategy that is compatible with the lattice Green's function (LGF) technique for solving viscous, incompressible flows on unbounded domains. The LGF method exploits the regularity of a finite-volume scheme on a formally unbounded Cartesian mesh to yield robust and computationally efficient solutions. The original method is spatially adaptive, but challenging to integrate with embedded mesh refinement as the underlying LGF is only defined for a fixed resolution. We present an ansatz for adaptive mesh refinement, where the solutions to the pressure Poisson equation are approximated using the LGF technique on a composite mesh constructed from a series of infinite lattices of differing resolution. To solve the incompressible Navier-Stokes equations, this is further combined with an integrating factor for the viscous terms and an appropriate Runge-Kutta scheme for the resulting differential-algebraic equations. The parallelized algorithm is verified through with numerical simulations of vortex rings, and the collision of vortex rings at high Reynolds number is simulated to demonstrate the reduction in computational cells achievable with both spatial and refinement adaptivity.

physics.comp-ph↗

arXiv2020

Dynamics and decay of a spherical region of turbulence in free space

We perform direct numerical simulation (DNS) and large eddy simulation (LES) of an initially spherical region of turbulence evolving in free space. The computations are performed with a lattice Green's function method, which allows the exact free-space boundary conditions to be imposed on a compact vortical region. LES simulations are conducted with the stretched vortex sub-grid stress model. The initial condition is spherically windowed, isotropic homogeneous incompressible turbulence. We study the spectrum and statistics of the decaying turbulence and compare the results with decaying isotropic turbulence, including cases representing different low wavenumber behavior of the energy spectrum (i.e. k^2 versus k^4). At late times the turbulent sphere expands with both mean radius and integral scale showing similar time-wise growth exponents. The low wavenumber behavior has little effect on the inertial scales, and we find that decay rates follow Saffman (1967) predictions in both cases, at least until about 400 initial eddy turnover times. The boundary of the spherical region develops intermittency and features ejections of vortex rings. These are shown to occur at the integral scale of the initial turbulence field and are hypothesized to occur due to a local imbalance of impulse on this scale.

physics.flu-dyn↗

arXiv2020

Understanding Deformable Alignment in Video Super-Resolution

Deformable convolution, originally proposed for the adaptation to geometric variations of objects, has recently shown compelling performance in aligning multiple frames and is increasingly adopted for video super-resolution. Despite its remarkable performance, its underlying mechanism for alignment remains unclear. In this study, we carefully investigate the relation between deformable alignment and the classic flow-based alignment. We show that deformable convolution can be decomposed into a combination of spatial warping and convolution. This decomposition reveals the commonality of deformable alignment and flow-based alignment in formulation, but with a key difference in their offset diversity. We further demonstrate through experiments that the increased diversity in deformable alignment yields better-aligned features, and hence significantly improves the quality of video super-resolution output. Based on our observations, we propose an offset-fidelity loss that guides the offset learning with optical flow. Experiments show that our loss successfully avoids the overflow of offsets and alleviates the instability problem of deformable alignment. Aside from the contributions to deformable alignment, our formulation inspires a more flexible approach to introduce offset diversity to flow-based alignment, improving its performance.

cs.CV↗

arXiv2020

Spectroscopic Observations of High-speed Downflows in a C1.7 Solar Flare

In this paper, we analyze the high-resolution UV spectra for a C1.7 solar flare (SOL2017-09-09T06:51) observed by the \textit{Interface Region Imaging Spectrograph} (\textit{IRIS}). {We focus on the spectroscopic observations at the locations where the cool lines of \ion{Si}{4} 1402.8 Å ($\sim$10$^{4.8}$ K) and \ion{C}{2} 1334.5/1335.7 Å ($\sim$10$^{4.4}$ K) reveal significant redshifts with Doppler velocities up to $\sim$150 km s$^{-1}$.} These redshifts appear in the rise phase of the flare, then increase rapidly, reach the maximum in a few minutes, and proceed into the decay phase. Combining the images from \textit{IRIS} and Atmospheric Imaging Assembly (AIA) on board the {\em Solar Dynamics Observatory} ({\em SDO}), we propose that the redshifts in the cool lines are caused by the downflows in the transition region and upper chromospheric layers, which likely result from a magnetic reconnection leading to the flare. In addition, the cool \ion{Si}{4} and \ion{C}{2} lines show gentle redshifts (a few tens of km s$^{-1}$) at some other locations, which manifest some distinct features from the above locations. This is supposed to originate from a different physical process.

astro-ph.SR↗

arXiv2020

Semi-Supervised Hierarchical Drug Embedding in Hyperbolic Space

Learning accurate drug representation is essential for tasks such as computational drug repositioning and prediction of drug side-effects. A drug hierarchy is a valuable source that encodes human knowledge of drug relations in a tree-like structure where drugs that act on the same organs, treat the same disease, or bind to the same biological target are grouped together. However, its utility in learning drug representations has not yet been explored, and currently described drug representations cannot place novel molecules in a drug hierarchy. Here, we develop a semi-supervised drug embedding that incorporates two sources of information: (1) underlying chemical grammar that is inferred from molecular structures of drugs and drug-like molecules (unsupervised), and (2) hierarchical relations that are encoded in an expert-crafted hierarchy of approved drugs (supervised). We use the Variational Auto-Encoder (VAE) framework to encode the chemical structures of molecules and use the knowledge-based drug-drug similarity to induce the clustering of drugs in hyperbolic space. The hyperbolic space is amenable for encoding hierarchical concepts. Both quantitative and qualitative results support that the learned drug embedding can accurately reproduce the chemical structure and induce the hierarchical relations among drugs. Furthermore, our approach can infer the pharmacological properties of novel molecules by retrieving similar drugs from the embedding space. We demonstrate that the learned drug embedding can be used to find new uses for existing drugs and to discover side-effects. We show that it significantly outperforms baselines in both tasks.

cs.LG↗

arXiv2020

IRIS Si IV Line Profiles at Flare Ribbons as Indications of Chromospheric Condensation

We present temporal variations of the Si IV line profiles at the flare ribbons in three solar flares observed by the Interface Region Imaging Spectrograph (IRIS). In the M1.1 flare on 2014 September 6 and the X1.6 flare on 2014 September 10, the Si IV line profiles evolve from wholly redshifted to red-wing enhanced with the flare development. However, in the B1.8 flare on 2016 December 2, the Si IV line profiles are wholly redshifted throughout the flare evolution. We fit the wholly redshifted line profiles with a single Gaussian function but the red-asymmetric ones with a double Gaussian function to deduce the corresponding Doppler velocities. In addition, we find that hard X-ray emission above 25 keV shows up in the two large flares, implying a nonthermal electron beam heating. In the microflare, there only appears weak hard X-ray emission up to 12 keV, indicative of a thermal heating mostly. We interpret the redshifts or red asymmetries of the Si IV line at the ribbons in the three flares as spectral manifestations of chromospheric condensation. We propose that whether the line appears to be wholly redshifted or red-asymmetric depends on the heating mechanisms and also on the propagation of the condensation.

astro-ph.SR↗

arXiv2019

A fast multi-resolution lattice Green's function method for elliptic difference equations

We propose a mesh refinement technique for solving elliptic difference equations on unbounded domains based on the fast lattice Green's function (FLGF) method. The FLGF method exploits the regularity of the Cartesian mesh and uses the fast multipole method in conjunction with fast Fourier transforms to yield linear complexity and decrease time-to-solution. We extend this method to a multi-resolution scheme and allow for locally refined Cartesian blocks embedded in the computational domain. Appropriately chosen interpolation and regularization operators retain consistency between the discrete Laplace operator and its inverse on the unbounded domain. Second-order accuracy and linear complexity are maintained, while significantly reducing the number of degrees of freedom and hence the computational cost.

physics.comp-ph↗

arXiv2019

EDVR: Video Restoration with Enhanced Deformable Convolutional Networks

Video restoration tasks, including super-resolution, deblurring, etc, are drawing increasing attention in the computer vision community. A challenging benchmark named REDS is released in the NTIRE19 Challenge. This new benchmark challenges existing methods from two aspects: (1) how to align multiple frames given large motions, and (2) how to effectively fuse different frames with diverse motion and blur. In this work, we propose a novel Video Restoration framework with Enhanced Deformable networks, termed EDVR, to address these challenges. First, to handle large motions, we devise a Pyramid, Cascading and Deformable (PCD) alignment module, in which frame alignment is done at the feature level using deformable convolutions in a coarse-to-fine manner. Second, we propose a Temporal and Spatial Attention (TSA) fusion module, in which attention is applied both temporally and spatially, so as to emphasize important features for subsequent restoration. Thanks to these modules, our EDVR wins the champions and outperforms the second place by a large margin in all four tracks in the NTIRE19 video restoration and enhancement challenges. EDVR also demonstrates superior performance to state-of-the-art published methods on video super-resolution and deblurring. The code is available at https://github.com/xinntao/EDVR.

cs.CV↗

arXiv2018

Deep Network Interpolation for Continuous Imagery Effect Transition

Deep convolutional neural network has demonstrated its capability of learning a deterministic mapping for the desired imagery effect. However, the large variety of user flavors motivates the possibility of continuous transition among different output effects. Unlike existing methods that require a specific design to achieve one particular transition (e.g., style transfer), we propose a simple yet universal approach to attain a smooth control of diverse imagery effects in many low-level vision tasks, including image restoration, image-to-image translation, and style transfer. Specifically, our method, namely Deep Network Interpolation (DNI), applies linear interpolation in the parameter space of two or more correlated networks. A smooth control of imagery effects can be achieved by tweaking the interpolation coefficients. In addition to DNI and its broad applications, we also investigate the mechanism of network interpolation from the perspective of learned filters.

cs.CV↗

arXiv2018

ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks

The Super-Resolution Generative Adversarial Network (SRGAN) is a seminal work that is capable of generating realistic textures during single image super-resolution. However, the hallucinated details are often accompanied with unpleasant artifacts. To further enhance the visual quality, we thoroughly study three key components of SRGAN - network architecture, adversarial loss and perceptual loss, and improve each of them to derive an Enhanced SRGAN (ESRGAN). In particular, we introduce the Residual-in-Residual Dense Block (RRDB) without batch normalization as the basic network building unit. Moreover, we borrow the idea from relativistic GAN to let the discriminator predict relative realness instead of the absolute value. Finally, we improve the perceptual loss by using the features before activation, which could provide stronger supervision for brightness consistency and texture recovery. Benefiting from these improvements, the proposed ESRGAN achieves consistently better visual quality with more realistic and natural textures than SRGAN and won the first place in the PIRM2018-SR Challenge. The code is available at https://github.com/xinntao/ESRGAN .

cs.CV↗

arXiv2018

Improving On-policy Learning with Statistical Reward Accumulation

Deep reinforcement learning has obtained significant breakthroughs in recent years. Most methods in deep-RL achieve good results via the maximization of the reward signal provided by the environment, typically in the form of discounted cumulative returns. Such reward signals represent the immediate feedback of a particular action performed by an agent. However, tasks with sparse reward signals are still challenging to on-policy methods. In this paper, we introduce an effective characterization of past reward statistics (which can be seen as long-term feedback signals) to supplement this immediate reward feedback. In particular, value functions are learned with multi-critics supervision, enabling complex value functions to be more easily approximated in on-policy learning, even when the reward signals are sparse. We also introduce a novel exploration mechanism called "hot-wiring" that can give a boost to seemingly trapped agents. We demonstrate the effectiveness of our advantage actor multi-critic (A2MC) method across the discrete domains in Atari games as well as continuous domains in the MuJoCo environments. A video demo is provided at https://youtu.be/zBmpf3Yz8tc.

cs.LG↗

arXiv2018

Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning

We investigate a novel approach for image restoration by reinforcement learning. Unlike existing studies that mostly train a single large network for a specialized task, we prepare a toolbox consisting of small-scale convolutional networks of different complexities and specialized in different tasks. Our method, RL-Restore, then learns a policy to select appropriate tools from the toolbox to progressively restore the quality of a corrupted image. We formulate a step-wise reward function proportional to how well the image is restored at each step to learn the action policy. We also devise a joint learning scheme to train the agent and tools for better performance in handling uncertainty. In comparison to conventional human-designed networks, RL-Restore is capable of restoring images corrupted with complex and unknown distortions in a more parameter-efficient manner using the dynamically formed toolchain.

cs.CV↗