arXiv subjects

Mingda Zhang

Publications and source records attributed to Mingda Zhang.

41 records · Page 3Linked to original sources

arXiv2019

Cap2Det: Learning to Amplify Weak Caption Supervision for Object Detection

Learning to localize and name object instances is a fundamental problem in vision, but state-of-the-art approaches rely on expensive bounding box supervision. While weakly supervised detection (WSOD) methods relax the need for boxes to that of image-level annotations, even cheaper supervision is naturally available in the form of unstructured textual descriptions that users may freely provide when uploading image content. However, straightforward approaches to using such data for WSOD wastefully discard captions that do not exactly match object names. Instead, we show how to squeeze the most information out of these captions by training a text-only classifier that generalizes beyond dataset boundaries. Our discovery provides an opportunity for learning detection models from noisy but more abundant and freely-available caption data. We also validate our model on three classic object detection benchmarks and achieve state-of-the-art WSOD performance. Our code is available at https://github.com/yekeren/Cap2Det.

cs.CV↗

arXiv2018

Learning to discover and localize visual objects with open vocabulary

To alleviate the cost of obtaining accurate bounding boxes for training today's state-of-the-art object detection models, recent weakly supervised detection work has proposed techniques to learn from image-level labels. However, requiring discrete image-level labels is both restrictive and suboptimal. Real-world "supervision" usually consists of more unstructured text, such as captions. In this work we learn association maps between images and captions. We then use a novel objectness criterion to rank the resulting candidate boxes, such that high-ranking boxes have strong gradients along all edges. Thus, we can detect objects beyond a fixed object category vocabulary, if those objects are frequent and distinctive enough. We show that our objectness criterion improves the proposed bounding boxes in relation to prior weakly supervised detection methods. Further, we show encouraging results on object detection from image-level captions only.

cs.CV↗

arXiv2018

Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text

Images and text in advertisements interact in complex, non-literal ways. The two channels are usually complementary, with each channel telling a different part of the story. Current approaches, such as image captioning methods, only examine literal, redundant relationships, where image and text show exactly the same content. To understand more complex relationships, we first collect a dataset of advertisement interpretations for whether the image and slogan in the same visual advertisement form a parallel (conveying the same message without literally saying the same thing) or non-parallel relationship, with the help of workers recruited on Amazon Mechanical Turk. We develop a variety of features that capture the creativity of images and the specificity or ambiguity of text, as well as methods that analyze the semantics within and across channels. We show that our method outperforms standard image-text alignment approaches on predicting the parallel/non-parallel relationship between image and text.

cs.CV↗

arXiv2018

A nanophotonic all-optical diode for non-reciprocal transmission of circularly polarized lights

All optical diodes (AODs) play an important role in quantum optics and information processing, in which the information is encoded by photons. Only circularly polarized lights are able to carry the spin states of photons, which has been intensively used in quantum computing and information processing and enable new research fields, such as chiral quantum optics. An ideal AOD should be able to work with arbitrary polarizations states, including circularly polarized lights, which has not been demonstrated yet. In this paper, we theoretically demonstrate for the first time a nanophotonic AOD that is able to work with circularly polarized lights. The AOD nanostructure is based on a heterostructure of two-dimension silica and silicon photonic crystals (PhCs). By controlling the effective refractive indices of the PhCs and using an inclined interface, we are able to exploit generalized total reflection principle to achieve non-reciprocal transmission of circularly polarized lights. In addition, the nanophotonic AOD is able to achieve high forward transmittance greater than 0.6 and high contrast ratio close to 1 in a broad wavelength range of 1497 nm to 1666 nm. The designed nanophotonic AOD will find broad applications in optical quantum information processing and computing.

physics.optics↗

arXiv2017

Automatic Understanding of Image and Video Advertisements

There is more to images than their objective physical content: for example, advertisements are created to persuade a viewer to take a certain action. We propose the novel problem of automatic advertisement understanding. To enable research on this problem, we create two datasets: an image dataset of 64,832 image ads, and a video dataset of 3,477 ads. Our data contains rich annotations encompassing the topic and sentiment of the ads, questions and answers describing what actions the viewer is prompted to take and the reasoning that the ad presents to persuade the viewer ("What should I do according to this ad, and why should I do it?"), and symbolic references ads make (e.g. a dove symbolizes peace). We also analyze the most common persuasive strategies ads use, and the capabilities that computer vision systems should have to understand these strategies. We present baseline classification results for several prediction tasks, including automatically answering questions about the messages of the ads.

cs.CV↗