arXiv subjects

James Baker

Publications and source records attributed to James Baker.

8 recordsLinked to original sources

arXiv2025

MONKEY: Masking ON KEY-Value Activation Adapter for Personalization

Personalizing diffusion models allows users to generate new images that incorporate a given subject, allowing more control than a text prompt. These models often suffer somewhat when they end up just recreating the subject image and ignoring the text prompt. We observe that one popular method for personalization, IP-Adapter, automatically generates masks that segment the subject from the background during inference. We propose to use this automatically generated mask on a second pass to mask the image tokens, thus restricting them to the subject, not the background, allowing the text prompt to attend to the rest of the image. For text prompts describing locations and places, this produces images that accurately depict the subject while definitively matching the prompt. We compare our method to a few other test time personalization methods, and find our method displays high prompt and source image alignment. We also perform a user study to validate whether end users would appreciate our method. Code available at https://github.com/jamesBaker361/monkey

cs.CV↗

arXiv2024

Style Ambiguity Loss Using CLIP

In this work, we explore using the style ambiguity training objective, originally used to approximate creativity, on a diffusion model. However, this objective requires the use of a pretrained classifier and a labeled dataset. We introduce new forms of style ambiguity loss that do not require training a new classifier or a labeled dataset. Instead of using a classifier, we generate centroids in the CLIP embedding space, and images are classified based on their relative distance to said centroids. We find the centroids via K-means clustering of an unlabeled dataset, as well as using text labels to generate CLIP embeddings, to be used as centroids. Code is available at https://github.com/jamesBaker361/clipcreate

cs.CV↗

arXiv2024

BRAT: Bonus oRthogonAl Token for Architecture Agnostic Textual Inversion

Textual Inversion remains a popular method for personalizing diffusion models, in order to teach models new subjects and styles. We note that textual inversion has been underexplored using alternatives to the UNet, and experiment with textual inversion with a vision transformer. We also seek to optimize textual inversion using a strategy that does not require explicit use of the UNet and its idiosyncratic layers, so we add bonus tokens and enforce orthogonality. We find the use of the bonus token improves adherence to the source images and the use of the vision transformer improves adherence to the prompt. Code is available at https://github.com/jamesBaker361/tex_inv_plus.

cs.CV↗

arXiv2024

Using Multimodal Foundation Models and Clustering for Improved Style Ambiguity Loss

Teaching text-to-image models to be creative involves using style ambiguity loss, which requires a pretrained classifier. In this work, we explore a new form of the style ambiguity training objective, used to approximate creativity, that does not require training a classifier or even a labeled dataset. We then train a diffusion model to maximize style ambiguity to imbue the diffusion model with creativity and find our new methods improve upon the traditional method, based on automated metrics for human judgment, while still maintaining creativity and novelty.

cs.CV↗

arXiv2023

ARTEMIS: Using GANs with Multiple Discriminators to Generate Art

We propose a novel method for generating abstract art. First an autoencoder is trained to encode and decode the style representations of images, which are extracted from source images with a pretrained VGG network. Then, the decoder component of the autoencoder is extracted and used as a generator in a GAN. The generator works with an ensemble of discriminators. Each discriminator takes different style representations of the same images, and the generator is trained to create images that create convincing style representations in order to deceive all of the generators. The generator is also trained to maximize a diversity term. The resulting images had a surreal, geometric quality. We call our approach ARTEMIS (ARTistic Encoder- Multi- Discriminators Including Self-Attention), as it uses the self-attention layers and an encoder-decoder architecture.

cs.CV↗

arXiv2023

The Heat is On: Thermal Facial Landmark Tracking

Facial landmark tracking for thermal images requires tracking certain important regions of subjects' faces, using images from thermal images, which omit lighting and shading, but show the temperatures of their subjects. The fluctuations of heat in particular places reflect physiological changes like bloodflow and perspiration, which can be used to remotely gauge things like anxiety and excitement. Past work in this domain has been limited to only a very limited set of architectures and techniques. This work goes further by trying a comprehensive suit of various models with different components, such as residual connections, channel and feature-wise attention, as well as the practice of ensembling components of the network to work in parallel. The best model integrated convolutional and residual layers followed by a channel-wise self-attention layer, requiring less than 100K parameters.

cs.CV↗

arXiv2023

In the Red(dit): Social Media and Stock Prices

Spearheaded by retail traders on the website reddit, the GameStop short squeeze of early 2021 shows that social media embeds information that correlates with market movements. This paper seeks to examine this relationship by using daily frequencies of classified comments and buzzwords as additional factors in a Fama-French three factor model. Comments are classified using an unsupervised clustering method, while past studies have used pretrained models that are not specific to the domains being studied.

cs.SI↗

arXiv2021

The effect of grain size on erosion and entrainment in dry granular flows

The entrainment of underlying erodible material by geophysical flows can significantly boost the flowing mass and increase the final deposition extent. The particle size of both the flowing material and the erodible substrate influence the entrainment mechanism and determine the overall flow dynamics. This paper examines these mechanisms experimentally by considering the flow of particles over an erodible bed using different particle size combinations for the incoming flow and the base layer in a laboratory-scale inclined flume. Dynamic X-ray radiography was used to capture the dynamics of the flow-erodible bed interface. The experiments found that the maximum downslope velocity depends on the ratio between the size of the flowing particles and the size of the bed particles, with higher ratios leading to faster velocities. Two techniques were then applied to estimate the evolving erosion depth: an established critical velocity method, and a novel particle-size-based method. Erosion rates were estimated from both of these methods. Interestingly, these two rates express different and contradictory conclusions. In the critical-velocity-based rate estimation, the normalized erosion rate increases with the flow to bed grain size ratio, whereas the erosion rates estimated from the particle-size-based approach find the opposite trend. We rationalise this discrepancy by considering the physical interpretation of both measurement methods, and provide insight into how future modelling can be performed to accommodate both of these complementary measures. This paper highlights how the erosion rate is entirely dependent on the method of estimating the erosion depth and the choice of measurement technique.

physics.geo-ph↗