arXiv subjects

Khalid Saifullah

Publications and source records attributed to Khalid Saifullah.

At least 19 recordsLinked to original sources

arXiv2025

DynaGuard: A Dynamic Guardian Model With User-Defined Policies

Guardian models play a crucial role in ensuring the safety and ethical behavior of user-facing AI applications by enforcing guardrails and detecting harmful content. While standard guardian models are limited to predefined, static harm categories, we introduce DynaGuard, a suite of dynamic guardian models offering novel flexibility by evaluating text based on user-defined policies, and DynaBench, a dataset for training and evaluating dynamic guardian models. Our models provide both rapid detection of policy violations and a chain-of-thought reasoning option that articulate and justify model outputs. Critically, DynaGuard not only surpasses static models in detection accuracy on traditional safety categories, but is competitive with frontier reasoning models on free-form policy violations, all in a fraction of the time. This makes DynaGuard an critical tool for language model guardrails.

cs.LG↗

arXiv2025

Exotic Lovelock black holes and extended quasitopological electromagnetism

The generalization of Birkhoff's theorem for higher dimensions in Lovelock gravity permits us to investigate the black hole solutions with horizon geometries of nonconstant curvature. We present a new class of exotic dyonic black holes in the context of Lovelock gravity and generalized quasitopological electromagnetism. First, we derive the polynomial equation that describes exotic dyonic black holes in Lovelock gravity with an arbitrary order. Next, the solutions that characterize dyonic exotic black holes of the Gauss-Bonnet and third order Lovelock gravities are worked out. Then we compute the basic thermodynamic quantities for these exotic dyonic black holes. It is also verified that these quantities satisfy the generalized first law and Smarr's relation. Furthermore, the impact of generalized quasitopological electromagnetism and topological parameters on the local stability of the resulting objects are also investigated.

gr-qc↗

arXiv2024

LiveBench: A Challenging, Contamination-Limited LLM Benchmark

Test set contamination, wherein test data from a benchmark ends up in a newer model's training set, is a well-documented obstacle for fair LLM evaluation and can quickly render benchmarks obsolete. To mitigate this, many recent benchmarks crowdsource new prompts and evaluations from human or LLM judges; however, these can introduce significant biases, and break down when scoring hard questions. In this work, we introduce a new benchmark for LLMs designed to be resistant to both test set contamination and the pitfalls of LLM judging and human crowdsourcing. We release LiveBench, the first benchmark that (1) contains frequently-updated questions from recent information sources, (2) scores answers automatically according to objective ground-truth values, and (3) contains a wide variety of challenging tasks, spanning math, coding, reasoning, language, instruction following, and data analysis. To achieve this, LiveBench contains questions that are based on recently-released math competitions, arXiv papers, news articles, and datasets, and it contains harder, contamination-limited versions of tasks from previous benchmarks such as Big-Bench Hard, AMPS, and IFEval. We evaluate many prominent closed-source models, as well as dozens of open-source models ranging from 0.5B to 405B in size. LiveBench is difficult, with top models achieving below 70% accuracy. We release all questions, code, and model answers. Questions are added and updated on a monthly basis, and we release new tasks and harder versions of tasks over time so that LiveBench can distinguish between the capabilities of LLMs as they improve in the future. We welcome community engagement and collaboration for expanding the benchmark tasks and models.

cs.CL↗

arXiv2024

CinePile: A Long Video Question Answering Dataset and Benchmark

Current datasets for long-form video understanding often fall short of providing genuine long-form comprehension challenges, as many tasks derived from these datasets can be successfully tackled by analyzing just one or a few random frames from a video. To address this issue, we present a novel dataset and benchmark, CinePile, specifically designed for authentic long-form video understanding. This paper details our innovative approach for creating a question-answer dataset, utilizing advanced LLMs with human-in-the-loop and building upon human-generated raw data. Our comprehensive dataset comprises 305,000 multiple-choice questions (MCQs), covering various visual and multimodal aspects, including temporal comprehension, understanding human-object interactions, and reasoning about events or actions within a scene. Additionally, we fine-tuned open-source Video-LLMs on the training split and evaluated both open-source and proprietary video-centric LLMs on the test split of our dataset. The findings indicate that although current models underperform compared to humans, fine-tuning these models can lead to significant improvements in their performance.

cs.CV↗

arXiv2024

Coercing LLMs to do and reveal (almost) anything

It has recently been shown that adversarial attacks on large language models (LLMs) can "jailbreak" the model into making harmful statements. In this work, we argue that the spectrum of adversarial attacks on LLMs is much larger than merely jailbreaking. We provide a broad overview of possible attack surfaces and attack goals. Based on a series of concrete examples, we discuss, categorize and systematize attacks that coerce varied unintended behaviors, such as misdirection, model control, denial-of-service, or data extraction. We analyze these attacks in controlled experiments, and find that many of them stem from the practice of pre-training LLMs with coding capabilities, as well as the continued existence of strange "glitch" tokens in common LLM vocabularies that should be removed for security reasons.

cs.LG↗

arXiv2023

Seeing in Words: Learning to Classify through Language Bottlenecks

Neural networks for computer vision extract uninterpretable features despite achieving high accuracy on benchmarks. In contrast, humans can explain their predictions using succinct and intuitive descriptions. To incorporate explainability into neural networks, we train a vision model whose feature representations are text. We show that such a model can effectively classify ImageNet images, and we discuss the challenges we encountered when training it.

cs.CV↗

arXiv2023

Bring Your Own Data! Self-Supervised Evaluation for Large Language Models

With the rise of Large Language Models (LLMs) and their ubiquitous deployment in diverse domains, measuring language model behavior on realistic data is imperative. For example, a company deploying a client-facing chatbot must ensure that the model will not respond to client requests with profanity. Current evaluations approach this problem using small, domain-specific datasets with human-curated labels. These evaluation sets are often sampled from a narrow and simplified distribution, and data sources can unknowingly be leaked into the training set which can lead to misleading evaluations. To bypass these drawbacks, we propose a framework for self-supervised evaluation of LLMs by analyzing their sensitivity or invariance to transformations on the input text. Self-supervised evaluation can directly monitor LLM behavior on datasets collected in the wild or streamed during live model deployment. We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence, in addition to sensitivity to grammatical structure and tokenization errors. When comparisons to similar human-labeled benchmarks are available, we find strong correlations between self-supervised and human-supervised evaluations. The self-supervised paradigm complements current evaluation strategies that rely on labeled data.

cs.CL↗

arXiv2023

On the Reliability of Watermarks for Large Language Models

As LLMs become commonplace, machine-generated text has the potential to flood the internet with spam, social media bots, and valueless content. Watermarking is a simple and effective strategy for mitigating such harms by enabling the detection and documentation of LLM-generated text. Yet a crucial question remains: How reliable is watermarking in realistic settings in the wild? There, watermarked text may be modified to suit a user's needs, or entirely rewritten to avoid detection. We study the robustness of watermarked text after it is re-written by humans, paraphrased by a non-watermarked LLM, or mixed into a longer hand-written document. We find that watermarks remain detectable even after human and machine paraphrasing. While these attacks dilute the strength of the watermark, paraphrases are statistically likely to leak n-grams or even longer fragments of the original text, resulting in high-confidence detections when enough tokens are observed. For example, after strong human paraphrasing the watermark is detectable after observing 800 tokens on average, when setting a 1e-5 false positive rate. We also consider a range of new detection schemes that are sensitive to short spans of watermarked text embedded inside a large document, and we compare the robustness of watermarking to other kinds of detectors.

cs.LG↗

arXiv2023

Learning UI-to-Code Reverse Generator Using Visual Critic Without Rendering

Automated reverse engineering of HTML/CSS code from UI screenshots is an important yet challenging problem with broad applications in website development and design. In this paper, we propose a novel vision-code transformer (ViCT) composed of a vision encoder processing the screenshots and a language decoder to generate the code. They are initialized by pre-trained models such as ViT/DiT and GPT-2/LLaMA but aligning the two modalities requires end-to-end finetuning, which aims to minimize the visual discrepancy between the code-rendered webpage and the original screenshot. However, the rendering is non-differentiable and causes costly overhead. We address this problem by actor-critic fine-tuning where a visual critic without rendering (ViCR) is developed to predict visual discrepancy given the original and generated code. To train and evaluate our models, we created two synthetic datasets of varying complexity, with over 75,000 unique (code, screenshot) pairs. We evaluate the UI-to-Code performance using a combination of automated metrics such as MSE, BLEU, IoU, and a novel htmlBLEU score. ViCT outperforms a strong baseline model DiT-GPT2, improving IoU from 0.64 to 0.79 and lowering MSE from 12.25 to 9.02. With much lower computational cost, it can achieve comparable performance as when using a larger decoder such as LLaMA.

cs.CV↗

arXiv2022

Power-Yang-Mills black holes and black branes in quartic quasi-topological gravity

We study higher dimensional quartic quasi-topological black holes in the framework of non-abelian power-Yang-Mills theory. It is shown that real solutions of the gravitational field equations exist only for positive values of quartic quasi-topological coefficient. Depending on the values of the mass parameter and Yang-Mills charge, they can be interpreted as black holes with one horizon, two horizons and naked singularity. It is also shown that the solution associated with these black holes has an essential curvature singularity at the centre $r=0$. Thermodynamic and conserved quantities for these black holes are computed and we show that the first law has been verified. We also check thermodynamic stability in both canonical and grand canonical ensembles. In addition to this, we also formulate new power-Yang-Mills black hole solutions in pure quasi-topological gravity. The physical and thermodynamic properties of these black holes are discussed as well. It is concluded that unlike Yang-Mills black holes there exist stability regions for smaller power-Yang-Mills black holes in grand canonical ensemble. Finally, we discuss the thermodynamics of horizon flat power-Yang-Mills rotating black branes and analyze their thermodynamic and conserved quantities by using the counter-term method inspired by AdS/CFT correspondence.

hep-th↗

arXiv2022

Conformal symmetries of the energy-momentum tensor of spherically symmetric static spacetimes

Conformal matter collineations of the energy-momentum tensor for a general spherically symmetric static spacetime are studied. The general form of these collineations is found when the energy-momentum tensor is non-degenerate, and the maximum number of independent conformal matter collineations is \emph{fifteen}. In the degenerate case of the energy-momentum tensor it is found that these collineations have infinite degrees of freedom. In some subcases of degenerate energy-momentum, the Ricci tensor is non-degenerate, that is, there exist non-degenerate Ricci inheritance collineations.

gr-qc↗

arXiv2021

Particle dynamics and geometric optics in Chern-Simons black holes

In this paper we study the effects of the coupling constant of the Chern-Simons modified gravity on some physical properties of black holes. The Hawking mass is one of the proposed definitions of quasilocal mass. We find that, for slowly rotating Chern-Simons black holes, the Hawking mass is independent of the coupling constant. Next, we show the dependence on the centre of mass energy, for two neutral colliding particles, of coupling constant and the rotation parameter. We also investigate energy extraction through Penrose process and find that the energy gain and efficiency of the Penrose process are independent of this coupling constant. Rotation of the polarization vector is also studied for dependence on the Chern-Simons coupling constant.

gr-qc↗

arXiv2021

The non-Kerr black hole with acceleration

The no-hair theorem can be tested in the strong gravity regime by using the top-bottom approach and the bottom-top approach. The non-Kerr spacetime of the later approach is an ideal framework to do the tests in the region very close to the black holes. In this work, we propose a non-Kerr black hole metric (and its charged extension) that is accelerating as well. These new objects are studied for their basic properties and thermodynamics.

gr-qc↗

arXiv2020

Lovelock black holes surrounded by dark fluid in power-Yang-Mills massive gravity

We consider a model where massive static spherically symmetric black hole, in the presence of power-Yang-Mills source, is surrounded by a dark fluid with non-linear equation of state. In this set up we construct a new class of magnetized Lovelock black hole solutions of the gravitational field equations. In particular, we work out the metric functions in both D-dimensional massive Einstein and massive Gauss-Bonnet gravities. We study thermodynamics of these black holes also and show that the mass and associated thermodynamic quantities like Hawking temperature and heat capacity depend on parameters of the dark fluid and the power-Yang-Mills magnetic source. We note that the entropy does not satisfy the area law in Gauss-Bonnet and higher order Lovelock black holes. Furthermore, phase transitions of black holes in each case are also discussed.

gr-qc↗

arXiv2020

Lorentz symmetry violating BTZ black holes in massive gravity

BTZ black holes provide excellent frameworks for studying theories that are at the interface of classical and quantum gravity. In this paper we couple the Riemannian spacetime with the bumblebee field, in the background of massive gravity, which produces the violation of spontaneous Lorentz symmetry. In this setup we construct a large family of static vacuum BTZ black hole solutions. We study the asymptotic behaviour of curvature invariants and show that our resulting solutions describe asymptotically AdS (2+1)-dimensional BTZ black holes with negative cosmological constant. For positive cosmological constant they are de Sitter. Thermodynamics of these black holes is also analysed.

gr-qc↗

arXiv2020

Thermodynamics of dynamical wormholes

We study thermodynamics of dynamical traversable wormholes. These wormholes are investigated in the background of different cosmological models, with and without the cosmological constant, and which include the power-law and exponential cosmologies also. We work out the generalized surface gravity for wormholes of different shapes. The surface gravity is evaluated at the trapping horizon and the unified first law of thermodynamics is set up. The thermodynamic stability of these wormholes has also been investigated. Some cases of asymptotically flat, de Sitter and anti-de Sitter wormholes have been considered as well. Our results generalize those that exist for static Morris-Thorne wormholes.

gr-qc↗

arXiv2019

Magnetized topological black holes of dimensionally continued gravity

In this paper, a large family of topological black hole solutions of dimensionally continued gravity are derived. The action of Lovelock gravity is coupled to the exponential electrodynamics and the equations of motion are solved in the presence of a pure magnetic source. We work out the metric functions in terms of the parameter $β$ of exponential electrodynamics, and magnetic charge. Further, we couple Lovelock gravity to power-Yang-Mills theory and construct black holes, in diverse dimensions, having Yang-Mills magnetic charge. We also discuss the asymptotic bahaviour of metric functions and curvature invariants at the origin for both the models. The thermodynamics of resulting magnetized black hole solutions in the framework of two different models is also studied. The thermodynamical quantities like Hawking temperature, entropy and specific heat capacity at constant charge are found and we show that the resulting quantities satisfy the first law of black hole thermodynamics. We also study the magnetized hairy black holes of dimensionally continued gravity.

gr-qc↗

arXiv2019

Unified first law for traversable wormholes in non-minimal coupling of curvature and matter

In this paper thermodynamics of static Morris-Thorne wormholes has been discussed in the context of $f(R)$ gravity. The generalized surface gravity, unified first law of thermodynamics and wormhole dynamics have been studied at trapping horizons. We have investigated thermodynamics in non-minimal coupling of curvature and matter which produces very complex equations. Our results generalize the results that have already been derived in Einstein's gravity in the absence of curvature-matter coupling.

gr-qc↗