arXiv subjects

Yan Ji

Publications and source records attributed to Yan Ji.

8 recordsLinked to original sources

arXiv2024

Automatic String Data Validation with Pattern Discovery

In enterprise data pipelines, data insertions occur periodically and may impact downstream services if data quality issues are not addressed. Typically, such problems can be investigated and fixed by on-call engineers, but locating the cause of such problems and fixing errors are often time-consuming. Therefore, automatic data validation is a better solution to defend the system and downstream services by enabling early detection of errors and providing detailed error messages for quick resolution. This paper proposes a self-validate data management system with automatic pattern discovery techniques to verify the correctness of semi-structural string data in enterprise data pipelines. Our solution extracts patterns from historical data and detects erroneous incoming data in a top-down fashion. High-level information of historical data is analyzed to discover the format skeleton of correct values. Fine-grained semantic patterns are then extracted to strike a balance between generalization and specification of the discovered pattern, thus covering as many correct values as possible while avoiding over-fitting. To tackle cold start and rapid data growth, we propose an incremental update strategy and example generalization strategy. Experiments on large-scale industrial and public datasets demonstrate the effectiveness and efficiency of our method compared to alternative solutions. Furthermore, a case study on an industrial platform (Ant Group Inc.) with thousands of applications shows that our system captures meaningful data patterns in daily operations and helps engineers quickly identify errors.

cs.DB↗

arXiv2024

PROF: Protected Order Flow in a Profit-Seeking World

Users of decentralized finance (DeFi) applications face significant risks from adversarial actions that manipulate the order of transactions to extract value from users. Such actions -- an adversarial form of what is called maximal-extractable value (MEV) -- impact both individual outcomes and the stability of the DeFi ecosystem. MEV exploitation, moreover, is being institutionalized through an architectural paradigm known Proposer-Builder Separation (PBS). This work introduces a system called PROF (PRotected Order Flow) that is designed to limit harmful forms of MEV in existing PBS systems. PROF aims at this goal using two ideas. First, PROF imposes an ordering on a set ("bundle") of privately input transactions and enforces that ordering all the way through to block production -- preventing transaction-order manipulation. Second, PROF creates bundles whose inclusion is profitable to block producers, thereby ensuring that bundles see timely inclusion in blocks. PROF is backward-compatible, meaning that it works with existing and future PBS designs. PROF is also compatible with any desired algorithm for ordering transactions within a PROF bundle (e.g., first-come, first-serve, fee-based, etc.). It executes efficiently, i.e., with low latency, and requires no additional trust assumptions among PBS entities. We quantitatively and qualitatively analyze incentive structure of PROF, and its utility to users compared with existing solutions. We also report on inclusion likelihood of PROF transactions, and concrete latency numbers through our end-to-end implementation.

cs.CR↗

arXiv2024

zkLogin: Privacy-Preserving Blockchain Authentication with Existing Credentials

For many users, a private key based wallet serves as the primary entry point to blockchains. Commonly recommended wallet authentication methods, such as mnemonics or hardware wallets, can be cumbersome. This difficulty in user onboarding has significantly hindered the adoption of blockchain-based applications. We develop zkLogin, a novel technique that leverages identity tokens issued by popular platforms (any OpenID Connect enabled platform e.g., Google, Facebook, etc.) to authenticate transactions. At the heart of zkLogin lies a signature scheme allowing the signer to sign using their existing OpenID accounts and nothing else. This improves the user experience significantly as users do not need to remember a new secret and can reuse their existing accounts. zkLogin provides strong security and privacy guarantees. Unlike prior works, zkLogin's security relies solely on the underlying platform's authentication mechanism without the need for any additional trusted parties (e.g., trusted hardware or oracles). As the name suggests, zkLogin leverages zero-knowledge proofs (ZKP) to ensure that the sensitive link between a user's off-chain and on-chain identities is hidden, even from the platform itself. zkLogin enables a number of important applications outside blockchains. It allows billions of users to produce \textit{verifiable digital content leveraging their existing digital identities}, e.g., email address. For example, a journalist can use zkLogin to sign a news article with their email address, allowing verification of the article's authorship by any party. We have implemented and deployed zkLogin on the Sui blockchain as an additional alternative to traditional digital signature-based addresses.

cs.CR↗

arXiv2019

BDoS: Blockchain Denial of Service

Proof-of-work (PoW) cryptocurrency blockchains like Bitcoin secure vast amounts of money. Their operators, called miners, expend resources to generate blocks and receive monetary rewards for their effort. Blockchains are, in principle, attractive targets for Denial-of-Service (DoS) attacks: There is fierce competition among coins, as well as potential gains from short selling. Classical DoS attacks, however, typically target a few servers and cannot scale to systems with many nodes. There have been no successful DoS attacks to date against prominent cryptocurrencies. We present Blockchain DoS (BDoS), the first incentive-based DoS attack that targets PoW cryptocurrencies. Unlike classical DoS, BDoS targets the system's mechanism design: It exploits the reward mechanism to discourage miner participation. Previous DoS attacks against PoW blockchains require an adversary's mining power to match that of all other miners. In contrast, BDoS can cause a blockchain to grind to a halt with significantly fewer resources, e.g., 21% as of March 2020 in Bitcoin, according to our empirical study. We find that Bitcoin's vulnerability to BDoS increases rapidly as the mining industry matures and profitability drops. BDoS differs from known attacks like Selfish Mining in its aim not to increase an adversary's revenue, but to disrupt the system. Although it bears some algorithmic similarity to those attacks, it introduces a new adversarial model, goals, algorithm, and game-theoretic analysis. Beyond its direct implications for operational blockchains, BDoS introduces the novel idea that an adversary can manipulate miners' incentives by proving the existence of blocks without actually publishing them.

cs.CR↗

arXiv2019

SquirRL: Automating Attack Analysis on Blockchain Incentive Mechanisms with Deep Reinforcement Learning

Incentive mechanisms are central to the functionality of permissionless blockchains: they incentivize participants to run and secure the underlying consensus protocol. Designing incentive-compatible incentive mechanisms is notoriously challenging, however. As a result, most public blockchains today use incentive mechanisms whose security properties are poorly understood and largely untested. In this work, we propose SquirRL, a framework for using deep reinforcement learning to analyze attacks on blockchain incentive mechanisms. We demonstrate SquirRL's power by first recovering known attacks: (1) the optimal selfish mining attack in Bitcoin [52], and (2) the Nash equilibrium in block withholding attacks [16]. We also use SquirRL to obtain several novel empirical results. First, we discover a counterintuitive flaw in the widely used rushing adversary model when applied to multi-agent Markov games with incomplete information. Second, we demonstrate that the optimal selfish mining strategy identified in [52] is actually not a Nash equilibrium in the multi-agent selfish mining setting. In fact, our results suggest (but do not prove) that when more than two competing agents engage in selfish mining, there is no profitable Nash equilibrium. This is consistent with the lack of observed selfish mining in the wild. Third, we find a novel attack on a simplified version of Ethereum's finalization mechanism, Casper the Friendly Finality Gadget (FFG) that allows a strategic agent to amplify her rewards by up to 30%. Notably, [10] show that honest voting is a Nash equilibrium in Casper FFG: our attack shows that when Casper FFG is composed with selfish mining, this is no longer the case. Altogether, our experiments demonstrate SquirRL's flexibility and promise as a framework for studying attack settings that have thus far eluded theoretical and empirical understanding.

cs.CR↗

arXiv2017

Updating the silent speech challenge benchmark with deep learning

The 2010 Silent Speech Challenge benchmark is updated with new results obtained in a Deep Learning strategy, using the same input features and decoding strategy as in the original article. A Word Error Rate of 6.4% is obtained, compared to the published value of 17.4%. Additional results comparing new auto-encoder-based features with the original features at reduced dimensionality, as well as decoding scenarios on two different language models, are also presented. The Silent Speech Challenge archive has been updated to contain both the original and the new auto-encoder features, in addition to the original raw data.

cs.CL↗

arXiv2008

A New Car-Following Model Inspired by Galton Board

Different from previous models based on scatter theory and random matrix theory, a new interpretation of the observed log-normal type time-headway distribution of vehicles is presented in this paper. Inspired by the well known Galton Board, this model views driver's velocity adjusting process similar to the dynamics of a particle falling down a board and being deviated at decision points. A new car-following model based on this idea is proposed to reproduce the observed traffic flow phenomena. The agreement between the empirical observations and the simulation results suggests the soundness of this new approach.

physics.data-an↗

arXiv2008

A Markov Process Inspired Cellular Automata Model of Road Traffic

To provide a more accurate description of the driving behaviors in vehicle queues, a namely Markov-Gap cellular automata model is proposed in this paper. It views the variation of the gap between two consequent vehicles as a Markov process whose stationary distribution corresponds to the observed distribution of practical gaps. The multiformity of this Markov process provides the model enough flexibility to describe various driving behaviors. Two examples are given to show how to specialize it for different scenarios: usually mentioned flows on freeways and start-up flows at signalized intersections. The agreement between the empirical observations and the simulation results suggests the soundness of this new approach.

physics.data-an↗