arXiv subjects

Ilya Valmianski

Publications and source records attributed to Ilya Valmianski.

11 recordsLinked to original sources

arXiv2023

Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue

Factual correctness is often the limiting factor in practical applications of natural language generation in high-stakes domains such as healthcare. An essential requirement for maintaining factuality is the ability to deal with rare tokens. This paper focuses on rare tokens that appear in both the source and the reference sequences, and which, when missed during generation, decrease the factual correctness of the output text. For high-stake domains that are also knowledge-rich, we show how to use knowledge to (a) identify which rare tokens that appear in both source and reference are important and (b) uplift their conditional probability. We introduce the ``utilization rate'' that encodes knowledge and serves as a regularizer by maximizing the marginal probability of selected tokens. We present a study in a knowledge-rich domain of healthcare, where we tackle the problem of generating after-visit care instructions based on patient-doctor dialogues. We verify that, in our dataset, specific medical concepts with high utilization rates are underestimated by conventionally trained sequence-to-sequence models. We observe that correcting this with our approach to knowledge injection reduces the uncertainty of the model as well as improves factuality and coherence without negatively impacting fluency.

cs.CL↗

arXiv2023

Dialogue-Contextualized Re-ranking for Medical History-Taking

AI-driven medical history-taking is an important component in symptom checking, automated patient intake, triage, and other AI virtual care applications. As history-taking is extremely varied, machine learning models require a significant amount of data to train. To overcome this challenge, existing systems are developed using indirect data or expert knowledge. This leads to a training-inference gap as models are trained on different kinds of data than what they observe at inference time. In this work, we present a two-stage re-ranking approach that helps close the training-inference gap by re-ranking the first-stage question candidates using a dialogue-contextualized model. For this, we propose a new model, global re-ranker, which cross-encodes the dialogue with all questions simultaneously, and compare it with several existing neural baselines. We test both transformer and S4-based language model backbones. We find that relative to the expert system, the best performance is achieved by our proposed global re-ranker with a transformer backbone, resulting in a 30% higher normalized discount cumulative gain (nDCG) and a 77% higher mean average precision (mAP).

cs.CL↗

arXiv2022

OSLAT: Open Set Label Attention Transformer for Medical Entity Retrieval and Span Extraction

Medical entity span extraction and linking are critical steps for many healthcare NLP tasks. Most existing entity extraction methods either have a fixed vocabulary of medical entities or require span annotations. In this paper, we propose a method for linking an open set of entities that does not require any span annotations. Our method, Open Set Label Attention Transformer (OSLAT), uses the label-attention mechanism to learn candidate-entity contextualized text representations. We find that OSLAT can not only link entities but is also able to implicitly learn spans associated with entities. We evaluate OSLAT on two tasks: (1) span extraction trained without explicit span annotations, and (2) entity linking trained without span-level annotation. We test the generalizability of our method by training two separate models on two datasets with low entity overlap and comparing cross-dataset performance.

cs.CL↗

arXiv2022

Learning functional sections in medical conversations: iterative pseudo-labeling and human-in-the-loop approach

Medical conversations between patients and medical professionals have implicit functional sections, such as "history taking", "summarization", "education", and "care plan." In this work, we are interested in learning to automatically extract these sections. A direct approach would require collecting large amounts of expert annotations for this task, which is inherently costly due to the contextual inter-and-intra variability between these sections. This paper presents an approach that tackles the problem of learning to classify medical dialogue into functional sections without requiring a large number of annotations. Our approach combines pseudo-labeling and human-in-the-loop. First, we bootstrap using weak supervision with pseudo-labeling to generate dialogue turn-level pseudo-labels and train a transformer-based model, which is then applied to individual sentences to create noisy sentence-level labels. Second, we iteratively refine sentence-level labels using a cluster-based human-in-the-loop approach. Each iteration requires only a few dozen annotator decisions. We evaluate the results on an expert-annotated dataset of 100 dialogues and find that while our models start with 69.5% accuracy, we can iteratively improve it to 82.5%. The code used to perform all experiments described in this paper can be found here: https://github.com/curai/curai-research/tree/main/functional-sections.

cs.CL↗

arXiv2021

Adding more data does not always help: A study in medical conversation summarization with PEGASUS

Medical conversation summarization is integral in capturing information gathered during interactions between patients and physicians. Summarized conversations are used to facilitate patient hand-offs between physicians, and as part of providing care in the future. Summaries, however, can be time-consuming to produce and require domain expertise. Modern pre-trained NLP models such as PEGASUS have emerged as capable alternatives to human summarization, reaching state-of-the-art performance on many summarization benchmarks. However, many downstream tasks still require at least moderately sized datasets to achieve satisfactory performance. In this work we (1) explore the effect of dataset size on transfer learning medical conversation summarization using PEGASUS and (2) evaluate various iterative labeling strategies in the low-data regime, following their success in the classification setting. We find that model performance saturates with increase in dataset size and that the various active-learning strategies evaluated all show equivalent performance consistent with simple dataset size increase. We also find that naive iterative pseudo-labeling is on-par or slightly worse than no pseudo-labeling. Our work sheds light on the successes and challenges of translating low-data regime techniques in classification to medical conversation summarization and helps guides future work in this space. Relevant code available at \url{https://github.com/curai/curai-research/tree/main/medical-summarization-ML4H-2021}.

cs.CL↗

arXiv2021

MEDCOD: A Medically-Accurate, Emotive, Diverse, and Controllable Dialog System

We present MEDCOD, a Medically-Accurate, Emotive, Diverse, and Controllable Dialog system with a unique approach to the natural language generator module. MEDCOD has been developed and evaluated specifically for the history taking task. It integrates the advantage of a traditional modular approach to incorporate (medical) domain knowledge with modern deep learning techniques to generate flexible, human-like natural language expressions. Two key aspects of MEDCOD's natural language output are described in detail. First, the generated sentences are emotive and empathetic, similar to how a doctor would communicate to the patient. Second, the generated sentence structures and phrasings are varied and diverse while maintaining medical consistency with the desired medical concept (provided by the dialogue manager module of MEDCOD). Experimental results demonstrate the effectiveness of our approach in creating a human-like medical dialogue system. Relevant code is available at https://github.com/curai/curai-research/tree/main/MEDCOD

cs.CL↗

arXiv2021

SmartTriage: A system for personalized patient data capture, documentation generation, and decision support

Symptom checkers have emerged as an important tool for collecting symptoms and diagnosing patients, minimizing the involvement of clinical personnel. We developed a machine-learning-backed system, SmartTriage, which goes beyond conventional symptom checking through a tight bi-directional integration with the electronic medical record (EMR). Conditioned on EMR-derived patient history, our system identifies the patient's chief complaint from a free-text entry and then asks a series of discrete questions to obtain relevant symptomatology. The patient-specific data are used to predict detailed ICD-10-CM codes as well as medication, laboratory, and imaging orders. Patient responses and clinical decision support (CDS) predictions are then inserted back into the EMR. To train the machine learning components of SmartTriage, we employed novel data sets of over 25 million primary care encounters and 1 million patient free-text reason-for-visit entries. These data sets were used to construct: (1) a long short-term memory (LSTM) based patient history representation, (2) a fine-tuned transformer model for chief complaint extraction, (3) a random forest model for question sequencing, and (4) a feed-forward network for CDS predictions. In total, our system supports 337 patient chief complaints, which together make up $>90\%$ of all primary care encounters at Kaiser Permanente.

cs.CL↗

arXiv2019

Intertwined magnetic, structural, and electronic transitions in V$_2$O$_3$

We present a coordinated study of the paramagnetic-to-antiferromagnetic, rhombohedral-to-monoclinic, and metal-to-insulator transitions in thin-film specimens of the classic Mott insulator V$_2$O$_3$ using low-energy muon spin relaxation, x-ray diffraction, and nanoscale-resolved near-field infrared spectroscopic techniques. The measurements provide a detailed characterization of the thermal evolution of the magnetic, structural, and electronic phase transitions occurring in a wide temperature range, including quantitative measurements of the high- and low-temperature phase fractions for each transition. The results reveal a stable coexistence of the high- and low-temperature phases over a broad temperature range throughout the transition. Careful comparison of temperature dependence of the different measurements, calibrated by the resistance of the sample, demonstrates that the electronic, magnetic, and structural degrees of freedom remain tightly coupled to each other during the transition process. We also find evidence for antiferromagnetic fluctuations in the vicinity of the phase transition, highlighting the important role of the magnetic degree of freedom in the metal-insulator transition.

cond-mat.str-el↗

arXiv2019

Evaluating robustness of language models for chief complaint extraction from patient-generated text

Automated classification of chief complaints from patient-generated text is a critical first step in developing scalable platforms to triage patients without human intervention. In this work, we evaluate several approaches to chief complaint classification using a novel Chief Complaint (CC) Dataset that contains ~200,000 patient-generated reasons-for-visit entries mapped to a set of 795 discrete chief complaints. We examine the use of several fine-tuned bidirectional transformer (BERT) models trained on both unrelated texts as well as on the CC dataset. We contrast this performance with a TF-IDF baseline. Our evaluation has three components: (1) a random test hold-out from the original dataset; (2) a "misspelling set," consisting of a hand-selected subset of the test set, where every entry has at least one misspelling; (3) a separate experimenter-generated free-text set. We find that the TF-IDF model performs significantly better than the strongest BERT-based model on the test (best BERT PR-AUC $0.3597 \pm 0.0041$ vs TF-IDF PR-AUC $0.3878 \pm 0.0148$, $p=7\cdot 10^{-5}$), and is statistically comparable to the misspelling sets (best BERT PR-AUC $0.2579 \pm 0.0079$ vs TF-IDF PR-AUC $0.2733 \pm 0.0130$, $p=0.06$). However, when examining model predictions on experimenter-generated queries, some concerns arise about TF-IDF baseline's robustness. Our results suggest that in certain tasks, simple language embedding baselines may be very performant; however, truly understanding their robustness requires further analysis.

cs.CL↗

arXiv2017

Non-equilibrium phase precursors to the insulator-metal transition in V2O3

The discovery of novel phases of matter is at the core of modern physics. In quantum materials, subtle variations in atomic-scale interactions can induce dramatic changes in macroscopic properties and drive phase transitions. Despite their importance, the mesoscale processes underpinning phase transitions often remain elusive because of the vast differences in timescales between atomic and electronic changes and thermodynamic transformations. Here, we photoinduce and directly observe with x-ray scattering an ultrafast enhancement of the structural long-range order in the archetypal Mott system V2O3. Despite the ultrafast change in crystal symmetry, the change of unit cell volume occurs an order of magnitude slower and coincides with the insulator-to-metal transition. The decoupling between the two structural responses in the time domain highlights the existence of a transient photoinduced precursor phase, which is distinct from the two structural phases present in equilibrium. X-ray nanoscopy reveals that acoustic phonons trapped in nanoscale blocks govern the dynamics of the ultrafast transition into the precursor phase, while nucleation and growth of metallic domains dictate the duration of the slower transition into the metallic phase. The enhancement of the long-range order before completion of the electronic transition demonstrates the critical role the non-equilibrium structural phases play during electronic phase transitions in correlated electrons systems.

cond-mat.str-el↗

arXiv2016

Low vibration high numerical aperture automated variable temperature Raman microscope

Raman micro-spectroscopy is well suited for studying a variety of properties and has been applied to wide- ranging areas. Combined with tuneable temperature, Raman spectra can offer even more insights into the properties of materials. However, previous designs of variable temperature Raman microscopes have made it extremely challenging to measure samples with low signal levels due to thermal and positional instability as well as low collection efficiencies. Thus, contemporary Raman microscope has found limited applicability to probing the subtle physics involved in phase transitions and hysteresis. This paper describes a new design of a closed-cycle, Raman microscope with full polarization rotation. High collection efficiency, thermal and mechanical stability are ensured by both deliberate optical, cryogenic, and mechanical design. Measurements on two samples, Bi2Se3 and V2O3, which are known as challenging due to low thermal conductivities, low signal levels and/or hysteretic effects, are measured with previously undemonstrated temperature resolution.

cond-mat.mtrl-sci↗