arXiv subjects

Irina Sorokina

Publications and source records attributed to Irina Sorokina.

9 recordsLinked to original sources

arXiv2024

MTUncertainty: Assessing the Need for Post-editing of Machine Translation Outputs by Fine-tuning OpenAI LLMs

Translation Quality Evaluation (TQE) is an essential step of the modern translation production process. TQE is critical in assessing both machine translation (MT) and human translation (HT) quality without reference translations. The ability to evaluate or even simply estimate the quality of translation automatically may open significant efficiency gains through process optimisation. This work examines whether the state-of-the-art large language models (LLMs) can be used for this purpose. We take OpenAI models as the best state-of-the-art technology and approach TQE as a binary classification task. On eight language pairs including English to Italian, German, French, Japanese, Dutch, Portuguese, Turkish, and Chinese, our experimental results show that fine-tuned gpt3.5 can demonstrate good performance on translation quality prediction tasks, i.e. whether the translation needs to be edited. Another finding is that simply increasing the sizes of LLMs does not lead to apparent better performances on this task by comparing the performance of three different versions of OpenAI models: curie, davinci, and gpt3.5 with 13B, 175B, and 175B parameters, respectively.

cs.CL↗

arXiv2024

Dissipative Soliton Resonance: Adiabatic Theory and Thermodynamics

We present the adiabatic theory of dissipative solitons (DS) of complex cubic-quintic nonlinear Ginzburg-Landau equation (CQGLE). Solutions in the closed analytical form in the spectral domain have the shape of Rayleigh-Jeans distribution for a positive (normal) dispersion. The DS parametric space forms a two-dimensional (or three-dimensional for the complex quintic nonlinearity) master diagram connecting the DS energy and a universal parameter formed by the ratio of four real and imaginary coefficients for dissipative and non-dissipative terms in CQGLE. The concept of dissipative soliton resonance (DSR) is formulated in terms of the master diagram, and the main signatures of transition to DSR are demonstrated and experimentally verified. We show a close analogy between DS and incoherent (semicoherent) solitons with an ensemble of quasi-particles confined by a collective potential. It allows applying the thermodynamical approach to DS and deriving the conditions for the DS energy scalability.

nlin.PS↗

arXiv2024

Neural Machine Translation of Clinical Text: An Empirical Investigation into Multilingual Pre-Trained Language Models and Transfer-Learning

We conduct investigations on clinical text machine translation by examining multilingual neural network models using deep learning such as Transformer based structures. Furthermore, to address the language resource imbalance issue, we also carry out experiments using a transfer learning methodology based on massive multilingual pre-trained language models (MMPLMs). The experimental results on three subtasks including 1) clinical case (CC), 2) clinical terminology (CT), and 3) ontological concept (OC) show that our models achieved top-level performances in the ClinSpEn-2022 shared task on English-Spanish clinical domain data. Furthermore, our expert-based human evaluations demonstrate that the small-sized pre-trained language model (PLM) won over the other two extra-large language models by a large margin, in the clinical domain fine-tuning, which finding was never reported in the field. Finally, the transfer learning method works well in our experimental setting using the WMT21fb model to accommodate a new language space Spanish that was not seen at the pre-training stage within WMT21fb itself, which deserves more exploitation for clinical knowledge transformation, e.g. to investigate into more languages. These research findings can shed some light on domain-specific machine translation development, especially in clinical and healthcare fields. Further research projects can be carried out based on our work to improve healthcare text analytics and knowledge transformation. Our data will be openly available for research purposes at https://github.com/HECTA-UoM/ClinicalNMT

cs.CL↗

arXiv2023

Investigating Massive Multilingual Pre-Trained Machine Translation Models for Clinical Domain via Transfer Learning

Massively multilingual pre-trained language models (MMPLMs) are developed in recent years demonstrating superpowers and the pre-knowledge they acquire for downstream tasks. This work investigates whether MMPLMs can be applied to clinical domain machine translation (MT) towards entirely unseen languages via transfer learning. We carry out an experimental investigation using Meta-AI's MMPLMs ``wmt21-dense-24-wide-en-X and X-en (WMT21fb)'' which were pre-trained on 7 language pairs and 14 translation directions including English to Czech, German, Hausa, Icelandic, Japanese, Russian, and Chinese, and the opposite direction. We fine-tune these MMPLMs towards English-\textit{Spanish} language pair which \textit{did not exist at all} in their original pre-trained corpora both implicitly and explicitly. We prepare carefully aligned \textit{clinical} domain data for this fine-tuning, which is different from their original mixed domain knowledge. Our experimental result shows that the fine-tuning is very successful using just 250k well-aligned in-domain EN-ES segments for three sub-task translation testings: clinical cases, clinical terms, and ontology concepts. It achieves very close evaluation scores to another MMPLM NLLB from Meta-AI, which included Spanish as a high-resource setting in the pre-training. To the best of our knowledge, this is the first work on using MMPLMs towards \textit{clinical domain transfer-learning NMT} successfully for totally unseen languages during pre-training.

cs.CL↗

arXiv2022

Atmospheric dispersion management in the mid-IR mode-locked oscillators

The atmospheric dispersion in the mid-infrared transparency windows presents and important albeit frequently neglected factor when developing ultra-short pulsed lasers. We show that it can amount to hundreds of fs^2 in 2-3 micron window with typical laser round-trip path lengths. Using the Cr:ZnS ultrashort-pulsed laser as a test-bed, we demonstrate the atmospheric dispersion influence on a femtosecond and chirped-pulse oscillator performance and show that the humidity fluctuations can be compensated by active dispersion control, greatly improving stability of mid-IR few-optical cycle laser sources. The approach can be readily extended to any ultrafast source in the mid-IR transparency windows.

physics.optics↗

arXiv2022

Examining Large Pre-Trained Language Models for Machine Translation: What You Don't Know About It

Pre-trained language models (PLMs) often take advantage of the monolingual and multilingual dataset that is freely available online to acquire general or mixed domain knowledge before deployment into specific tasks. Extra-large PLMs (xLPLMs) are proposed very recently to claim supreme performances over smaller-sized PLMs such as in machine translation (MT) tasks. These xLPLMs include Meta-AI's wmt21-dense-24-wide-en-X (2021) and NLLB (2022). In this work, we examine if xLPLMs are absolutely superior to smaller-sized PLMs in fine-tuning toward domain-specific MTs. We use two different in-domain data of different sizes: commercial automotive in-house data and clinical shared task data from the ClinSpEn2022 challenge at WMT2022. We choose popular Marian Helsinki as smaller sized PLM and two massive-sized Mega-Transformers from Meta-AI as xLPLMs. Our experimental investigation shows that 1) on smaller-sized in-domain commercial automotive data, xLPLM wmt21-dense-24-wide-en-X indeed shows much better evaluation scores using SacreBLEU and hLEPOR metrics than smaller-sized Marian, even though its score increase rate is lower than Marian after fine-tuning; 2) on relatively larger-size well prepared clinical data fine-tuning, the xLPLM NLLB tends to lose its advantage over smaller-sized Marian on two sub-tasks (clinical terms and ontology concepts) using ClinSpEn offered metrics METEOR, COMET, and ROUGE-L, and totally lost to Marian on Task-1 (clinical cases) on all official metrics including SacreBLEU and BLEU; 3) metrics do not always agree with each other on the same tasks using the same model outputs; 4) clinic-Marian ranked No.2 on Task- 1 (via SACREBLEU/BLEU) and Task-3 (via METEOR and ROUGE) among all submissions.

cs.CL↗

arXiv2022

High-Power Mid-IR Few-Cycle Frequency Comb from Quadratic Solitons in an Optical Parametric Oscillator

Powerful and efficient optical frequency combs in the mid-infrared (MIR) spectral region are highly desirable for a broad range of applications. Despite extensive efforts utilizing various techniques, MIR frequency comb sources are still lacking power, efficiency, or bandwidth for many applications. Here, we report the generation of an intrinsically locked frequency comb source centered at 4.18 μm from an optical parametric oscillator (OPO) operating in the simulton regime, in which formation of purely quadratic solitons lead to enhanced performance. We show advantages of operation in the simulton regime in direct experimental comparisons to the conventional regime, which are also supported by simulation and theory. We achieve 565 mW of average power, 900 nm of instantaneous 3-dB bandwidth, 350% of slope efficiency, and 44% of conversion efficiency; a performance that is superior to previous OPO demonstrations and other sources in this wavelength range. This work opens a new avenue toward MIR frequency comb generation with high power and efficiency and suggests the great potential of soliton generation based on quadratic nonlinearity in the MIR spectral region.

physics.optics↗

arXiv2021

cushLEPOR: customising hLEPOR metric using Optuna for higher agreement with human judgments or pre-trained language model LaBSE

Human evaluation has always been expensive while researchers struggle to trust the automatic metrics. To address this, we propose to customise traditional metrics by taking advantages of the pre-trained language models (PLMs) and the limited available human labelled scores. We first re-introduce the hLEPOR metric factors, followed by the Python version we developed (ported) which achieved the automatic tuning of the weighting parameters in hLEPOR metric. Then we present the customised hLEPOR (cushLEPOR) which uses Optuna hyper-parameter optimisation framework to fine-tune hLEPOR weighting parameters towards better agreement to pre-trained language models (using LaBSE) regarding the exact MT language pairs that cushLEPOR is deployed to. We also optimise cushLEPOR towards professional human evaluation data based on MQM and pSQM framework on English-German and Chinese-English language pairs. The experimental investigations show cushLEPOR boosts hLEPOR performances towards better agreements to PLMs like LaBSE with much lower cost, and better agreements to human evaluations including MQM and pSQM scores, and yields much better performances than BLEU (data available at \url{https://github.com/poethan/cushLEPOR}). Official results show that our submissions win three language pairs including \textbf{English-German} and \textbf{Chinese-English} on \textit{News} domain via cushLEPOR(LM) and \textbf{English-Russian} on \textit{TED} domain via hLEPOR.

cs.CL↗

arXiv2021

Measuring Uncertainty in Translation Quality Evaluation (TQE)

From both human translators (HT) and machine translation (MT) researchers' point of view, translation quality evaluation (TQE) is an essential task. Translation service providers (TSPs) have to deliver large volumes of translations which meet customer specifications with harsh constraints of required quality level in tight time-frames and costs. MT researchers strive to make their models better, which also requires reliable quality evaluation. While automatic machine translation evaluation (MTE) metrics and quality estimation (QE) tools are widely available and easy to access, existing automated tools are not good enough, and human assessment from professional translators (HAP) are often chosen as the golden standard \cite{han-etal-2021-TQA}. Human evaluations, however, are often accused of having low reliability and agreement. Is this caused by subjectivity or statistics is at play? How to avoid the entire text to be checked and be more efficient with TQE from cost and efficiency perspectives, and what is the optimal sample size of the translated text, so as to reliably estimate the translation quality of the entire material? This work carries out such motivated research to correctly estimate the confidence intervals \cite{Brown_etal2001Interval} depending on the sample size of the translated text, e.g. the amount of words or sentences, that needs to be processed on TQE workflow step for confident and reliable evaluation of overall translation quality. The methodology we applied for this work is from Bernoulli Statistical Distribution Modelling (BSDM) and Monte Carlo Sampling Analysis (MCSA).

cs.CL↗