arXiv subjects

Jack H. Culbert

Publications and source records attributed to Jack H. Culbert.

5 recordsLinked to original sources

arXiv2025

Investigating the originality of scientific papers across time and domain: A quantitative analysis

The study of creativity in science has long sought quantitative metrics capable of capturing the originality of the scientific insights contained within articles and other scientific works. In recent years, the field has witnessed a substantial expansion of research activity, enabled by advances in natural language processing and network analysis, and has utilised both macro- and micro-scale approaches with success. However, they often do not examine the text itself for evidence of originality. In this paper, we apply a computational measure correlating with originality from creativity science, Divergent Semantic Integration (DSI), to a set of 51,200 scientific abstracts and titles sourced from the Web of Science. To adapt DSI for application to scientific texts, we advance the original BERT method by incorporating SciBERT (a model trained on scientific corpora) into the computation of DSI. In our study, we observe that DSI plays a more pronounced role in the accrual of early citations for papers with fewer authors, varies substantially across subjects and research fields, and exhibits a declining correlation with citation counts over time. Furthermore, by modelling SciBERT- and BERT-DSI as predictors of the logarithm of 5-year citation counts alongside field, publication year, and the logarithm of author count, we find statistically significant relationships, with adjusted R-squared of 0.103 and 0.101 for BERT-DSI and SciBERT-DSI. Because existing scientometric measures rarely assess the originality expressed in textual content, DSI provides a valuable means of directly quantifying the conceptual originality embedded in scientific writing.

cs.DL↗

arXiv2025

Originality in scientific titles and abstracts can predict citation count

In this research-in-progress paper, we apply a computational measure correlating with originality from creativity science: Divergent Semantic Integration (DSI), to a selection of 99,557 scientific abstracts and titles selected from the Web of Science. We observe statistically significant differences in DSI between subject and field of research, and a slight rise in DSI over time. We model the base 10 logarithm of the citation count after 5 years with DSI and find a statistically significant positive correlation in all fields of research with an adjusted $R^2$ of 0.13.

cs.DL↗

arXiv2024

Utilizing Large Language Models for Named Entity Recognition in Traditional Chinese Medicine against COVID-19 Literature: Comparative Study

Objective: To explore and compare the performance of ChatGPT and other state-of-the-art LLMs on domain-specific NER tasks covering different entity types and domains in TCM against COVID-19 literature. Methods: We established a dataset of 389 articles on TCM against COVID-19, and manually annotated 48 of them with 6 types of entities belonging to 3 domains as the ground truth, against which the NER performance of LLMs can be assessed. We then performed NER tasks for the 6 entity types using ChatGPT (GPT-3.5 and GPT-4) and 4 state-of-the-art BERT-based question-answering (QA) models (RoBERTa, MiniLM, PubMedBERT and SciBERT) without prior training on the specific task. A domain fine-tuned model (GSAP-NER) was also applied for a comprehensive comparison. Results: The overall performance of LLMs varied significantly in exact match and fuzzy match. In the fuzzy match, ChatGPT surpassed BERT-based QA models in 5 out of 6 tasks, while in exact match, BERT-based QA models outperformed ChatGPT in 5 out of 6 tasks but with a smaller F-1 difference. GPT-4 showed a significant advantage over other models in fuzzy match, especially on the entity type of TCM formula and the Chinese patent drug (TFD) and ingredient (IG). Although GPT-4 outperformed BERT-based models on entity type of herb, target, and research method, none of the F-1 scores exceeded 0.5. GSAP-NER, outperformed GPT-4 in terms of F-1 by a slight margin on RM. ChatGPT achieved considerably higher recalls than precisions, particularly in the fuzzy match. Conclusions: The NER performance of LLMs is highly dependent on the entity type, and their performance varies across application scenarios. ChatGPT could be a good choice for scenarios where high recall is favored. However, for knowledge acquisition in rigorous scenarios, neither ChatGPT nor BERT-based QA models are off-the-shelf tools for professional practitioners.

cs.CL↗

arXiv2024

Analysis of the Publication and Document Types in OpenAlex, Web of Science, Scopus, PubMed and Semantic Scholar

The assignment of document and publication types in scholarly databases plays an important role in bibliometrics, for example in decision-making or university rankings. However, scholarly databases apply different curation strategies and taxonomies when classifying documents which makes it difficult to compare results from different database providers. In this study, the bibliometric databases OpenAlex, Web of Science, Scopus, PubMed and Semantic Scholar are used to analyse the extent of data variation and compare different approaches to taxonomy and data curation. Using a shared corpus of 9,575,603 publications from 2012 to 2022, we found large differences in the classification of document types such as research articles and editorials in these databases. We can also show that many of the records that lack a publication type in OpenAlex are classified as conference proceedings in Scopus and Semantic Scholar.

cs.DL↗

arXiv2023

4TCT, A 4chan Text Collection Tool

4chan is a popular online imageboard which has been widely studied due to an observed concentration of far-right, antisemitic, racist, misogynistic, and otherwise hateful material being posted to the site, as well as the emergence of political movements and the evolution of memes which are posted there, discussed in Section 1.1. We have created a tool developed in Python which utilises the 4chan API to collect data from a selection of boards. This paper accompanies the release of the code via the github repository: https://github.com/jhculb/4TCT. We believe this tool will be of use to academics studying 4chan by providing a tool for collection of data from 4chan to sociological researchers, and potentially contributing to GESIS' Digital Behavioural Data project.

cs.DL↗