arXiv subjects

Stevan Harnad

Publications and source records attributed to Stevan Harnad.

At least 19 recordsLinked to original sources

arXiv2026

Tolerance Principle and Small Language Model Learning

Modern language models like GPT-3, BERT, and LLaMA require massive training data, yet with sufficient training they reliably learn to distinguish grammatical from ungrammatical sentences. Children aged as young as 14 months already have the capacity to learn abstract grammar rules from very few exemplars, even in the presence of non-rule-following exceptions. Yang's (2016) Tolerance Principle defines a precise threshold for how many exceptions a rule can tolerate and still be learnable. The present study explored the minimal amount and quality of training data necessary for rules to be generalized by a transformer-based language model to test the predictions of the Tolerance Principle. We trained BabyBERTa (Huebner et al. 2021), a transformer model optimized for small datasets, on artificial grammars. The training sets varied in size, number of unique sentence types, and proportion of rule-following versus exception exemplars. We found that, unlike human infants, BabyBERTa's learning dynamics do not align with the Tolerance Principle.

cs.CL↗

arXiv2024

Language Writ Large: LLMs, ChatGPT, Grounding, Meaning and Understanding

Apart from what (little) OpenAI may be concealing from us, we all know (roughly) how ChatGPT works (its huge text database, its statistics, its vector representations, and their huge number of parameters, its next-word training, and so on). But none of us can say (hand on heart) that we are not surprised by what ChatGPT has proved to be able to do with these resources. This has even driven some of us to conclude that ChatGPT actually understands. It is not true that it understands. But it is also not true that we understand how it can do what it can do. I will suggest some hunches about benign biases: convergent constraints that emerge at LLM scale that may be helping ChatGPT do so much better than we would have expected. These biases are inherent in the nature of language itself, at LLM scale, and they are closely linked to what it is that ChatGPT lacks, which is direct sensorimotor grounding to connect its words to their referents and its propositions to their meanings. These convergent biases are related to (1) the parasitism of indirect verbal grounding on direct sensorimotor grounding, (2) the circularity of verbal definition, (3) the mirroring of language production and comprehension, (4) iconicity in propositions at LLM scale, (5) computational counterparts of human categorical perception in category learning by neural nets, and perhaps also (6) a conjecture by Chomsky about the laws of thought. The exposition will be in the form of a dialogue with ChatGPT-4.

cs.CL↗

arXiv2019

Category learning can alter perception and its neural correlates

Learned Categorical Perception (CP) occurs when the members of different categories come to look more dissimilar (between-category separation) and/or members of the same category come to look more similar (within-category compression) after a new category has been learned. To measure learned CP and its physiological correlates we compared dissimilarity judgments and Event Related Potentials (ERPs) before and after learning to sort multi-featured visual textures into two categories by trial and error with corrective feedback. With the same number of trials and feedback, about half the participants succeeded in learning the categories (learners: criterion 80% accuracy) and the rest did not (non-learners). At both lower and higher levels of difficulty, successful learners showed significant between-category separation in pairwise dissimilarity judgments after learning compared to before; their late parietal ERP positivity (LPC, usually interpreted as decisional) also increased and their occipital negativity (N1) (usually interpreted as perceptual) decreased. LPC increased with response accuracy and N1 amplitude decreased with between-category separation for the Learners. Non-learners showed no significant changes in dissimilarity judgments, LPC or N1, within or between categories. This is behavioral and physiological evidence that category learning can alter perception. We sketch a neural net model for this effect.

q-bio.NC↗

arXiv2019

Codes, communication and cognition

Brette (2019) criticizes the notion of neural coding because it seems to entail that neural signals need to be decoded by or for some receiver in the head. If that were so, then neural coding would indeed be homuncular (Brette calls it dualistic), requiring an entity to decipher the code. But I think the plea of Brett to think instead in terms of complex, interactive causal throughput is preaching to the converted. Turing (not Shannon) has already shown the way. In any case, the metaphor of neural coding has little to do with the symbol grounding problem.

q-bio.NC↗

arXiv2018

The rent's too high: Self-archive for fair online publication costs

The main contributors of scientific knowledge, researchers, generally aim to disseminate their findings far and wide. And yet, publishing companies have largely kept these findings behind a paywall. With digital publication technology markedly reducing cost, this enduring wall seems disproportionate and unjustified; moreover, it has sparked a topical exchange concerning how to modernize academic publishing. This discussion, however, seems to focus on how to compensate major publishers for providing open access through a "pay to publish" model, in turn transferring financial burdens from libraries to authors and their funders. Large publishing companies, including Elsevier, Springer Nature, Wiley, PLoS, and Frontiers, continue to earn exorbitant revenues each year, hundreds of millions of dollars of which now come from processing charges for open-access articles. A less expensive and equally accessible alternative exists: widespread self-archiving of peer-reviewed articles. All we need is awareness of this alternative and the will to employ it

cs.DL↗

arXiv2018

Learning-induced categorical perception in a neural network model

In human cognition, the expansion of perceived between-category distances and compression of within-category distances is known as categorical perception (CP). There are several hypotheses about the causes of CP (e.g., language, learning, evolution) but no functional model. Whether CP is essential to categorisation or simply a by-product of it is not yet clear, but evidence is accumulating that CP can be induced by category learning. We provide a model for learning-induced CP as expansion and compression of distances in hidden-unit space in neural nets. Basic conditions from which the current model predicts CP are described, and clues as to how these conditions might generalize to more complex kinds of categorization begin to emerge.

cs.LG↗

arXiv2016

The Latent Structure of Dictionaries

How many words (and which ones) are sufficient to define all other words? When dictionaries are analyzed as directed graphs with links from defining words to defined words, they reveal a latent structure. Recursively removing all words that are reachable by definition but that do not define any further words reduces the dictionary to a Kernel of about 10%. This is still not the smallest number of words that can define all the rest. About 75% of the Kernel turns out to be its Core, a Strongly Connected Subset of words with a definitional path to and from any pair of its words and no word's definition depending on a word outside the set. But the Core cannot define all the rest of the dictionary. The 25% of the Kernel surrounding the Core consists of small strongly connected subsets of words: the Satellites. The size of the smallest set of words that can define all the rest (the graph's Minimum Feedback Vertex Set or MinSet) is about 1% of the dictionary, 15% of the Kernel, and half-Core, half-Satellite. But every dictionary has a huge number of MinSets. The Core words are learned earlier, more frequent, and less concrete than the Satellites, which in turn are learned earlier and more frequent but more concrete than the rest of the Dictionary. In principle, only one MinSet's words would need to be grounded through the sensorimotor capacity to recognize and categorize their referents. In a dual-code sensorimotor-symbolic model of the mental lexicon, the symbolic code could do all the rest via re-combinatory definition.

cs.CL↗

arXiv2016

Estimating Open Access Mandate Effectiveness: The MELIBEA Score

MELIBEA is a Spanish database that uses a composite formula with eight weighted conditions to estimate the effectiveness of Open Access mandates (registered in ROARMAP). We analyzed 68 mandated institutions for publication years 2011-2013 to determine how well the MELIBEA score and its individual conditions predict what percentage of published articles indexed by Web of Knowledge is deposited in each institution's OA repository, and when. We found a small but significant positive correlation (0.18) between MELIBEA score and deposit percentage. We also found that for three of the eight MELIBEA conditions (deposit timing, internal use, and opt-outs), one value of each was strongly associated with deposit percentage or deposit latency (immediate deposit required, deposit required for performance evaluation, unconditional opt-out allowed for the OA requirement but no opt-out for deposit requirement). When we updated the initial values and weights of the MELIBEA formula for mandate effectiveness to reflect the empirical association we had found, the score's predictive power doubled (.36). There are not yet enough OA mandates to test further mandate conditions that might contribute to mandate effectiveness, but these findings already suggest that it would be useful for future mandates to adopt these three conditions so as to maximize their effectiveness, and thereby the growth of OA.

cs.DL↗

arXiv2013

Hidden Structure and Function in the Lexicon

How many words are needed to define all the words in a dictionary? Graph-theoretic analysis reveals that about 10% of a dictionary is a unique Kernel of words that define one another and all the rest, but this is not the smallest such subset. The Kernel consists of one huge strongly connected component (SCC), about half its size, the Core, surrounded by many small SCCs, the Satellites. Core words can define one another but not the rest of the dictionary. The Kernel also contains many overlapping Minimal Grounding Sets (MGSs), each about the same size as the Core, each part-Core, part-Satellite. MGS words can define all the rest of the dictionary. They are learned earlier, more concrete and more frequent than the rest of the dictionary. Satellite words, not correlated with age or frequency, are less concrete (more abstract) words that are also needed for full lexical power.

cs.CL↗

arXiv2012

Testing the Finch Hypothesis on Green OA Mandate Ineffectiveness

We have now tested the Finch Committee's Hypothesis that Green Open Access Mandates are ineffective in generating deposits in institutional repositories. With data from ROARMAP on institutional Green OA mandates and data from ROAR on institutional repositories, we show that deposit number and rate is significantly correlated with mandate strength (classified as 1-12): The stronger the mandate, the more the deposits. The strongest mandates generate deposit rates of 70%+ within 2 years of adoption, compared to the un-mandated deposit rate of 20%. The effect is already detectable at the national level, where the UK, which has the largest proportion of Green OA mandates, has a national OA rate of 35%, compared to the global baseline of 25%. The conclusion is that, contrary to the Finch Hypothesis, Green Open Access Mandates do have a major effect, and the stronger the mandate, the stronger the effect (the Liege ID/OA mandate, linked to research performance evaluation, being the strongest mandate model). RCUK (as well as all universities, research institutions and research funders worldwide) would be well advised to adopt the strongest Green OA mandates and to integrate institutional and funder mandates.

cs.DL↗

arXiv2012

Alan Turing and the "Hard" and "Easy" Problem of Cognition: Doing and Feeling

The "easy" problem of cognitive science is explaining how and why we can do what we can do. The "hard" problem is explaining how and why we feel. Turing's methodology for cognitive science (the Turing Test) is based on doing: Design a model that can do anything a human can do, indistinguishably from a human, to a human, and you have explained cognition. Searle has shown that the successful model cannot be solely computational. Sensory-motor robotic capacities are necessary to ground some, at least, of the model's words, in what the robot can do with the things in the world that the words are about. But even grounding is not enough to guarantee that -- nor to explain how and why -- the model feels (if it does). That problem is much harder to solve (and perhaps insoluble).

cs.AI↗

arXiv2012

Green and Gold Open Access Percentages and Growth, by Discipline

Most refereed journal articles today are published in subscription journals, accessible only to subscribing institutions, hence losing considerable research impact. Making articles freely accessible online ("Open Access," OA) maximizes their impact. Articles can be made OA in two ways: by self-archiving them on the web ("Green OA") or by publishing them in OA journals ("Gold OA"). We compared the percent and growth rate of Green and Gold OA for 14 disciplines in two random samples of 1300 articles per discipline out of the 12,500 journals indexed by Thomson-Reuters-ISI using a robot that trawled the web for OA full-texts. We sampled in 2009 and 2011 for publication year ranges 1998-2006 and 2005-2010, respectively. Green OA (21.4%) exceeds Gold OA (2.4%) in proportion and growth rate in all but the biomedical disciplines, probably because it can be provided for all journals articles and does not require paying extra Gold OA publication fees. The spontaneous overall OA growth rate is still very slow (about 1% per year). If institutions make Green OA self-archiving mandatory, however, it triples percent Green OA as well as accelerating its growth rate.

cs.DL↗

arXiv2012

The Causal Topography of Cognition

The causal structure of cognition can be simulated but not implemented computationally, just as the causal structure of a comet can be simulated but not implemented computationally. The only thing that allows us even to imagine otherwise is that cognition, unlike a comet, is invisible (to all but the cognizer).

cs.AI↗

arXiv2010

Open Access Mandates and the "Fair Dealing" Button

We describe the "Fair Dealing Button," a feature designed for authors who have deposited their papers in an Open Access Institutional Repository but have deposited them as "Closed Access" (meaning only the metadata are visible and retrievable, not the full eprint) rather than Open Access. The Button allows individual users to request and authors to provide a single eprint via semi-automated email. The purpose of the Button is to tide over research usage needs during any publisher embargo on Open Access and, more importantly, to make it possible for institutions to adopt the "Immediate-Deposit/Optional-Access" Mandate, without exceptions or opt-outs, instead of a mandate that allows delayed deposit or deposit waivers, depending on publisher permissions or embargoes (or no mandate at all). This is only "Almost-Open Access," but in facilitating exception-free immediate-deposit mandates it will accelerate the advent of universal Open Access.

cs.DL↗

arXiv2010

Self-Selected or Mandated, Open Access Increases Citation Impact for Higher Quality Research

Articles whose authors make them Open Access (OA) by self-archiving them online are cited significantly more than articles accessible only to subscribers. Some have suggested that this "OA Advantage" may not be causal but just a self-selection bias, because authors preferentially make higher-quality articles OA. To test this we compared self-selective self-archiving with mandatory self-archiving for a sample of 27,197 articles published 2002-2006 in 1,984 journals. The OA Advantage proved just as high for both. Logistic regression showed that the advantage is independent of other correlates of citations (article age; journal impact factor; number of co-authors, references or pages; field; article type; or country) and greatest for the most highly cited articles. The OA Advantage is real, independent and causal, but skewed. Its size is indeed correlated with quality, just as citations themselves are (the top 20% of articles receive about 80% of all citations). The advantage is greater for the more citeable articles, not because of a quality bias from authors self-selecting what to make OA, but because of a quality advantage, from users self-selecting what to use and cite, freed by OA from the constraints of selective accessibility to subscribers only.

cs.CY↗

arXiv2009

Hierarchies in Dictionary Definition Space

A dictionary defines words in terms of other words. Definitions can tell you the meanings of words you don't know, but only if you know the meanings of the defining words. How many words do you need to know (and which ones) in order to be able to learn all the rest from definitions? We reduced dictionaries to their "grounding kernels" (GKs), about 10% of the dictionary, from which all the other words could be defined. The GK words turned out to have psycholinguistic correlates: they were learned at an earlier age and more concrete than the rest of the dictionary. But one can compress still more: the GK turns out to have internal structure, with a strongly connected "kernel core" (KC) and a surrounding layer, from which a hierarchy of definitional distances can be derived, all the way out to the periphery of the full dictionary. These definitional distances, too, are correlated with psycholinguistic variables (age of acquisition, concreteness, imageability, oral and written frequency) and hence perhaps with the "mental lexicon" in each of our heads.

cs.CL↗

arXiv2009

COMMENTARY ON: Citing and Reading Behavours in High-Energy Physics (arXiv:0906.5418)

Evidence confirming that OA increases impact will not be sufficient to induce enough researchers to provide OA; only mandates from their institutions and funders can ensure that. HEP researchers continue to submit their papers to peer-reviewed journals, as they always did, depositing both their unrefereed preprints and their refereed postprints. None of that has changed. In fields like HEP and astrophysics, the journal affordability/accessibility problem is not as great as in many other fields, where it the HEP Early Access impact advantage translates into the OA impact advantage itself. Almost no one has ever argued that Gold OA provides a greater OA advantage than Green OA. The OA advantage is the OA advantage, whether Green or Gold.

cs.DL↗

arXiv2009

On Fodor on Darwin on Evolution

Jerry Fodor argues that Darwin was wrong about "natural selection" because (1) it is only a tautology rather than a scientific law that can support counterfactuals ("If X had happened, Y would have happened") and because (2) only minds can select. Hence Darwin's analogy with "artificial selection" by animal breeders was misleading and evolutionary explanation is nothing but post-hoc historical narrative. I argue that Darwin was right on all counts.

cs.NE↗