Tuesday, August 30, 2005
Synfire chains and cortical songs -- authors report long, repeatable with millisecond accuracy chains of neuron activations, which they call "cortial songs".
Monday, August 29, 2005
Modeling STDP
Spike timing dependent plasticity (STDP). These guys come up with a biophysical model for STDP and do some modeling in C++, but I didn't think the solution was very useful from a software standpoint.
Spike timing dependent plasticity (STDP). These guys come up with a biophysical model for STDP and do some modeling in C++, but I didn't think the solution was very useful from a software standpoint.
Polychronization: computation with spikes
A very interesting read; Not really about computing with spikes (yet), just running-it-and-seeing-what-happens. The main find is that if edge (axon) delays are introduced, certain neurons start ringing in a loop, because edge weights sum up to a value that causes the loop to self-sustain. Another result is that the number of such groups can be much greater than even the number of synapses in the network.
"In our view, attention is not a command that comes from the "higher" or "execute" center and tells which input to attend to. Instead, we view attention as an emerging property of simultaneous and regenerative activation (via positive feedback) of a large subset of groups representing a stimulus, thereby impeding activation of other groups corresponding to other stimuli"
A very interesting read; Not really about computing with spikes (yet), just running-it-and-seeing-what-happens. The main find is that if edge (axon) delays are introduced, certain neurons start ringing in a loop, because edge weights sum up to a value that causes the loop to self-sustain. Another result is that the number of such groups can be much greater than even the number of synapses in the network.
"In our view, attention is not a command that comes from the "higher" or "execute" center and tells which input to attend to. Instead, we view attention as an emerging property of simultaneous and regenerative activation (via positive feedback) of a large subset of groups representing a stimulus, thereby impeding activation of other groups corresponding to other stimuli"
A scalable cortical simulation framework
These guys can run 35,000 neurons and 6.1 million synapses per node, and the number of nodes are scalable. Performance scales linearly with nodes (assuming you don't have N^2 connectivity!)
These guys can run 35,000 neurons and 6.1 million synapses per node, and the number of nodes are scalable. Performance scales linearly with nodes (assuming you don't have N^2 connectivity!)
Sunday, August 28, 2005
Which model to use for cortical spiking neurons?
A good overview of the various types of spiking and bursting neurons observed in nature.
The author Izhikevich also proposes a neuron model which takes only 13 flops to implement, while having biologically realistic behavior. The guy's web page is here : http://www.izhikevich.com
Here is another article, analyzing the various possible spiking/bursting behaviors with really nice and intuitive 2-d illustrations of their behavior:
A good overview of the various types of spiking and bursting neurons observed in nature.
The author Izhikevich also proposes a neuron model which takes only 13 flops to implement, while having biologically realistic behavior. The guy's web page is here : http://www.izhikevich.com
Here is another article, analyzing the various possible spiking/bursting behaviors with really nice and intuitive 2-d illustrations of their behavior:
Saturday, August 27, 2005
"When the presynaptic input arrives to assist the postsynaptic neuron to discharge action potentials, LTP (long-term potentiation) is observed, and when the presynaptic input arrives after the postsynaptic neuron discharges, LTD (long-term depression) is induced, revealing a causal-reward/acausal-punishment. This form of plasticity is referred to as spike-time-dependent plasticity (STDP)"
(SOB,p533)
(SOB,p533)
"One theory holds that during theta (exploratory) activity (5-10hz), the hippocampus is acquiring a new representation of its environment, whereas during sharp-wave (quiet) activity (and also during slow-wave sleep), the hippocampus is facilitating the consolidation of this information in the form of long-term memories elsewhere in the cortex"
(SOB, p497)
(SOB, p497)
Friday, August 26, 2005
"An analysis of the circuits of other cortical structures such as the olfactory cortex and hippocampus reveals that they, too, bear many resemblances to the circuits of the neocortex. Thus, it is tempting to suppose that there may be some common basic principles that underly the organization and operation of all cortical circuits..."
(The synaptic organization of the brain, p556)
(The synaptic organization of the brain, p556)
"Presently the most intensely studied hypothesis is that this binding (between neurons) is accomplished by temporal correlation, a synchronization of the neural activity in even widely separated nodes that constitutde parts of the whole" [PNCC, p278]
"Every neuron in every neocortical microcircuit receives direct synaptic inputs from the cortically projecting modulatory systems of brainstem and basilar forebrain origin." [Mountcastle, Perceptual Neuroscience: The Cerebral Cortex (from now on PNCC)]
Which means that there exists control for exciting & inhibiting sections of the brain.
Which means that there exists control for exciting & inhibiting sections of the brain.
"Helmholtz contended that this seamless flow is accomplished by referring the central displays of sensory stimuli to "mental constructs" of the world and events within it. These constructs are thought to be generated by past experience and to be stored in and readily recalled from memory. Perceptions are then thought to be produced by the comparison of recalled and evoked neural images and perceptual identification inferred by the likeness between them, the simplest and most appropriate recalled construction winning the day. On this hypothesis, perhaps what we perceive are patterns of neural activity recalled from memory for the matching operation, rather than the activity evoked directly by sensory stimuli themselves."
(Mountcastle, Perception and the Cerebral Cortex)
(Mountcastle, Perception and the Cerebral Cortex)
Monday, August 22, 2005
"The main purpose of language is to provide information about who does what to whom"
But isn't that just a different way of saying that sentences are primarily of the form "subject verb object"? It's just one of the possible forms that a valid sentence can take. Consider the sentence "if you're OK with it, fuck off". What form does it have? It's a modal-imperative, but does it even make sense to categorize it? Its meaning is in the effect.
It seems to me that the reason that NLP (natural language processing) is so difficult to do is because the grown-up human language is really a scripting language -- the grammar itself is not that difficult (every major language on earth has had parsers written for it), but that it shamelessly utilizes all the other faculties that are so easily avaliable to a human -- sound, vision, general intelligence.
Try porting a Windows app to another platform. You'll find that it relies on a bunch of uniniteresting in themselves, but difficult-to-reproduce features. The primary purpose of language, is of course, communication, and the meaning of communication is always measured by what effect it has on the recepient. To simulate the proper effect on an NLP program, that program would have to have much more than a parser and a wordnet.
Wordnet
But isn't that just a different way of saying that sentences are primarily of the form "subject verb object"? It's just one of the possible forms that a valid sentence can take. Consider the sentence "if you're OK with it, fuck off". What form does it have? It's a modal-imperative, but does it even make sense to categorize it? Its meaning is in the effect.
It seems to me that the reason that NLP (natural language processing) is so difficult to do is because the grown-up human language is really a scripting language -- the grammar itself is not that difficult (every major language on earth has had parsers written for it), but that it shamelessly utilizes all the other faculties that are so easily avaliable to a human -- sound, vision, general intelligence.
Try porting a Windows app to another platform. You'll find that it relies on a bunch of uniniteresting in themselves, but difficult-to-reproduce features. The primary purpose of language, is of course, communication, and the meaning of communication is always measured by what effect it has on the recepient. To simulate the proper effect on an NLP program, that program would have to have much more than a parser and a wordnet.
Wordnet
Amazon.com: Books: On Intelligence
The premise of this book is that the brain barely does any computing at all, it just looks things up in its huge memory. Oh yes, and it also predicts, but only one step ahead. What's meant by prediction is that the brain knows where it is inside of any number of sequences, which span all scales in distance, time, and logical hierarchy, and expects the next item in the sequence by preemptively firing a group of cells. Each group of cells thus logically represents a "name" for an item in the sequence. This behavior is assumed to be the same for vision, hearing, touch, proprioception, and language (yes, language).
But wait, more simplifications are on the way. The cortex has a uniform structure, so Hawkins conjectures that it starts as a blank slate, acquiring all of its features simply by analyzing sensory data. This is where Hawkins runs into trouble, in my opinion. He seems to be woefully ignorant of language. If the cortex started out blank and simply feature-detected its way to general intelligence, we would expect to find a more randomized distribution of various cortical modules. In particular, there would no reason to expect someone to be able to construct complicated, recursive sentences by the age of five; it would be a high school subject, like algebra and programming. We would also expect people to handle center-embedded sentences with the same ease as right-branched sentences.
onintelligence.org forums attract crackpots, but what did Hawkins expect, preaching revolutionary ideas about how general intelligence to the uneducated public?
"When you see, feel, or hear something, the cortex takes the detailed, highly specific input and converts it to an invariant form. It is the invariant form that is stored in memory." Great, forming invariant representations is only the biggest question in vision.
"The next higher region recognizes sequences of phonemes to create words. The next higher region recognizes sequences of words to create phrases, and so on." -- You really have to demonstrate a mechanism for generating phrases and reconcile it with existing knowledge about language on planet Earth. Dismissing all of linguistics with "create phrases, and so on" exposes a profound ignorance of what language really is.
"Consciousness is what it feels like to have a cortex" -- How blase
The premise of this book is that the brain barely does any computing at all, it just looks things up in its huge memory. Oh yes, and it also predicts, but only one step ahead. What's meant by prediction is that the brain knows where it is inside of any number of sequences, which span all scales in distance, time, and logical hierarchy, and expects the next item in the sequence by preemptively firing a group of cells. Each group of cells thus logically represents a "name" for an item in the sequence. This behavior is assumed to be the same for vision, hearing, touch, proprioception, and language (yes, language).
But wait, more simplifications are on the way. The cortex has a uniform structure, so Hawkins conjectures that it starts as a blank slate, acquiring all of its features simply by analyzing sensory data. This is where Hawkins runs into trouble, in my opinion. He seems to be woefully ignorant of language. If the cortex started out blank and simply feature-detected its way to general intelligence, we would expect to find a more randomized distribution of various cortical modules. In particular, there would no reason to expect someone to be able to construct complicated, recursive sentences by the age of five; it would be a high school subject, like algebra and programming. We would also expect people to handle center-embedded sentences with the same ease as right-branched sentences.
onintelligence.org forums attract crackpots, but what did Hawkins expect, preaching revolutionary ideas about how general intelligence to the uneducated public?
"When you see, feel, or hear something, the cortex takes the detailed, highly specific input and converts it to an invariant form. It is the invariant form that is stored in memory." Great, forming invariant representations is only the biggest question in vision.
"The next higher region recognizes sequences of phonemes to create words. The next higher region recognizes sequences of words to create phrases, and so on." -- You really have to demonstrate a mechanism for generating phrases and reconcile it with existing knowledge about language on planet Earth. Dismissing all of linguistics with "create phrases, and so on" exposes a profound ignorance of what language really is.
"Consciousness is what it feels like to have a cortex" -- How blase
Amit, D.J. (1994). The Hebbian paradigm reintegrated: Local reverberations as internal representations.
quotes:
...Hebb's paradigm ... can be summarized as a process generating the feed-back connectivity required for maintaining reverberations (persistent spike distributions) in a local network by the activity in the same network.
...It is known anatomically, physiologically and neurologically that as one proceeds along the elaboration path in the cortex, one always finds back projections, as far back as into the primary sensory areas. On the other hand, it is a very familiar experience to have a given sensory power notably improved when the content of the observed stimulus is known. For example, when vision is impeded by distance or haze so that a given object cannot be discerned (or read), receiving a cue as to the nature of the object (or the written text) often produces a clear perception of the target.
quotes:
...Hebb's paradigm ... can be summarized as a process generating the feed-back connectivity required for maintaining reverberations (persistent spike distributions) in a local network by the activity in the same network.
...It is known anatomically, physiologically and neurologically that as one proceeds along the elaboration path in the cortex, one always finds back projections, as far back as into the primary sensory areas. On the other hand, it is a very familiar experience to have a given sensory power notably improved when the content of the observed stimulus is known. For example, when vision is impeded by distance or haze so that a given object cannot be discerned (or read), receiving a cue as to the nature of the object (or the written text) often produces a clear perception of the target.
