EFTA01075547.pdf

REPORTS abwdance (3). We normalized the AD values to 'fre- 20. J. H. Xin, 8. P. Brandhont, R. J. Britten, E. H. Davidson, Shared worm genes were defined as those that had quency' values by referring the AD values on each chip Dev. Slat. 89. 527 (1982). significant homology to one or more genes in yeast to a calibration curve constructed from the AD values 21. S. A. Cherritz et at, Science 282, 2022 (1998). or fly (BLASTP expectation values E < 1 x 10 bk, as for the 11 control transaipts with known abundances 22. G. M. Rubin et at, Science 287, 2204 (2003). described by Rubin et at) Discrepancies between that were spiked into each hybridization (9). This 'fre- 23. To classify ?KOMI genes, we used the sequence com- these numbers of ORFs and temperable counts in quency normalization" allowed comparison of tran- parison results of Rubin et at (22). In the period Rubin et at are accounted for by the facts (I) that 2% script measurements across multiple way experiments. between the date of our chip design (early 1999) and of worm ORFs were not monitored directly by the Frequency values for each gene %we expressed in num- the date of this more recent sequence comparison arrays and (ii) that to be conservative, we excluded ber concentrations (transcripts per million or ppei) (early 2000). some of the worm ORFs that had been from consideration ORFs that had either been delet- under the assumptions described (9). included in the chip designs had been deleted or ed or altered in the Worrnpep database in the interval 13. P. Tamayo et at, Proc. Nett Aced. Sc). U.SA. 96, altered by worm sequence curators (for details on the between the chip design and the sequence compari- 2907 (1999). °Nation of the Wcempep database, see http://vemy son of Rubin et at. 14. Supplemental information is available at www sangersc.uk/Projects/C_elegans/wcempept). To rec- 24. We thank M. Whitley fee advice on array experi- sciencernagorg/feature/data/1053496.shl oncile worm ORFs in the sequence comparison with ments; K. Griffiths fee array bioinformatics support E. 15, L L Johnstone. J. D. Barry, EM8O). 15. 3633 (1996). worm ORFs on the arrays, we limited our analysis to 16. M. C Costanzo et at. Nucleic Acids Res. 28, 73 worm ORFs on the arrays that were unchanged be- Wilson A. Velasco, and H. Horton for technical as- sistance; J. Freeman for bioinformatics support during (2000 The Proteome WormPD database is available tween the two data sets, using creation lists main- the chip design protest G. Sherlock for sharing yeast at tained at the Sanger Centre (the Wonnpephistory 17. C. E. Rocheleau et at, Cell 90, 707 (1997). file). Thus, we identified on the arrays 2905 worm and worm sequence comparison dattc 11. Yandelt for sharing yeast, fly, and worm sequence comparison 18. P. W. Carter*, J. M. Roos, K. J. Kemphues, Mot. Gen. genes shared between yeast, worm and fly (core data; and D. Moots for providing purified oocytes. Genet. 221, 72 (1990). genes); 3150 warn genes shared between worm and 19. Additional details are available as supplemental in- fly, but not yeast (animal genes); and 8741 worm formation (II). genes that were unique to the worm (worm genes). 26 June 2003; accepted 29 September 2000 Song Replay During Sleep and forebrain song system. During singing, RA neurons exhibit short bursts of activity, whose identity varies with the note that im- Computational Rules for mediately follows the burst (4). In awake birds, outside the context of vocalizations, Sensorimotor Vocal Learning RA neurons are regularly firing. RA neurons also prominently burst "spontaneously" and Amish S. Dave and Daniel Margoliash. respond to sounds, but only during sleep (5). With the goal of comparing motor, auditory, Songbirds learn a correspondence between vocal-motor output and auditory and ongoing bursting activity, we recorded feedback during development. For neurons in a motor cortex analog of adult single neurons in the RA of singing male zebra finches, we show that the timing and structure of activity elicited by the zebra finches, permitted the animals to fall playback of song during sleep matches activity during daytime singing. The asleep by turning off the lights, and then motor activity leads syllables, and the matching sensory response depends on tested the sante neurons' sensory and ongoing a sequence of typically up to three of the preceding syllables. Thus. sensori- discharge properties (6. 7). motor correspondence is reflected in temporally precise activity patterns of The spiking patterns of RA neurons in single neurons that use long sensory memories to predict syllable sequences. singing birds consisted of phasic patterns of Additionally, "spontaneous" activity of these neurons during sleep matches premotor excitation superimposed over a their sensorimotor activity, a form of song "replay." These data suggest a model background of profound inhibition (4) (Fig. whereby sensorimotor correspondences are stored during singing but do not I, B and C). This premotor activity was modify behavior, and off-line comparison (e.g., during sleep) of rehearsed motor virtually invariant for multiple occurrences of output and predicted sensory feedback is used to adaptively shape motor the same sound. After the lights were turned output. off, RA auditory responses were initially weak but gained strength with time, reflect- In reinforcement learning, systems learn cle dynamics) is of high dimensionality (a ing the gradual transition into sleep (5). Re- through interaction with the environment by many-to-many dynamic mapping). Methods sponses to playback of the bird's own song trying to optimize some measure of perfor- developed in the field of machine learning (BOS) also consisted of phasic patterns of mance. Biological systems may experience a solve the problem of reinforcement learning excitation separated by inhibition that were substantial delay between prentotor activity with delayed reward (2), and a variety of similar for multiple occurrences of the same and assessment of performance through sen- biological solutions have been proposed to sound, differing mainly in the strength of sory feedback (I). This delay poses the prob- the problem of learning sequences of actions response rather than pattern (8). lem of how to reward or punish a premotor (3). Here, we report on neuronal data that The timing of auditory responses to the circuit when that circuit is participating in a represent a solution to the problem of senso- BOS was very well aligned to the timing of different task by the time the reward or pun- rimotor mapping in the bird vocal-motor premotor activity (Fig. IF). The only excep- ishment is computed. Reinforcement learning ("song") system. The physiological proper- tions were instances of silence following the is further complicated in systems such as ties observed during sleep also suggest an end of a motif or the end of song, where the vocal learning, where the mapping of sensory algorithmic implementation for reinforce- auditory response could include an additional feedback (fundamentally represented as fre- ment learning of song. burst that corresponded with the syllable that quency versus time) onto motor output (mus- Zebra finch songs are organized hierarchi- would have followed if the song had continued cally, with one or more notes composing a without pause. To compare motor and auditory Department of Organismal Biology and Anatomy, syllable, and sequences of syllables forming a activity, we analyzed the singing-related activ- University of Chicago, 1027 East 57 Street, Chicago, IL motif, which are repeated to form song. We ity surrounding each syllable of sang (4, 9). The 60637, USA. investigated neurons in the forebrain nucleus spike patterns from the response to the BOS • To whom correspondence should be addressed. E- robustus archistriatalis (RA), whose descend- playback were then compared with the spike mail: dantebigbird.udskago.edu ing projections represent the output of the patterns from premotor activity derived from 812 27 OCTOBER 2000 VOL 290 SCIENCE www.sciencemagorg EFTA01075547  REPORTS Fig. 1. (A) Schematic of the song system. Audi- A E tory and premotor activity converge onto the HVc The HVc projects directly to the RA, which projects to brainstem motor centers. The HVc also projects to area X, which projects to the DIM. The DIM projects to the IMAN, which projects to the RA. Feedback loops arise from the I A B C DE FGH A B C RA and IMAN. (B and C) Activity of RA single neurons during singing is premotor (I.e., neuronal activity leads syllables). Spectrographs of the sound that the bird produced are shown in a color scale (frequency, 0 to 10 kHz, is on the ordinate; time is on the abscissa). Corresponding 500ms raw traces of the neural activity (amplitude ver- sus time) are shown below the spectrographs. Data from one neuron for two similar examples of singing are shown in (B). The neuron's activity patterns are the same for the two examples of singing, except where the vocalizations differ, and the difference in neuronal discharge (at IA B COE 1IK B arrow) precedes the difference in the vocaliza- tions. Both sequences of vocalizations occurred frequently; the neuronal pattern associated with each sequence was stereotyped. In (C) [different II II I I IIIIII neuron, same bird as in (8)1, the bird produced syllable "C (marked by arrows) twice (each syllable is identified by a letter), with the song ending prematurely after the second occurrence. C The neuronal discharge following the second C was affected. Activity during calling (not shown) • also clearly demonstrates that RA activity leads vocalizations [see also (4)). (D) Cross-correlation 000ms of auditory and motor activity for the first neu- ron shown in (E) (positive time shifts imply that 11411r A I C F auditory response lags premotor activity). (E) Examples of the match between auditory and 200ms motor activity from one neuron in each of two D 0.6 birds. The spectrographs show the BOS used as stimuli during playback experiments, and each 0.5 41 4 111. 1 I ii I syllable is identified by a letter. The rasters marked "Aud." represent the neuron's auditory response during playback while the birds were OA 0.3 I / 1 4 t ill 01 I asleep. The rasters marked "Mot." were con- 0.2 structed from neuronal activity of the same -100 -60 0 50 100 neurons during singing (4). The correspondence Auditory Lag (ms) 200ms between the two patterns of activity is visually striking. In contrast to singing, however, during song playback, neurons exhibited ongoing discharge, not inhibition, for some syllables. (F) Example of raw traces showing the match between activity during playback and singing [same neuron as in (B)). AN Fig. 2. Deletion experiments. (A) The bird was 7 presented songs during sleep. Below the spec- 2 4 trograph of the last motif of the bird's song are Target Syllable • 8 nine histograms of the response of one neuron, I A B C DE FGH D 1 5 6 2 7 representing 30 repetitions each of nine differ- 8 h I...1 32.s E 1 6 7 6 8 ent stimuli. The BOS histogram is the response to the unaltered motif. For each of the eight La -A F 1 6 7 8 9 other stimuli, one of the syllables from A to H was replaced with background noise. For syllable G 1 1 3 9 F, for example, the neuron responded with two 6 At I s bursts, with both bursts occurring during syllable F. The first burst (but not the second) is statis- A B C D E F G all I -C tically significantly reduced (8) by the elimina- Deleted Sy table tion of syllables C, D. E. or F. The second burst is affected by the elimination of syllable F. The burst at syllable H is affected by eliminating syllables F, G, and H. (B) in one bird with the most oa.s.Aassrairsallaela.....-a.satual complex song, 10 neurons were tested with deletion stimuli. Each cell of the matrix gives the I. 1 -E number of neurons in which the deletion of a syllable (specified by the column) significantly altered the response during the target syllable (specified by the row). The last syllable of the song, H, was excluded because appropriate control data were unavailable (an earlier syllable had always been deleted). The matrix diagonal represents the effect of deleting a syllable on the neuronal response during that same syllable. The numbers to the right of the diagonal are the number of neurons for which there were a statistically significant response during the target syllable. It can la ` 1 be seen that the deletion of a syllable commonly affected the neuronal response several syllables later. For example, of eight neurons responding to syllable E, the response was suppressed for one, 200 ms six. seven, and six neurons when syllables B, C, D, and E, respectively, were deleted. wwwsciencemag.org SCIENCE VOL 290 27 OCTOBER 2000 813 EFTA01075548  REPORTS the corresponding syllables, showing that the The auditory activity was only slightly sensed the effect on the neuronal activity timing of excitation and inhibition during audi- delayed in relation to motor activity (by 8 -± during the same or subsequent syllables (11). tory stimulation was well aligned to such tinting 2 ms; range, 4 to 13 nu). Because premotor The deletion of a syllable substantially re- during singing (Fig. I, D and E). A cross- activity in RA can lead the onset of syllables duced the neuronal activity occurring one to correlation procedure revealed a strong, signif- by up to —40 ms (4), this was surprising and three syllables later (Fig. 2A), up to —250 ins icant (P < 0.02) correlation (10) between pre- suggested that the sensory patterns represent- (8). This property was ubiquitous for all RA motor and sensory spike patterns in all 17 neu- ing subsequent syllables were generated by neurons that were auditory (14 neurons from rons (from three birds) (mean normalized peak responses to previous syllables. To character- three birds) (12) (Fig. 2B). These response correlation = 0.49 1- - 0.13 SD). Thus, sensori- ize the extent of temporal integration in the properties are suggestive of temporal combi- motor transformations in the song system result auditory responses, we presented stimuli in nation sensitivity, in which a sensory neu- in a correspondence between temporally pre- which a syllable chosen at random front the ron's response is nonlinearly dependent on cise sensory and motor activity observed at the final motif of the BOS was substituted by a the temporal sequence of preceding syllables. level of individual cells. background of equal duration, and we as- Such responses have also been described for neurons in the nucleus HVc, which projects to the RA (13). Thus, in the RA as well as in A the liVc (4, 13), the integration time of indi- vidual neurons appears to be considerably Set Sate tAllk greater when in the sensory (auditory) state than during singing. Given the alignment of C D E F G H auditory and motor activity in the RA, one way of interpreting these results is that audi- tory responses to song syllables represent a prediction of subsequent premotor activity. l ilt We also searched for similarities between ongoing bursting activity during sleep (5) and the sensorimotor patterns of RA neurons. For each cell, a visual inspection of samples of activity from long stretches (15 to 60 min) of undisturbed sleep identified repeated exam- ples of one or more complex burst patterns, E F suggestive of the patterns that we had ob- C D served in the cell's pre:motor activity. To h - -Rd 4 quantify this match, we developed a proce- dure to automate burst detection (14), con- sidering only bursts of eight spikes or more to t NV Ilil 1 100ms ease the computational burden and to allow for statistical analysis. By this procedure, 7.1 ± 5.3% of all spikes (14 neurons from B three birds) occurred in bursts, an average of 175.4 ± 144.6 (range, 38 to 581) bursts per AS OW ire, ims Ow- cell. For each cell, a measure of similarity between each burst and the single longest A B C A B bout of the cell's premotor activity (4 to 8 s, consisting of several motifs or songs) was ill-IrkRh ,4-11i1\ \w l l d -1-41I ,4111 computed and tested for significance (15, 16). The results showed that 15.3 ± 6.5% of I bursts (range, 2.6 to 26.8%) significantly 11 Ilil Mil matched prentotor activity. Only the cell with 2.6% matching bursts failed to exceed the 5% level expected by chance (16). Examples of matches between longer sequences of com- plex ongoing bursts and premotor activity were particularly compelling (Fig. 3A). In an exceptional case when two RA neurons were recorded simultaneously from different elec- 100m8 trodes during sleep, both neurons commonly Fig. 3. Neuronal replay during undisturbed sleep. (A) Raw traces of neuronal activity (900 ms) exhibited simultaneous bursting, with the dif- during sleep ("Spoil in two different neurons for one bird. For each sample, a representative ferent burst patterns for each neuron corre- corresponding sample of premotor activity ("Mot.") and a color spectrograph of the song that the sponding to the same sequences of syllables bird sang are shown. (B) Raw traces (1400 ms) of simultaneous recordings from two neurons (-400 (Fig. 3B). This suggests that populations of µm apart) in another bird. (The second neuron's activity is visible in the background of the first RA neurons burst in a coordinated fashion neuron's signal, an artifact of the pairing of signals used to achieve differential recordings resistant during sleep. Bursts (and matching bursts) to movement-induced artifacts.) Both neurons simultaneously burst during sleep, with complex burst structures that match premotor activity. Apparent temporal expansion (first motif: A, B, and preferentially occurred during periods when C) and compression (second motif: A, B, and C) is highlighted by the blue lines. This phenomenon the rate of ongoing discharge was lower and has also been reported in population activity of hippocampal neurons (23). more variable (Fig. 4). Such modulation may 814 27 OCTOBER 2000 VOL 290 SCIENCE www.sciencemag.org EFTA01075549  REPORTS correspond to specific phases of the sleep mechanises, whereby during singing, IIVc ac- References and Notes cycle. tivity in response to auditory feedback from 1. In songbirds, there is a minimum delay of —70 ms or longer between the production of a burst of In the sensorimotor phase of vocal learn- sequences of syllables is delayed through the neuronal activity contributing to song generation ing, the mapping between auditory feedback AFP to produce a prediction of activity in the and the reception of processed auditory input from and vocal output is the fundamental compu- RA of a subsequent syllable. The data collected the resulting vocalization. In zebra finches, sylla- bles are typically SO to 200 ms in duration and may tational problem to be solved (17). A solution during sleep, however, also suggest "off-line" comprise one or several notes. The minimal delay is to this problem is reflected in the sensorimo- models for learning that address the problems the sum of premotor lead (-50 ms) and sensory tor activity patterns of RA neurons. Precision of feedback delay and sequence generation. lag. Sensory lag is the sum of the minimal latency for response (-20 ms) and the sensory integration of spike timing has been observed in a num- Such models share some similarities with tem- time, which, for song system neurons, can be tens ber of systems and provides evidence for poral-difference models of reinforcement learn- to hundreds of milliseconds. temporally based neural codes in sensory pro- ing and sequence generation that are prominent 2. R. S. Sutton, A. G. Berta, Reinforcement Learning: An Introduction (MIT Press, London, 1998). cessing, although only in a few cases has the in mammalian work on basal ganglia and the 3. R. E. Sur!, W. Schultz. Exp. Brain Res. 121, 350 (1998); behavioral relevance been directly demon- cerebellum, in that they reward or modify the D. G. Beiser. J. C. Houk, f. NewrophysieL 79, 3168 strated (18). The observed correspondence system on the basis of its overall performance, (1998); G. S. Berns, T. J. Sejnowski, f. foga. Nearest). between auditory activity and vocal output not on the basis of the performance of individ- 10, 108 (1998); J. Brown, D. Bullock, S. Grossberg f. Neumset 19, 10502 (1999). demonstrates that, in the RA, sensorimotor ual components or movements (3). The AFP 4. A C. Yu, D. Margoliash, Science 273, 1871 (1996). mapping is based on a temporal code. This has been likened to a mammalian cortienlywkl 5. A S. Dave, A C Yu. D. Margoliash, Science 282, 2250 correspondence is likely to arise from audi- ganglia-thalarnocortical loop (20, 21). (1998) 6. All neuronal data reported in this report are from tory input recruiting similar components of In the vocal learning model motivated single units. Eighteen neurons (10. 5, and 3) from the RA pattern-generating circuits as those by the present data, signals that arise in the three birds contributed to the data reported herein. recruited during singing. In the hierarchical RA during singing train the AFP to gener- Two additional neurons from a fourth bird exhibited a match between auditory responses and ongoing organization of the song system (4, 19), the ate a prediction of auditory feedback; dur- bursts. These data were the genesis for this study but sensorimotor correspondence may first ing sleep rehearsal, the AFP's predicted did not include premotor data. emerge at the single-cell level within the RA. feedback provides reinforcement to RA 7. The chronic recording procedures and conduct of the experiments are described in detail on Science Online The data suggest that, during vocal develop- neurons. During singing, sensorimotor ef- (24). ment, the song system learns to generate pre- ference copy signals (premotor output and 8, Additional procedures and data regarding the audi- motor commands by association with a pre- expected auditory feedback) traverse the tory responses of the netecos are described on Sci- ence Online (24). diction of future commands based on the AFP, and via the lateral subdivision of the 9. The pattern of RA neuronal activity depends on tinting of auditory feedback from preceding magnocellular nucleus of the anterior neo- notes. which are constituents of syllables. By defini- syllables. This can be interpreted as learning striatum (IMAN) projection onto area X, tion, the same pattern of notes is repeated for a given the match between the auditory response to a are compared with real auditory feedback syllable type. We analyzed RA activity at the syllable level to ease the analysis and to minimize the number sequence of syllables with the premotor pat- arriving in area X from the HVc (Fig. IA). of deletion stimuli required in subsequent experi- tern for a subsequent syllable or as learning Efference copy is brought into temporal ments. FolloiMng established procedures, we identi- the match between the prediction ofa sensory register with auditory feedback using the fkd each syllable that the bird sang, and the times of asset and offset were manually determined from representation of a syllable with the premotor long (-50 ms) synaptic delays observed in spectrographs and oscillographs of the acoustic sig- representation of the same syllable. the medial subdivision of the dorsolateral nal. The spectrographs of all exemplars for each In the birdsong system, RA receives input nucleus of the thalamus (DLM) (21). This syllable type were cross-correlated to establish opti- mal time shifts; these were then applied to the from the IIVc and from an anterior forebrain stimulates area X neurons that are sensitive corresponding spike bursts. The aligned time-shifted pathway (AFP) (Fig. IA). Sensorimotor song to temporally coincident input. The output spikes were the basis for further analysis. learning could result in part from "online" of the IMAN onto the RA has a reduced 10. Peak cross-correlation values were tested for signifi- cance by using a bootstrap procedure, described in effect because the IMAN is not in temporal detail on Science Online (24). register with driving input from the HVc. 11. Multiple stimuli were derived from the 8O5, one for During sleep, replay of song premotor pat- each syllable deleted from the last complete motif of the SOS; these were presented in one block during terns via ongoing bursting generates coher- the night while the bird was asleep. Deleted syllables ent activity throughout the song system that were replaced with samples from silent intervals is similar to singing in the absence of actual between motifs or syllables. Spike rates were com- sound production and perception. The out- puted over the duration of each syllable. For each syllable from the second to the penultimate, we put of the IMAN represents a prediction of compared the spike rates over the interval of the the real auditory feedback that would have target syllable when a previous syllable or the target resulted from the burst-generated motor syllable itself was deleted (experimental data) against the spike rates for the target syllable when command, is in near coincidence with IlVc only a subsequent syllable was deleted (control data), bursts driving the RA, and is used to mod- using a Mann-Whitney U test at the 95% confidence ify RA neurons that are sensitive to tempo- level The last syllable in the song was excluded because, by definition, control data were not Orval- rally coincident input. able. (The response to the SOS. presented at the The proposed algorithm for birdsong beginning of the night's recordings as the bird tran- learning depends on circadian modulation of sitioned into sleep, showed fluctuations in the Time (min) strength of response but not in the timing of spikes. neuronal activity patterns (22). Our observa- Thus, it was appropriate for correlation testing of Fig. 4. (Top) The firing rate during recordings of tion of neuronal replay of sensorimotor pat- sensorimotor comparisons but not for a comparison RA ongoing activity over almost 1 hour of sleep, terns during sleep is consistent with data front with spike counts of deleted syllable stimuli.) estimated from a 100-point moving average of 12. The three birds presented with deletion stimuli had hippocampal studies suggesting that sleep is the interspike intervals (there was a gap in the songs with eight, five, and three syllables per motif. important for the consolidation of neuronal All 10 neurons tested in the eight-syllable bird (Fig. data collection of —3.25 min). (Bottom) A histo- temporal codes for spatial memory (23, 24). 28) and all 3 neurons tested in the three-syllable gram (30-s bins) of the number of bursts identi- bird showed a loss of excitatory response when the fied by a burst-finding procedure. The number of The fundamental prediction of our model is target or a previous syllable was deleted. One of bursts that significantly matched the premotor that birdsong learning depends on sleep or two neurons tested in the five-syllable bird showed activity is shown in blue. other off-line computations. no response to any syllable of the 8O5; the other wwwsciencemag.org SCIENCE VOL 290 27 OCTOBER 2000 815 EFTA01075550  REPORTS neuron responded at the second syllable, and the with the greatest number of recordings, we also 19. E. T. vu, M. E. Mazur& Kuo, J. Neurosct 14, response was suppressed by deletion of the first or compared each neuron's ongoing discharges during 6924 (1994). second syllable. sleep with the premotor data from other neurons. 2a S. W. Bottler, F. Johnson, J. Neurobiot 33, 602 11 D. Margolies/O. Necutod. 3. 1039 (1983); D. Mar- For seven out of eight neurons, there were more (1997). goliash, E. S. Fortune, J. Neustod. 12, 4309 (1992). matches with the neuron's own cremator data 21. M. Luo, D. J. Perkel, J. Neurosc). 19. 6700 (1999). 14. We gathered sufficient data from 14 neurons (from than with the premotor data from the other neu- 2Z G. E. Hinton, P. Days; 8. J. Frey, R. M. Neal. Science three birds) to permit quantitative comparisons be- rons. The same neuron failed to achieve signifi- 268. 1