Psychosomatics 47:471-478, December 2006
doi: 10.1176/appi.psy.47.6.471
© 2006 Academy of Psychosomatic Medicine
Phenomenology of Delirium in Cancer Patients, as Described by The Memorial Delirium Assessment Scale (MDAS) and The Delirium Rating Scale (DRS)
Marco Bosisio, Psy.D.,
Augusto Caraceni, M.D., and
Luigi Grassi, M.D.
Received October 25, 2005; revised January 13, 2006; accepted January 20, 2006. From the Psychology Unit, National Cancer Institute of Milan (Italy). Send correspondence and reprint requests to Dr. Marco Bosisio. e-mail: marco.bosisio{at}istitutotumori.mi.it
the Italian Delirium Study Group (IDSG)
|
ABSTRACT
|
This study was based on the data collected on a consecutive sample of 106 cancer patients referred for mental status evaluation. All patients were evaluated by use of the Confusion Assessment Method (CAM) algorithm, the Delirium Rating Scale (DRS), the Memorial Delirium Assessment Scale (MDAS), and a question about the subjective perception of delirium. After comparing the diagnostic criteria of delirium on the DSM-III-R and DSM-IV, authors evaluated the ability of all DRS and MDAS items to discriminate delirium versus non-delirium patients, testing the difference in the distribution of the individual MDAS and DRS item scores. Authors also assessed the relationship between delirium diagnosis and the subjective perception of delirium. The MDAS showed a greater number of discriminating items. The items that proved to be less discriminating were "Hallucinations" and "Lability of Mood" on the DRS. Subjective perception only partially discriminated delirium from non-delirium patients. The way in which the DRS and MDAS reflected the DSM criteria are therefore partially different.
|
INTRODUCTION
|
Delirium is one of the most frequent neuropsychiatric complications found in medically ill patients, with significant impact on clinical course, recovery, length of hospitalization, and survival. For this reason, the methods for screening, diagnosing, and describing this syndrome have recently received significant attention.1–4
The ICD-10 and DSM classifications are the most frequently used diagnostic systems for delirium. The Confusion Assessment Method (CAM)5 and the Delirium Symptom Interview (DSI)6 have both been both used as operating algorithms for the diagnosis of delirium.
Among a series of other tools for the assessment of delirium, the Delirium Rating Scale (DRS)7 and the Memorial Delirium Assessment Scale (MDAS)8 have been regularly used because of their high levels of validity and reliability. Both instruments evaluate the clinical characteristics of delirium and the symptom intensity, allowing clinicians to rate delirium severity. Different levels of cut-off scores on both instruments have also been proposed in order to obtain more satisfactory levels of accuracy in the diagnosis of delirium in studies of medically ill patients.7,9,10
The aims of this study were 1) to describe the distribution of the symptoms and the characteristics of delirium, as assessed by the DRS and MDAS, among cancer patients, and 2) to assess the ability of each DRS and MDAS item to discriminate between delirium and non-delirium patients. The patients’ perception of their own cognitive impairment was also studied and compared with the clinical diagnosis.
|
METHOD
|
Subjects
As reported in more detail elsewhere,11 a series of cancer patients were recruited in six centers in Italy, including four medical oncology units and two palliative-care units. All consecutive inpatients presenting with a mental status change who were referred to the consultation–liaison psychiatry service or palliative-care unit were evaluated with a battery of tests conducted by a research psychologist trained in neuropsychological assessment and by neurological or psychiatric consultants. Patients in terminal status (prognosis of 24 hours or less), in the postoperative period, or younger than age 14 were excluded.
Psychologists who were trained in the administration of the study instruments examined patients with a structured interview and completed the CAM, the DRS, and the MDAS. Neurological or psychiatric opinion confirmed the diagnosis of delirium rendered by the CAM.
Before the study, all researchers participated in a meeting where the aims and the use of the instruments were discussed. Reliability tests were carried out by means of a videotaped case example (mean reliability criterion [ ] 0.80).11
Tools
The CAM5 is an algorithm for the diagnosis of delirium based on the assessment of acute outset of change in mental status, difficulty in attention during the interview, disorganization of thought, and alteration of the state of consciousness. The CAM has high sensitivity (94% to 100%) and specificity (90% to 95%), with a positive predictive value of 91% to 94% and negative predictive value of 90% to 100%. The CAM is valid when administered by different medical specialists, with an agreement between different interviewers (geriatricians and psychiatrists) for the presence of the delirium equal to 100%.5,12
The DRS7 is a 10-item, clinician-rated symptom rating scale developed to identify delirium and measure delirium severity in medically ill patients. Items are derived from DSM-III; they are scored from 0 to 3 or 0 to 4, and they evaluate temporal onset of symptoms, perceptual disturbances, hallucinations, delusions, psychomotor behavior, cognitive status, presence of an organic underlying pathology, sleep–wake cycle disturbances, and fluctuation of symptoms. The DRS rating is based on a 24-hour period of observation. All available information from the patient interview, mental status examination, nursing observations, and family reports tend to concur with the DRS rating. The total DRS score is obtained by summing the scores on the 10 items (range: 0 to 32). Cut-off score of 10 and 12 have been suggested to distinguish patients with delirium from patients with other neuropsychiatric diseases. The DRS scores ranged from 12 to 30 in a sample of individuals that showed symptoms of delirium. Such outcomes were correlated with the Mini-Mental State Exam (MMSE; r =–0.43) and with the Trailmaking B test (r=0.66).7 These correlations suggest a relationship between severity of delirium symptoms and severity of cognitive impairment. The DRS outcomes differentiate patients with delirium from those with dementia or schizophrenia and from subjects in a healthy-control group.7,13
The MDAS8 is specifically designed to quantify the severity of delirium symptoms. It is composed of 10 observer-rated 4-point items that were developed by the authors to be consistent with the proposed DSM-IV diagnostic criteria for delirium. It integrates objective cognitive testing and evaluation of behavioral symptoms. Repeated daily evaluations are possible so as to capture short-term fluctuations of symptoms and to document response to treatment. The MDAS yields a global score ranging from 0 to 30, with a suggested cut-off score of 13 for delirium. It includes items on level of consciousness, disorientation, short-term memory, digit span, attention, disorganized thinking, perceptual disturbances, delusions, psychomotor activity, and sleep–wake cycle disturbances. In an Italian validation study, the MDAS outcomes were correlated with the DRS (r=0.88), the MMSE (r = –0.91), and the global clinical judgment of severity of delirium (r=0.89).8,11
The subjective perception of mental confusion was investigated with two questions contained in the interview: "In the last 24 hours, did you feel confused?"; "If yes, did you feel: slightly confused, moderately confused, or extremely confused?"
Statistical Analysis
The distribution frequency of the scores of the individual MDAS and DRS items were compared among patients assessed by the CAM as delirium or non-delirium patients, by means of the Kruskal-Wallis test. The distributions were considered very probably different if p<0.001, probably different if p<0.01, and otherwise not different.
|
RESULTS
|
Sample
The study sample consisted of 106 cancer patients, of whom there were 56 men (52.8%) and 50 women (47.2%). The patients’ ages ranged from 24 to 99 years (mean: 67.7; standard deviation [SD]: 13.18 years). The primary tumor site was lung or bronchus (N=22; 20.75%), breast (N=20; 18.9%), gastrointestinal system (N=17; 16%), genitourinary system (N=17; 16%), head or neck (N=4; 3.8%), and leukemia (N=4; 3.8%). Twenty-two cases had other different primary cancer locations (20.75%). The majority of the subjects (N=77; 72.6%), were in a metastatic stage of illness, whereas the remaining 29 patients (27.4%) had local or locally advanced disease (in this group we also considered leukemia and all kinds of lymphomas).
According to the CAM criteria, 66 patients (62.3%) were termed to have delirium (33 men [31.15%] and 33 women [31.15%]), and 40 patients (37.7%) did not have delirium (23 men [21.7%] and 17 women [16%]). More specifically, among the non-delirium patients, 7 (17.5%) did not receive any psychiatric diagnosis; 13 (32.5%) presented symptoms of acute change in cognition probably due to medical factors, without fulfilling the criteria for delirium; 7 (17.5%) met the criteria for a diagnosis of dementia; 8 (20%) had a diagnosis of adjustment disorder; and 4 (10%) had longstanding cognitive symptoms not fulfilling the diagnosis of dementia. For one patient (2.5%), no diagnosis was available.
Phenomenology of Delirium
The phenomenology of delirium was separately examined according to the DRS and MDAS items. Figure 1 and Figure 2 show the distributions of each DRS and MDAS item score according to the absence or the presence of delirium diagnosis. The histograms show the level of probability that an individual item differentiates between delirium and non-delirium patients.
View larger version (36K):
[in this window]
[in a new window]
|
FIGURE 1. Distributions of DRS Items by the Absence (CAM: 0) or Presence (CAM: 1) of Delirium
DRS: Delirium Rating Scale; CAM: Confusion Assessment Method; MMSE: Mini-Mental State Exam.
Each pair of histograms shows the percentage of patients (y axis) in each group presenting the symptom according to DRS scores (x axis). The histograms also show the level of probability that the individual item differentiated between delirium and non-delirium patients.
a p<0.001; b 0.01>p0.001; c p0.01.
|
View larger version (40K):
[in this window]
[in a new window]
|
FIGURE 2. Distribution of MDAS Items by the Absence (CAM: 0) or Presence (CAM: 1) of Delirium
MDAS: Memorial Delirium Assessment Scale; CAM: Confusion Assessment Method.
Each pair of histograms shows the percentage of patients (y axis) in each group presenting the symptom according to MDAS scores (x axis). The histograms also show the level of probability that the individual item differentiated between delirium and non-delirium patients.
a p<0.001; b p<0.01.
|
Delirium Rating Scale (DRS) Data
Practically all delirium subjects presented with an acute or subacute change (ranging from 1 month to few days or less) of behavior or of personality (Figure 1, Section 1). Half the sample showed slight (feelings of depersonalization or derealization) or moderate (visual illusions or misperception) perceptual disturbances. The subjects who did not experience perceptual problems were approximately three times the number of those with serious problems, such as not being able to distinguish between dream and reality (Figure 1, Section 2).
Hallucinations were found in about half the delirium patients. Patients had only visual hallucinations, noticeable from observation of the patient or from the personal report (Figure 1, Section 3).
Delusions were not present in 60% of delirium patients. The remaining patients who experienced delusions had either not well-circumscribed, transient, and poorly organized delusions or manifested in their answers evident distortions of reality (Figure 1, Section 4). As far as psychomotor activity, 75% of cases showed mild-to-moderate abnormalities (retardation or agitation); however, the DRS does not allow for describing hypo- versus hyperactive behavior (Figure 1, Section 5).
At formal cognitive evaluation, over half of the subjects with delirium showed cognitive impairment, characterized by significant deficit in various tested areas, including frequent periods of disorientation in time and space, anomalies of memory recording and recall, and deficits in ability to concentrate. The remaining subjects were evenly distributed between a group with severe cognitive abnormalities (motor and verbal perseveration, confabulation, memory failure) and a second group with moderate deficit (in these cases only one of the main tested areas was compromised; Figure 1, Section 6).
As might be predicted, delirium patients presented more frequently with potentially specific etiological factors than non-delirium patients, although, in this cancer population, mostly with advanced disease, all delirium and non-delirium patients had a physical disorder that was potentially identifiable as at least temporarily associated with the alteration of behavior or mental state (Figure 1, Section 7).
Among delirium subjects, 70% had mild-to-moderate sleep–wake cycle disturbances such as frequent napping during the day and nighttime insomnia or, less frequently, occasional daytime drowsiness with mild sleep-continuity disturbance at night (Figure 1, Section 8).
In the study population, it was quite rare to find significant mood changes, and, indeed, mood was evaluated as stable or mildly altered in most cases as determined by the DRS (Figure 1, Section 9).
Fluctuating intensity of symptoms (waxing and waning during a 24-hour period; Item #10, rated on three levels, with possible scores being 0, 2, and 4) was found in half the individuals, whereas the remainder showed a lower prevalence of stable symptoms, mostly present during the daytime, rather than symptoms worsening at night (Figure 1, Section 10).
When comparing the frequency of the distribution and severity of different symptoms, between delirium and non-delirium patients, it is clear from Figure 1 that the most significant differences were found in the areas of cognitive status, psychomotor behavior, sleep–wake cycle, and specificity of the physical disorder. The figure also shows that the different distribution for the items "Hallucinations" and "Lability of Mood" did not reach statistical significance at the level of p<0.01.
Memorial Delirium Assessment Scales (MDAS) Data
Item #1 showed a mild reduction in level of consciousness in about 40% of the delirium patients. Four of ten patients are, in fact, unaware of some elements in the environment or not spontaneously interacting appropriately with the interviewer; however, they become fully aware and appropriately interactive when stimulated (Figure 2, Section 1).
Almost all of the affected patients showed disorientation in space, time, or person. Half of these showed a severe level of impairment, 25% showed moderate, and 25% showed mild disturbances (Figure 2, Section 2).
Ninety percent of the patients with delirium presented with moderate-to-severe short-term memory impairment (Figure 2, Section 3), and all had an impairment in ability to perform the digit-span task (Figure 2, Section 4).
All patients with delirium had abnormal attention capacity, with at least mild (25%) or moderate-to-severe impairment (75%; Figure 2, Section 5).
Two-thirds of delirium subjects showed mild or moderate disorganization of thinking (Figure 2, Section 6).
The most frequent perceptual disturbances (30%) were of mild severity: misperceptions or illusions related to sleep or fleeting hallucinations on few occasions and without inappropriate behavior (Figure 2, Section 7). The majority (54.5%) of the delirium patients did not present misinterpretations or delusions, whereas 21% presented delusions admitted by the patient or evidenced by relatively mild behavioral abnormalities (Figure 2, Section 8).
Psychomotor activity mildly or moderately deviant from the norm (increased or decreased) was verifiable in three-quarters of the subjects (Figure 2, Section 9); 70% of these patients showed hypoactive behavior.
Three individuals out of four showed mild or moderate sleep–wake cycle disturbance (Figure 2, Section 10). In the first case, there was difficulty in falling asleep, transient night awakenings, and periods of drowsiness during the day or during the interview. In the second case, night awakenings were repeated and prolonged, and there was frequent and prolonged napping during the day or during the interview.
The comparison of the differences between delirium and non-delirium patients in Figure 2 shows that all items reached statistical significance. The difference is particularly clear-cut for changes in consciousness and attention (Items #1 and #5), cognition (Items #2 and #4), thought process (Item #6), psychomotor activity (Item #9), and sleep (Item #10).
Relationship Between Objective Evaluation (the CAM) and Subjective Perception of Delirium
Figure 3 illustrates the percentage of patients (y axis) presenting different levels of subjective perception of confusion (x axis). The answers to the questions about the subjective perception of the level of mental confusion showed a rather uniform distribution of the number of subjects only for the delirium patients. Thirteen with acute confusion (21%) answered that they did not feel confused at all; 17 (27.4%) that confusion was light; 18 (29%), that it was moderate; and 14 (22.6%) considered themselves extremely confused. Information was missing for four subjects, excluded from the statistical analyses, who objectively showed severe delirium. Among the 40 non-delirium subjects, 22 (55%) did not report any subjective confusion; however, 9 (22.5%) described themselves as slightly confused; 7 (17.5%), moderately; and only 2 (5%) felt themselves to be extremely confused. The distribution of the answers between the two groups showed a statistically significant difference (Kruskal-Wallis p=0.0004).
View larger version (30K):
[in this window]
[in a new window]
|
FIGURE 3. Relationship Between the Level of Subjective Perception of Confusion Reported by Patients (x axis) and Delirium Diagnosis According to CAM Criteria (y axis)
CAM: Confusion Assessment Method. The level of probability that the subjective perception differentiated between non-delirium (CAM: 0) and delirium (CAM: 1) patients was significant (p<0.001).
|
|
DISCUSSION
|
In this study, we examined the distribution of the single items of the Delirium Rating Scale (DRS)7 and the Memorial Delirium Assessment Scale (MDAS) among cancer patients with and without delirium. The criteria for delirium diagnosis have been reduced from five, in the DSM III-R, to four, in the more recent DSM versions (IV and IV-TR; Table 1). The impairment of the level of consciousness (awareness) and attention are mostly emphasized in DSM-IV Criterion A. In the previous (DSM-III-R) version, this symptom is classified as a sub-criterion, together with five other symptoms (i.e., disturbances of perception, sleep–wakefulness cycle, memory, psychomotor behavior, and orientation, of which at least two must be present). The criteria on cognitive impairment, temporal onset, and variability of symptoms substantially overlap.
View this table:
[in this window]
[in a new window]
|
TABLE 1. Differences in the Diagnostic Criteria for the Diagnosis of Delirium Among the DSM-III-R, DSM-IV, and DSM-IV–TR Classifications
|
The etiologic criteria differ, since the DSM-IV imposes various etiological classifications (i.e., delirium due to a general-medical condition; substance-intoxication delirium; substance-withdrawal delirium; delirium due to multiple etiologies; and delirium, not otherwise specified), whereas DSM-III-R required identification or inference of an etiology in cases where there is not clear-cut evidence for one. The Delirium Rating Scale (DRS) was developed according to the DSM-III criteria, whereas the MDAS was developed to include the DSM-IV criteria (Table 2).
A first distinction between the two instruments is that the assessment of altered attention and consciousness is not considered in the DRS, whereas the MDAS has two items specifically reporting this criterion: "reduced level of consciousness" and "reduced ability to maintain and shift attention."
Cognitive impairment is represented in the DRS by "Perceptual Disturbance" and by "Cognitive Status" items. The DRS suggests formally testing cognition with any tools the clinician or the investigator would choose. In the MDAS, there are several items that can be considered sensitive to cognitive impairment, and to attention failure, in particular: "Disorientation," "Short-Term Memory Impairment," and "Impaired Digit Span." The sensitivity of the areas tested by the DRS to a change in the level of consciousness and attention is probably high, but the MDAS chooses specific items from already-validated testing systems, such as the digit and verbal spans that have been proven to be sensitive to changes in overall attentional abilities.14
Both the DRS and the MDAS, however, retain, as individual items, some of the DSM-III-R criteria15 that are partially de-emphasized by DSM-IV,16 such as delusions, hallucinations, sleep–wakefulness cycle alterations, and altered psychomotor behavior.
The criteria for temporal onset and variability of symptoms and etiology are exclusively considered by the DRS. The MDAS does not consider variability of symptoms or acuteness of onset because it is designed to assess mental condition at the moment of the patient’s evaluation and is to be used in repeated administrations over a short time. The DRS, instead, investigates biographical and historical information and collects information relevant to the last 24 hours. This element has been partially modified in the most recent version (DRS-R98),17 which refers to a period to be defined by the clinician.
The analysis of the distribution of the single items may suggest that that the MDAS structure is more closely related to the DSM-IV criteria than is that of the DRS. Single, well-characterized symptoms, such as hallucinations, predict less well probably because of the possibility of partial syndromes (organic hallucinations, according to in the old taxonomy). Mood lability, as assessed by the DRS, is probably not very specific to this clinical condition.
An interesting finding pertains to the distribution of subjective perception of delirium in the two clinical groups, which can be of value in considering the use of subjective and quality-of-life scales in this population of patients. With regard to these issues, the patient’s subjective evaluation may provide clinicians with complementary information. In fact, most delirium patients (79%) reported perception of being confused.18 On the other hand, there were some delirium patients (21%) who did not report any subjective confusion, and some patients without clinical delirium who, instead, did report feeling confused (45%). All the four patients who were not able to answer the self-assessment questionnaire had severe delirium.
We should recognize that variability about the meaning of the term "confusion" may account for some of the variability in answers; however, at least partial self-awareness of change in mental state seems to emerge, along with the most impaired patients’ inability to participate in self-assessment, as important indicators. This observation is also supported by the recent finding that patients recovering from acute confusional states can recall their experience.18
Also, the nine patients who were normal at formal examination but reported feeling significantly confused (moderate or extreme level of confusion) yield significant information. The items of the two assessment tools that were most closely associated to the diagnosis of delirium explored the clinical dimensions of altered level of consciousness and attentional capacity (MDAS), together with abnormal cognitive processing (DRS and MDAS), psychomotor, sleep (MDAS and DRS), and thought-process disturbances (MDAS), whereas the specificity of hallucinations, delusions, and mood lability as assessed by these instruments is less evident.
From the results reported here, we can conclude that the DRS and the MDAS are structurally different objective assessment tools for measuring delirium when compared with clinically-based diagnostic criteria, with the MDAS more closely reflecting these criteria. This finding can be important in clinical settings when using a scale measuring the change in symptoms over time. However, there is a need to examine this issue more closely in the new version of the DRS (DRS-R–98)17 in order to provide clinicians with a set of instruments helping them to monitor the several dimensions of delirium in a single patient. With respect to this, even with some limitations, subjective perception could also be considered18 as a potential source of information in the overall assessment of mental status in medically ill patients.
|
ACKNOWLEDGMENTS
|
The following clinicians also participated in the IDSG: E. Beltrami, C. Borreani, F. De Conno, M. Luzzani, M. Maltoni, S. Mercadante, M. Monti, and M. Zamorani.
|
REFERENCES
|
- Smith MJ, Breitbart WS, Platt MM: A critique of instruments and methods to detect, diagnose, and rate delirium. J Pain Symptom Manage 1994; 10:35–77[CrossRef]
- Trzepacz PT: A review of delirium assessment instruments. Gen Hosp Psychiatry 1994; 16:397–495[CrossRef][Medline]
- Hjermstad M, Loge J, Kaasa S: Methods for assessment of cognitive failure and delirium in palliative-care patients: implications for practice and research. Palliat Med 2004; 18:494–506[Abstract/Free Full Text]
- Caraceni A, Grassi L: Delirium: Acute Confusional States in Palliative Medicine. Oxford, UK, Oxford University Press, 2003
- Inouye SK, van Dyck CH, Alessi CA, et al.: Clarifying confusion: The Confusion Assessment Method: a new method for detection of delirium. Ann Intern Med 1990; 113:941–948[Medline]
- Albert MS, Levkoff SE, Reilly CR et al: The Delirium Symptom Interview: an interview for the detection of delirium symptoms in hospitalized patients. J Geriatr Psychiatry Neurol 1992; 5:14–21[Medline]
- Trzepacz PT, Brenner RP, Coffman GC, et al: Delirium in liver-transplantation candidates: discriminant analysis of multiple test variables. Biol Psychiatry 1988; 24:3–15[CrossRef][Medline]
- Breitbart W, Rosenfeld B, Roth A, et al: The Memorial Delirium Assessment Scale. J Pain Symptom Manage 1997; 13:128–137[CrossRef][Medline]
- Rosen J, Sweet R, Mulsant B, et al: The Delirium Rating Scale in the psychogeriatric inpatient setting. J Neuropsychiatry Clin Neurosci 1994; 6:30–35[Abstract/Free Full Text]
- Rockwood K, Cosway S, Stolee P, et al: Increasing recognition of delirium in elderly patients. J Am Geriat Soc 1994; 42:252–256[Medline]
- Grassi L, Caraceni A, Beltrami E, et al: Assessing delirium in cancer patients: the Italian versions of the Delirium Rating Scale and the Memorial Delirium Assessment Scale. J Pain Symptom Manage 2001; 21:59–68[CrossRef][Medline]
- Inouye SK, Foreman MD, Mion LC, et al: Nurses’ recognition of delirium and its symptoms: comparison of nurse and researcher ratings. Arch Intern Med 2001; 161:2467–2473[Abstract/Free Full Text]
- Trzepacz PT: The Delirium Rating Scale: its use in consultation-liaison research. Psychosomatics 1999; 40:193–204[Abstract/Free Full Text]
- Lezak M, Lesa M, Howieson D, et al: Neuropsychological Assessment. Oxford, UK, Oxford University Press, 2004
- American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders, 3rd Edition, Revised. Washington, DC, American Psychiatric Press, Inc., 1987
- American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders, 4th Edition, Text Revision (DSM-IV-TR). Washington, DC, American Psychiatric Press, 2000
- Trzepacz PT, Mittal D, Torres R, et al: Validation of The Delirium Rating Scale, Revised-98: comparison with The Delirium Rating Scale and The Cognitive Test for Delirium. J Neuropsychiatry Clin Neurosci 2001; 13:229–242[Abstract/Free Full Text]
- Breitbart W, Gibson C, Tremblay A: The delirium experience: delirium recall and delirium-related distress in hospitalized patients with cancer, their spouse/caregivers, and nurses. Psychosomatics 2002; 43:183–194[Abstract/Free Full Text]
Get information about faster international access.
Privacy Policy
Copyright © 2006
Academy of Psychosomatic Medicine.
All rights reserved.
Home
| Search
| Current Issue
| Past Issues
| Subscribe
| All APPI Journals
| Help
| Contact Us
|
TOP
ABSTRACT
INTRODUCTION
