HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in ISI Web of Science Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Request Reprints Citing Articles Citing Articles via ISI Web of Science (8) Citing Articles via Google Scholar Google Scholar Articles by Hauben, M. Articles by Reich, L. Search for Related Content PubMed PubMed Citation Articles by Hauben, M. Articles by Reich, L. Social Bookmarking                   What's this?

Potential Utility of Data-Mining Algorithms for Early Detection of Potentially Fatal/Disabling Adverse Drug Reactions: A Retrospective Evaluation

From Pfizer Inc, New York (Dr Hauben, Dr Reich) and the Department of Medicine, New York University School of Medicine, New York (Dr Hauben); and Departments of Pharmacology and Community and Preventive Medicine, New York Medical College, Valhalla, New York (Dr Hauben).

Address for reprints: Lester Reich, MD, Pfizer Inc, 150 E 42nd Street (150-3-78), New York, NY 10017.

	ABSTRACT

TOP ABSTRACT METHODS RESULTS DISCUSSION REFERENCES

 
The objective of this study was to apply 2 data-mining algorithms to a drug safety database to determine if these methods would have flagged potentially fatal/disabling adverse drug reactions that triggered black box warnings/drug withdrawals in advance of initial identification via "traditional" methods. Relevant drug-event combinations were identified from a journal publication. Data-mining algorithms using commonly cited disproportionality thresholds were then applied to the US Food and Drug Administration database. Seventy drug-event combinations were considered sufficiently specific for retrospective data mining. In a minority of instances, potential signals of disproportionate reporting were provided clearly in advance of initial identification via traditional pharmacovigilance methods. Data-mining algorithms have the potential to improve pharmacovigilance screening; however, for the majority of drug-event combinations, there was no substantial benefit of either over traditional methods. They should be considered as potential supplements to, and not substitutes for, traditional pharmacovigilance strategies. More research and experience will be needed to optimize deployment of data-mining algorithms in pharmacovigilance.

Key Words: Data mining • adverse drug reactions (ADEs)

There are 2 basic types of DMAs: "simple" disproportionality analysis such as proportional reporting ratios (PRRs)¹ and reporting odds ratios (RORs)² and methods that use additional statistical adjustments and Bayesian modeling such as the multi-item gamma Poisson shrinker (MGPS)³ and the Bayesian confidence propagation neural network (BCPNN).⁴ Both approaches provide metrics related to the background probability of drug (across all events) and event (across all/most drugs) to derive the aforementioned internal control or model of expected reporting frequency in the absence of external data on the level of drug exposure. The Bayesian methods seek to improve the "SDR/noise" ratio by down-weighting (shrinking) scores based on small numbers of reports that are associated with higher statistical variability, but they may be associated with some decreased capacity for early detection of SDRs compared to simple disproportionality analysis (ie, PRRs or RORs) when commonly cited thresholds of disproportionality are used.^5-8

Until 2004, published research in this area had been conducted mostly by local or worldwide governmental health agencies, with these researchers providing findings from the DMA currently under investigation or in use by their respective institution (ie, MGPS [US Food and Drug Administration],³ PRRs [UK Medicines Control Agency],¹ ROR [Netherlands Pharmacovigilance Foundation Lareb],^2,9 BCPNN [World Health Organization Uppsala Monitoring Centre]^4,10). Comparison of the respective DMA to traditional methods of pharmacovigilance, an objective of our research by comparing the relative timing of the SDRs, was not emphasized in these research papers. Except for one instance,⁹ comparative performance between DMAs was not assessed by these investigators. With one exception,¹⁰ the DMA studied was also not assessed against standard literature sources. Except for a few specific examples in some of the papers,^3,10,11 the data have been anonymized with respect to drug and event, making it difficult, if not impossible, to characterize the nature of the events from the perspective of their medical importance and level of evidence. For the institution (eg, pharmaceutical manufacturer) assessing the potential utility of DMAs for use in its pharmacovigilance program, the absolute or relative absence of substantive information on the nature of adverse events/medical importance in these publications, the lack of assessment against standard literature sources, and the lack of emphasis on comparative assessments between DMAs and "traditional methods" of pharmacovigilance represent gaps in the literature. In an attempt to add to the collective knowledge and partially fill in these gaps and to help us better understand the potential of DMA(s) to usefully supplement our pharmacovigilance program, we undertook this research exercise using a literature source discussing adverse drug events that were all medically important with relatively strong evidentiary support.¹²

Of particular concern to public safety are a subset of adverse events that have been described as "designated medical events" (eg, agranulocytosis and toxic epidermal necrolysis). These often raise concerns with as few as 1 to 3 reports because of their rarity, medical importance, and high drug attributable risk.¹³ At this time, it is inadvisable to rely on existing DMAs to detect signals of these events. Black box warnings deal with special safety issues (eg, those leading to death or serious injury, including in many instances these designated medical events), and these warnings may be required by the US Food and Drug Administration (FDA) to be placed in the product labeling. Because ADEs triggering black box warnings/drug withdrawals in the United States have important implications for public safety, it is of interest whether DMAs may provide SDRs of such events.

For this analysis, we compared a Bayesian algorithm (MGPS) to a form of simple disproportionality analysis (PRRs) when applied to the US FDA Adverse Event Reporting System (AERS) database to determine if these methods would have provided SDRs from a sample of ADEs obtained from a published compilation of black box warnings/drug withdrawals¹ in advance of their initial identification via traditional methods that were in operation at the time. Traditional methods used by industry or health authorities can include analysis of clinical trial and epidemiological data as well as the previously noted spontaneously reported and published postmarketing reports on an ongoing basis or at predefined time intervals. As described above, the latter methods are dependent on skillful observations by clinicians who are attuned to the possibility of ADEs and the need to report them.¹⁴

	METHODS

TOP ABSTRACT METHODS RESULTS DISCUSSION REFERENCES

 
A recent peer-reviewed publication summarizing ADEs that were the subject of black box warnings/drug withdrawals between 1975 and 2000 in the United States provided the sample for this analysis.¹² Because an adverse event can be listed some time in the product label prior to elevation to a black box warning/drug withdrawal, a manual review of the annual Physicians' Desk Reference (PDR) was performed to determine the year in which the given adverse events first appeared.

The PDR is published in cooperation with participating manufacturers. Each entry provides an exact copy of the product's FDA-approved labeling. The Code of Federal Regulations, Title 21, Section 201.100(d)(1), requires that all wording in the PDR that pertains to, among other things, warnings, contraindications, side effects, and hazards be "same in language and emphasis" as the approved labeling, which means verbatim use of the language in the approved product label.¹⁵ Deadlines for submission to the annual PDR of product labeling or changes/modifications to product labeling, such as the addition of an adverse event, occur in the year prior to the edition year in which they will appear. Thus, changes submitted by the appropriate deadline in 2003 would appear in the 2004 PDR, published in November 2003. Labeling-submission deadlines for manufacturers start from June through August and depend on a manufacturer's place in the alphabetical listings of the publication.¹⁶ For example, manufacturers whose names begin with A or B would have been required to submit their information in June 2003 for publication in the main 2004 PDR. If a manufacturer misses the submission deadline, it has the option of including the information in 1 of 2 PDR supplements that are published the following July and September.¹⁶ Alternately, the manufacturer can distribute updated labeling via the PDR Addendum program, which allows manufacturers to disseminate product labeling to the practitioners throughout the year. It is important to note that the latest changes to product labeling, even if made prior to submission deadlines, are not always submitted to the PDR at the time of change and are often submitted for publication in subsequent supplements or later editions of the PDR.¹⁶ We used the aforementioned submission guidelines as a guide to estimate the latest year by which the respective drug-event combination (DEC) must have been initially recognized and evaluated by the traditional methods operant at the time and submitted for publication. For example, if an ADR appeared in the 1998 annual PDR,it could have been submitted by the manufacturer in the second half of 1996 or the first half of 1997. This metric is imperfect because of the 2-year window and because it favors the DMAs over traditional methods because submission of an ADR for publication in the PDR is the end result of a process of initial signal identification and evaluation. The time delay associated with this process is unknown and probably highly variable. For those DECs that were not listed in the annual PDR because of an early drug withdrawal, the year of withdrawal was the year that was used in the analysis.

The FDA AERS database is a computerized information database for postapproval safety surveillance. It functions as an early warning system for ADEs not detected during preapproval testing. It contains ADE reports with approved drugs and therapeutic biological products submitted in accordance with mandatory reporting obligations by pharmaceutical companies and voluntarily by health care professionals and consumers. Adverse events are submitted on MedWatch forms. Adverse event reports are reviewed and coded for data entry in accordance with the standardized terminology of the Medical Dictionary for Medical Regulatory Activities (MedDRA). Quarterly extracts are available through the National Technical Information Service (NTIS). These quarterly updates are subjected to extensive cleaning (ie, removal of redundant drug nomenclature and duplicate reports) prior to data mining. The data extract used for the current analysis included data in the AERS from 1968 through the first quarter of 2003.¹⁷

Only those black box warnings/drug withdrawals describing specific ADE(s) were considered for this analysis. For each specific ADE or group of ADEs, the verbatim term(s) from the paper or the PDR were used for data mining in addition to MedDRA Preferred Terms that were considered clinically equivalent or closely related to the verbatim term (eg, hepatic encephalopathy and liver transplant with hepatic failure), as determined by the first author. The second author reviewed and expanded in some cases the initial findings following "open-ended" (ie, all drug-event combinations [DECs] generating an SDR were reviewed for relevance) data mining. Any discrepancies in results were identified and adjudicated between the authors.

The 2 DMAs chosen for this analysis were PRRs¹ and MGPS (Lincoln Technologies, Wellesley Hills, Mass).³

The PRR is a simple metric relating the proportional representation of an event of interest with a drug of interest compared to the proportional representation of that event among all other drugs in the database (Table I). For this analysis, a PRR>2 with an associated ² > 4 (with Yates correction) was considered an SDR, as this degree of disproportionality has been frequently used in published studies of data mining.¹

The theoretical basis of MGPS has been described in detail elsewhere^3,18 but briefly is as follows. Expected counts for item sets (DECs) are based on the product of the marginal probabilities of each item (drug and event) in the database. The observed-to-expected (O/E) ratio is initially calculated as a crude disproportionality metric. Because the same ratio could be obtained from cell counts (frequencies) of markedly different sizes (O/E ratios based on smaller cell counts being considered more variable or imprecise), further modeling of the O/E ratios using maximum likelihood estimation and Bayesian inference is used to adjust the crude O/E ratios based on the respective cell counts. Each cell is considered to represent a Poisson process in which the Poisson parameter distribution is related to a mixture of 2 gamma distributions. The prior probability distribution of the gamma parameters is obtained by applying an interactive maximum likelihood algorithm to a negative binomial mixture likelihood. Posterior estimates of the gamma parameters are obtained by updating the prior with the individual cell counts via the Bayes theorem providing the posterior distribution of O/E ratios.

By using logarithmic transformations or taking the lower 5% cutoff of the posterior distribution (EB₀₅), an expectation value that adjusts for the variability by down-weighting or "shrinking" the parameters associated with low cell counts is obtained. These metrics are known as the empirical Bayes geometric mean (EBGM) and the EB₀₅.AnEB₀₅ of 8 may therefore be interpreted to mean that reports of the particular DEC occur in the database 8 times more frequently than would be expected if drug and event were independently distributed in the database. The SDR metric used for a threshold in the current analysis was the lower 5% cutoff of the empirical Bayes geometric mean > 2 (EB₀₅ >2).¹⁹ It has been stated that for EB₀₅ 2, "experience indicates that the signals using this cutoff have high enough specificity to deserve further investigation."³

A variety of data-mining options and parameters exist, including basic covariate adjustment (stratification by age, gender, and year of report) and cumulative subsetting. Stratification tends to reduce spurious associations due to confounding and markedly decreases the volume of disproportionalities.^18,20

For the present analysis, the data mining was performed on suspect drug-ADR pairs using stratification by age, gender, and FDA year of report, with cumulative subsetting by year (for EB₀₅ calculations).

	RESULTS

TOP ABSTRACT METHODS RESULTS DISCUSSION REFERENCES

 
The peer-reviewed published analysis listed 65 new black box warnings involving 45 drugs plus 11 additional drugs for which marketing authorization was withdrawn. A total of 354 DECs involving these 56 drugs were preselected and investigated by 1 author. The other author identified 19 additional events (12 drugs) following "open-ended" (ie, all DECs generating an SDR were reviewed for relevance) data mining.

Ten of these drugs were approved by the US FDA in the 1970s, 25 were approved in the 1980s, and 21 were approved in the 1990s. Of these 56 drugs, 1 was not in the AERS database. For the remaining 55 drugs, 70 DECs were considered sufficiently specific for data mining (an example of a DEC that was excluded was "oral form not as effective as intravenous" for an antiviral agent). Fifty-nine DECs were associated with an SDR with MGPS (11 were not) and 65 DECs with an SDR with PRRs (5 were not). For those DECs associated with SDRs, performance similarities and differences were observed in the time to appearance of the SDR between the 2 methods and their relationship to estimated time of submission to the PDR or year of drug withdrawal. A summary of these findings is provided in Table II.

There were 16 DECs with PRRs and 13 DECs with MGPS that exceeded thresholds but could not otherwise be categorized because there were no reports in AERS at the time of estimated submission to the PDR or drug withdrawal, or it could not be determined if there were reports in AERS at the time of estimated submission because of our 2-year window for estimated submission time.

Of the 65 SDRs with PRRs, 46 (71%) occurred in the first year the DEC was identified in AERS. Of the 59 SDRs with MGPS, 21 (32%) occurred in the first year. A summary of the relationship of the SDR to the number of years of reporting experience in AERS is provided in Table III.

For the 30 DECs in which an SDR was generated by PRRs in advance of MGPS, 13 generated an SDR with 1 or 2 reports, with 5 before the estimated time of submission to the PDR or drug withdrawal. If a commonly used case count (n) threshold of n > 2 for these latter 5 DECs had been used,¹ 4 of 5 still would have generated an SDR with PRRs before the estimated time of submission to the PDR or drug withdrawal but concurrently with MGPS. The mean delay in the generation of an SDR with PRRs for the aforementioned 13 DECS using n > 2 was 2.9 years. MGPS never generated an SDR with n < 3. For the 17 of the aforementioned 30 DECs initially generating an SDR with an n > 2 with PRRs, up to 25 reports were needed to generate a SDR with PRRs (mean = 7.1 reports).

	DISCUSSION

TOP ABSTRACT METHODS RESULTS DISCUSSION REFERENCES

 
Although DMAs such as MGPS and PRRs have the potential to improve pharmacovigilance screening processes, their utility may be highly situation dependent. For the majority of DECs from a published compilation of black box warnings/drug withdrawals, there was no obvious benefit of either over traditional methods in terms of initial "signal" detection. In a minority of instances (MGPS,⁹ PRRs¹⁵), there was a benefit in that a potential SDR was provided clearly in advance of the estimated PDR submission. However, when an SDR did occur, irrespective of its relationship to the PDR submission date, PRRs trended toward signaling earlier than MGPS (eg, 71% of initial signals with PRRs occurred in the first year of reporting vs 36% with MGPS). This enhanced "sensitivity" with PRRs may be associated with an overabundance of "signals," including "false-positive signals" of no importance to public safety that may require additional triage criteria for practical implementation. This may be particularly problematic when considering disproportionalities associated with case counts of 1 or 2. In practice, this is often mitigated by applying case count thresholds (eg, n > 2) or by choosing higher PRR thresholds. For the 30 DECs in which an SDR was generated by PRRs in advance of MGPS, 13 generated an SDR with 1 or 2 reports, with 5 before the estimated time of submission to the PDR or drug withdrawal. When a threshold of the number of reports (n > 2) for these latter 5 DECs had been used, 4 of 5 still would have generated an SDR in advance of the estimated time of submission to the PDR or drug withdrawal, but they no longer would have generated an SDR in advance of MGPS. Thus, it appears that one aspect of the performance gradient between PRRs and MGPS disappeared in this subsetting, if the case count threshold of n > 2 was used. With PRRs, a case count threshold of n > 2 would still be expected to generate more SDRs relative to MGPS, and thus for this exercise, MGPS may be considered under these circumstances more efficient if case count thresholds are used. However, the overabundance of signals associated with simple forms of disproportionality analysis, in the absence of case count thresholds (eg, n > 2), may not be prohibitive from an operational point of view because expert safety reviewers may quickly filter out many "false-positive" associations by integrating prior knowledge of drugs, events, patient populations, and diseases in a process that may resemble an informal application of Bayesian hypothesis refinement. This remains uncertain given that the frequently used and reasonable case count thresholds results in a paucity of published data over the full range of reporting experience in naturalistic pharmacovigilance settings. Pharmacovigilance is not based on a firm theoretical foundation and is both art and science. Therefore, there are scenarios in which the observation of disproportionalities from individual cases might prompt a refined assessment and monitoring strategy (eg, selected, clinically similar events in separate reports). David Finney, who originally delineated numerical approaches to SRS data, said "the essence is to collect facts that individually tell little, but collectively form a clue to drug dangers."²¹

Because commonly cited thresholds are unvalidated, subjective, and adjustable, the clinical significance of these findings is currently unclear. In developing a pharmacovigilance strategy that incorporates DMAs, the trade-offs in "sensitivity" and "specificity" associated with the choice of algorithms and thresholds (disproportionality and case counts) should be carefully considered, including the statistical instability associated with disproportionalities based on 1 or 2 reports. However, it should be noted that for "designated medical events," coincidental associations are so unlikely and the consequences of delayed recognition so high that even 1 report could constitute a strong warning, prompting further investigation. It is hoped that published findings related to disproportional reporting over the full range of reporting experience will help to optimize threshold selection.

There are several significant limitations to this analysis. The sample of DECs examined from the US FDA database represents a tiny fraction of reported DECs to the FDA, and this study did not address the significance of SDRs generated for events not included in black box warnings. This nonsystematic analysis cannot be used to draw inferences about the overall performance characteristics of these techniques. It is likely that the incremental utility of DMAs is highly situation dependent. Although not all of the ADEs examined were designated medical events, they were clustered at the high end of the spectrum of medical seriousness and would be expected to be subject to close scrutiny by traditional methods. The incremental utility of DMAs could therefore be higher for nonserious or less serious events not subject to the same level of scrutiny. Another intriguing possibility is the use of DMAs to screen for higher order associations such as complex drug-drug interactions or syndromes that might be less amenable to detection by clinical cognition. In addition, there are numerous nuances and pitfalls in the systematic validations and performance characterization of automated signal detection methods, including the lack of standardized data-mining procedures (eg, selection and combination of adverse event terms, numerical thresholds, and dictionary hierarchy thresholds), variations in database and dictionary architectures, and the multiple biases, confounding factors, and data quality limitations inherent to voluntary reporting systems. For example, as noted in the Results section, there was disagreement between the 2 authors in terms of event selection. The fact that 1 author identified 19 additional events following "open-ended" data mining points out the inherent subjective nature of event selection for retrospective analyses. Although these 19 events were a small proportion (5.4%) of the total of 354 DECs initially investigated, it demonstrates the importance of an "open-ended" event analysis to improve/confirm the reliability of the data upon which any conclusions are based, and it is a practice that we would recommend. However, caution and sensible medical judgment are warranted due to the susceptibility to various interpretive biases. For example, in a process akin to multiple comparisons and orientation bias, a data miner with a strong incentive to believe in a particular outcome may use nonspecific case definitions (ie, adverse event terms) of dubious clinical relevance in hopes of avoiding results that contradict preexisting expectations. Data mining is a welcome addition to the pharmacovigilance tool kit that has the potential to improve our ability to monitor the safety of medicines, but data dredging or torturing should be avoided.

In addition, we subsetted our data-mining analysis by year, and it is possible that an analysis employing increased temporal resolution (eg, quarterly instead of yearly) might have detected differences in those DECs that appeared to have been identified concurrently by traditional methods and the DMA.

Reports in the AERS database are not the only source of labeling information. Labeling updates can occur as a result of new data from several sources, including clinical trials, the evolving safety profile of other drugs in the same class, and epidemiological information that would not be captured in AERS. It should also be noted that our retrospective analysis may not reflect the full range of prospective data-mining practices in "real-life" data mining because they are exploratory in nature and would incorporate various subjective processes and judgments related to human cognition that defy explicit characterization. It is important to appreciate that retrospective validation exercises such as ours (eg, the data miner knows the ADE "to be signaled") are uniquely susceptible to various interpretive biases due to the use of multiple unvalidated thresholds metrics, algorithms, and post hoc definitions.⁸ Performance with a prospective analysis could be better or worse. Any performance differentials between PRRs and MGPS similar to what we observed are likely to be significantly mitigated when these methods are used as one element of a comprehensive pharmacovigilance program that uses multiple approaches to signal detection.

In summary, we found that when the defined minimum time delay between recognition of a potential ADE and amendment of the corresponding product label in the PDR was considered, the use of 2 DMAs with preselected thresholds failed to outperform traditional pharmacovigilance practices for the majority of DECs that triggered a black box warning or drug withdrawal. In a minority of instances, the DMAs highlighted an association in advance of traditional methods, and this suggests the potential of DMAs to be useful supplements to, but not substitutes for, traditional pharmacovigilance techniques. However, the incremental utility of DMAs with be highly contingent on the opportunity costs associated with "false alarms." As expected, trade-offs in "sensitivity" and "specificity" were observed that are highly dependent on the thresholds chosen. Data mining in pharmacovigilance is a dynamic field, and additional study and experience should be strongly encouraged. However, we advise against overattention to the development of DMAs at the expense of research and discussion on improving the quality of the data and the methods by which clinical judgment and medical knowledge are applied to the process of "signal" detection. Additional systematic data and prospective experience are needed to more fully understand the performance characteristics of these methods.

	REFERENCES

TOP ABSTRACT METHODS RESULTS DISCUSSION REFERENCES

 

1. Evans SJW, Waller P, Davis S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf. 2001;10: 483-486.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]

2. Van Puijenbroek E, Diemont W, van Groothest K. Application of quantitative signal detection in the Dutch spontaneous reporting system for adverse drug reactions. Drug Saf. 2003;26: 293-301.[Medline] [Order article via Infotrieve]

3. Szarfman A, Machado SG, O'Neill RT. Use of screening algorithms and computer systems to efficiently signal higher-than expected combinations of drugs and events in the US FDA's spontaneous reports database. Drug Saf. 2002;25: 381-392.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]

4. Bate A, Lindquist M, Edwards IR, et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol. 1998;54: 315-321.[Medline] [Order article via Infotrieve]

5. Hauben M. Trimethoprim-induced hyperkalemia: lessons in data mining. Br J Clin Pharm. 2004;58: 338-339.[Medline] [Order article via Infotrieve]

6. Hauben M, Reich L. Safety related drug-labelling changes: findings from two data mining algorithms. Drug Saf. 2004;27: 735-744.[Medline] [Order article via Infotrieve]

7. Moseley JNS, Heeley S, Elkins-Daukes S, Evans S. Preliminary comparison of 2 signal detection methodologies in the UK regulatory authority spontaneous ADR database. Drug Saf. 2004;27: 950-951.

8. Hauben M. Application of an empiric Bayesian data mining algorithm to reports of pancreatitis associated with atypical antipsychotics. Pharmacotherapy. 2004;24: 1122-1129.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]

9. Van Puijenbroek EP, Bate A, Leufkens HGM, Lindquist M, Orre R, Egberts ACG. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf. 2002;11: 3-10.[Medline] [Order article via Infotrieve]

10. Lindquist M, Stahl M, Bate A, Edwards IR, Meyboom RHB. A retrospective evaluation of a data mining approach to aid finding new adverse drug reactions signals in the WHO international database. Drug Saf. 2000;23: 533-542.[Medline] [Order article via Infotrieve]

11. Almenoff JS, DuMouchel W, Kindman LA, Yang X, Fram D. Disproportionality analysis using empirical Bayes data mining: a tool for the evaluation of drug interactions in the post-marketing setting. Pharmacoepidemiol Drug Saf. 2003;12: 517-521.[Medline] [Order article via Infotrieve]

12. Lasser K, Allen P, Woolhandler S, Himmelstein D, Wolfe S, Bor D. Timing of black box warnings and withdrawals for prescription medications. JAMA. 2002;287: 2215-2220.[Abstract/Free Full Text]

13. Begaud B, Moride Y, Tubert-Bitter P. False positives in spontaneous reporting: should we worry about them. Br J Clin Pharmacol. 1994;38: 401-404.[Web of Science][Medline] [Order article via Infotrieve]

14. Trontell A. Expecting the unexpected: drug safety, pharmacovigilance and the prepared mind. N Engl J Med. 2004;351: 1385-1387.[Free Full Text]

15. Code of Federal Regulations, 21 CFR 201.100(d)(1) (2004).

16. Personal communication, Mukesh Mehta, Thomson Healthcare, Inc, 2004.

17. US Food and Drug Administration, Center for Drug Evaluation and Research. Adverse Events Reporting System (AERS). Availabile at: http://www.fda.gov/cder/aers/default.htm.

18. DuMouchel W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. Am Stat. 1999;53: 177-190.

19. Gould AL. Practical pharmacovigilance analysis strategies. Pharmacoepidemiol Drug Saf. 2003;12: 559-574.[Medline] [Order article via Infotrieve]

20. Hauben M. A brief primer on automated signal detection. Ann Pharmacother. 2003;37: 1117-1123.[Abstract/Free Full Text]

21. Finney DJ. The detection of adverse reactions to therapeutic drugs. Stat Med. 1982;1: 153-161.[Medline] [Order article via Infotrieve]
CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in ISI Web of Science Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Request Reprints Citing Articles Citing Articles via ISI Web of Science (8) Citing Articles via Google Scholar Google Scholar Articles by Hauben, M. Articles by Reich, L. Search for Related Content PubMed PubMed Citation Articles by Hauben, M. Articles by Reich, L. Social Bookmarking                   What's this?