As pointed out by a correspondent, the Duke web site supporting
their Nature Medicine paper on chemoresponse has disappeared; trying
to connect to http://data.cgt.duke.edu/NatureMedicine.php
yields a "403 Forbidden" response. The same correspondent pionted out
that you can still obntain old vesions at one of the following
URLs:
As promised in our post of 7 November 2007, we have now examined some of the new data available at the Duke web site for the Nature Medicine paper by Potti et al. We have prepared an overview of our new findings, which serves as an introduction to additional reports prepared in Sweave. Here is a brief summary.
The volume of Nature Medicine where our correspondence was published contains a response from Potti and Nevins. Not surprisingly, we disagree with several of their assertions. Here we provide a point-by-point rebuttal.
We appreciate the extra details that they have now made available. This description, however, was not part of the original article (or supplements) published by Nature Medicine. It is also not consistent with the statement in the published methods that indicated that they used GI50 values as the primary measure of sensitivity or resistance. As we have stated, the sensitive and resistant cell lines used in the original publication have GI50 values that overlap ( Supplementary Report 3), which still calls into question their classification as the most sensitive and resistant cohorts. Further, the taxtotere example that they provide in their new web page includes a footnote pointing out that they arbitrarily excluded the cell line COLO 205, which was by far the most sensitive to taxotere. This fact could not have been ascertained from the original paper.
This statment is misleading. As our supplementary reports show, we tried to reproduce all of the rest of the steps in their analysis using both the cell lines we thought were most sensitive and resistant as well as the cell lines they used in their reported analysis.
We simply note here that their response does not explicitly address point (4) of our published critique: 14 of the genes that were not produced by their own software for the taxotere signature were taken directly from the paper by Chang et al. that described the breast cancer test set used by Potti and Nevins.
We stand by our analysis: using their own software on the combined data consisting of a training set from the cell lines they say they used and a test set from the Chang breast cancer data, the resulting model does not predict outcome any better than chance. Our code is still available for inspection; they have only provided unsupported assertions to the contrary.
This sentence skirts two important issues. First, as we point out on page 15 of Supplementary Report 9, the other genes that mysteriously appeared on their list were the only genes that were mentioned by name in their publication as part of the biological justification for believing the signatures were meaningful. Second, as we discuss in our letter (and continue below), their model uses data from the test set as well as from the training set.
We, obviously, disagree, at a very fundamental level. Potti and Nevins implicitly acknowledge that the model you get when you combine training data and test data is different from the model you would get using only the training data. In particular, this approach means that you cannot fully specify the model before the test set is available. Further, it suggests that the model morphs into a different model for every new test set. These observations imply, to us, that the independence of the training and test sets is not maintained. Moreover, they imply that they do not actually have a single predictive model that can be generalized to new patient samples.
That figure shows two things. First, if you build the model (with their software) using only their training data, then you cannot make useful predictions on the test data. Second, when you combine the test data and the training data to build the model, then you can at least get predictions of two different classes in the test data. Note, however, that the report does not address whether those predictions are correct. We are preparing an additional report (which will be posted here soon) showing that the predictive accuracy is no better than chance. (Interested readers are encouraged to reserve judgement until that time, or to run our existing code and then test the accuracy of the predictions themselves, since otherwise we have simply made the same sort of unsupported assertions that we have complained about in our letter.)
We have our own opinions on the reproducibility and correctness of the results they cite in the publication by Hsu et al. in the Journal of Clinical Oncology, which we will be sure to discuss in an appropriate venue....
Our letter to Nature Medicine was published on 7 November 2007. The full reference (with the new title given to it by the editors is Coombes, KR, Wang J, Baggerly KA. Microarrays: retracing steps. Nature Medicine, 2007; 13(11):1276-1277.
After we communicated our original analysis to Drs. Potti and Nevins, they posted revised gene lists to the Nature Medicine web site in an attempt to correct the off-by-one indexing error. They apparently attempted to fix the problem manually, which introduced a number of different errors into the lists. Our review of the new gene lists is posted here: