The files in this directory comprise the code that was used
in our analysis of the 3 SELDI ovarian cancer data sets 
available from http://clinicalproteomics.steem.com.

There are 3 major subdirectories, and 3 main MATLAB scripts:

Data/
Figs/
bin/
our_processing.m
our_analysis.m
paper_figs.m

an additional matlab mat-file of the processed 
results of our_analysis.m is avaliable as MatFile.zip 
(about 65M).

The basic chronological structure is as follows.

-------------------------------------------------------------
-------------------------------------------------------------

First, we assemble the raw data. This is stored in 

Data/

We began by acquiring the 3 zip files of spectra 
from the web site (as of 10/11/03):

"Lancet Ovarian Data 2-16-02.zip"
"Ovarian Dataset 4-3-02.zip"
"Ovarian Dataset 8-7-02.zip"

The last of these has some supporting clinical information,
 
"Ovarian Sample Info 8-7-02.xls"

which we downloaded as well. We note in passing that 
the last data set was first made available in June of '02 
(when we initially acquired it), but shortly thereafter 
(in August of '02) this data set was replaced with the 
data set that is there now. We had some concerns over 
this, as we had already begun looking at the June data set, 
but it turns out that the spectra were numerically equivalent
and that only the file names had been changed in the interim. 
Thus, if your group has the other dataset, the same results
apply.

The zip files were then uncompressed in this directory for
further parsing. For example, unzipping the first
zip file, "Lancet Ovarian Data 2-16-02.zip",
creates the directory structure

Lancet Ovarian Data 2-14-02/
       Benign Disease/
	      s-[01-16] modified.csv (16 files)
       Ovarian Cancer/
	      a-[01-50] modified.csv (50 files)
	      b-[01-50] modified.csv (50 files)
       Normal Healthy/
	      c-[01-50] modified.csv (50 files)
	      d-[01-50] modified.csv (50 files)

unzipping the second zip file, "Ovarian Dataset 4-3-02.zip",
creates the directory structure

Ovarian Data WCX2 CSV/
       benign/
	      S[01-16].csv (16 files)
       cancer/
              Group A/
	              A[01-50].csv (50 files)
	      Group B/
	              B[01-50].csv (50 files)
       control/
              Group C/
	              C[01-50].csv (50 files)
	      Group D/
	              D[01-50].csv (50 files)

and unzipping the third zip file, 
"Ovarian Dataset 8-7-02.zip", creates

Ovarian Cancer/
       Ovarian Cancer daf-[0601-0780].csv (162 files; there are some gaps)
Control/
       Control daf-[0181-0281].csv (91 files; again there are some gaps).


Running wc (word count) under Linux showed that all of 
the .csv files in the above structures had the same
number of rows, 15155, so we checked to see if these could
be combined into a matrix in some nice way. Quick skimming
of the files showed that after a header row, each file 
contained two columns, one of m/z values and one of 
intensities. A quick sampling suggested that all of the 
m/z vectors were the same. The Perl script 

grab_intensities.pl 

was used to grab the intensity columns from all of the 
files within a subgroup and splice them together. 
When run, it produces:

S_02_16_02.txt (15154 by 16)
A_02_16_02.txt (15154 by 50)
B_02_16_02.txt (15154 by 50)
C_02_16_02.txt (15154 by 50)
D_02_16_02.txt (15154 by 50)

S_4_3_02.txt (15154 by 16)
A_4_3_02.txt (15154 by 50)
B_4_3_02.txt (15154 by 50)
C_4_3_02.txt (15154 by 50)
D_4_3_02.txt (15154 by 50)

A_8_7_02.txt (15154 by 162)
C_8_7_02.txt (15154 by 91)

running the related script

grab_mz_vec.pl

produces

moverz.txt (15154 by 1)

(These Perl scripts may need to be slightly modified to work
on your system, but this should be a matter of changing the 
Perl home directory and possibly the globs for Windows.)
A simple modification of this script (included as a 
commented out pair of lines) allowed us to check the 
M/Z entries of the separate files and confirm that they were
all the same. Actually, this is not precisely true. 
Using the grab_intensities Perl script, we established that the 
m/z vectors were exactly the same across all spectra in the August 
groups, but these m/z values are slightly different from those 
given in the February and April data sets (which are all the same).
The difference, however, is only that the number of significant
figures supplied is larger in the more recent spectra. Thus, we have

(Feb, Apr)	(Jun, Aug) 
-7.86E-05	-7.8602611e-005
2.18E-07	2.1773576e-007
9.60E-05	9.6021472e-005
0.000366014	0.00036601382
...		...
19992.874	19992.874
19995.513	19995.513

We decided that these extra few digits would not make a functional
difference in the analysis and elected to work just with the m/z
vector from one of the initial sets. As these m/z values are derived 
from a quadratic equation, the parameters of the quadratic must 
be the same in all cases. 

-------------------------------------------------------------
-------------------------------------------------------------

Second, having acquired the data and set it up in a somewhat
nicer block structure, we read it into MATLAB for our initial
analysis with the script

our_processing.m

This proceeds by 
1) loading the data into matlab
2) normalizing the spectra and flagging the identified key features
3) computing Euclidean distances using the peaks
4) computing two-sample t-statistics contrasting cancers with normals
   for all features
5) checking a few statistics not reported on in much detail in the paper, 
   such as rank sum tests
6) looking at minimum classification errors using a single peak
   (this uses the class1d script in bin/)
7) explicit testing for pairs of peaks achieving good separation,
   filtering on large t-values and using logistic regression.

Several of the tests in 5-7 are applied to Data Set 3 only. 

The end of the file contains a variety of scripts for generating
Figures displaying one aspect or another of the data.

-------------------------------------------------------------
-------------------------------------------------------------

Third, we performed our own analysis of Data Set 3, using
some approximate methods for baseline subtraction and peak 
identification before searching for peaks with good t-statistic 
values. These steps are described in 

our_analysis.m

and invoke several subroutines stored in bin/.

*****************
WARNING.
*****************

The results presented in an earlier version of our paper
(available as a tech report) were based on code we have
since found contained a bug. We have redone the analysis
using better (we think) code, but our qualitative conclusions
have not changed. The differences are described
in detail in the comments of our_analysis.m.

*****************
END WARNING.
*****************

Running the processing takes a while (it took a few hours
on my laptop) so this may be more useful in terms of suggesting
thoughts for algorithms than for anything else. 

The outline of our approach is as follows:

Because peak
finding and baseline correction are intertwined, we dealt with them in
an iterative fashion.  We first found all local maxima in each
spectrum, and recorded left and right ``downslope extents'' for each
local maximum. We only included local maxima that were reached by
climbing in both directions at least four steps larger than the
noise. For these purposes, ``noise'' was taken to equal one-half of
the median absolute value of the first differences of the 
entire spectrum.  The
``peaks'' were removed from the spectrum and replaced with straight
lines. We then fit a local minimum (using a window of 256 clock ticks)
and subtracted these values as an initial estimate of the baseline. We
then reran the peak finding and trimming steps, subtracted off the
baseline, and reintroduced the peaks.  After baseline correction, the
individual spectra were normalized to have the same total ion current
(summed intensities) in a region that excluded the matrix noise. We
normalized using intensities above M/Z 1000. (Equivalent results were
obtained normalizing to the region above M/Z 2000.) To
restrict our analysis to definite peaks, we computed a local
signal-to-noise ratio for each peak. For these purposes, the local
noise was taken to equal the median absolute deviation from the
median in a window of
256 clock ticks, and we retained peaks whose signal-to-noise ratio
exceeded three. This procedure identified between 224 and 301 peaks
per spectrum, with both a mean and a median of 266 peaks per spectrum.
We next aligned peaks across spectra, using the rule that peaks that
were separated by less than 0.05\% of the mass or by fewer than 3
clock ticks could not be distinguished. 


-------------------------------------------------------------
-------------------------------------------------------------

Fourth, we collect the scripts generating the figures
appearing in the paper in 

paper_figs.m

We have tried to make the code for each figure self-contained,
so there is redundancy in terms of loading data files and
whatnot.

-------------------------------------------------------------
-------------------------------------------------------------

That's most of the chronology. Various figures are collected
in Figs/, some of which were not in the paper. The file
CAPTIONS.txt in Figs/ gives a brief description of what 
we're looking at in each case. 

