Carlos-Francisco Méndez-Cruz / lcg-bioinfoI-bionlp

Go to a project
Toggle navigation Toggle navigation pinning
Carlos-Francisco Méndez-Cruz
4d7657b9 1 parent 8b9489f8

Training and testing binding thrombin dataset

Inline Side-by-side
Showing 8 changed files with 878 additions and 0 deletions
The test set consists of 634 data points, each of which represents
a molecule that is either active (A) or inactive (I). The test set
has the same format as the training set, with the exception that the
activity value (A or I) for each data point is missing, that is, has
been replaced by a question mark (?). Please submit one prediction,
A or I, for each data point. Your submission should be in the form
of a file that starts with your contact information, followed by a
line with 5 asterisks, followed immediately by your predictions, with
one line per data point. The predictions should be in the same order
as the test set data points. So your prediction for the first example
should appear on the first line after the asterisks, your prediction
for the second example should appear on the second line after the
asterisks, etc. Hence, after your contact information, the prediction
file will consist of 635 lines and have the form:
*****
I
I
A
I
A
I
etc.
You may submit your prediction by email to page@biostat.wisc.edu
or by anonymous ftp to ftp.biostat.wisc.edu, placing the file
into the directory dropboxes/page/. If using email, please use
the subject line "KDDcup <name> thrombin" where <name> is your
name. If using ftp, please name the file KDDcup.<name>.thrombin
where <name> is your name. For example, my submission would be
named KDDcup.DavidPage.thrombin
Only one submission per person per task is permitted. If you do not
receive email confirmation of your submission within 24 hours, please
email page@biostat.wisc.edu with subject "KDDcup no confirmation".
For group entries, the contact information should include the names
of everyone to be credited as a member of the group should your entry
achieve the highest score. But no person is to be listed on more than
one entry per task.
Prediction of Molecular Bioactivity for Drug Design -- Binding to Thrombin
--------------------------------------------------------------------------
Drugs are typically small organic molecules that achieve their desired
activity by binding to a target site on a receptor. The first step in
the discovery of a new drug is usually to identify and isolate the
receptor to which it should bind, followed by testing many small
molecules for their ability to bind to the target site. This leaves
researchers with the task of determining what separates the active
(binding) compounds from the inactive (non-binding) ones. Such a
determination can then be used in the design of new compounds that not
only bind, but also have all the other properties required for a drug
(solubility, oral absorption, lack of side effects, appropriate duration
of action, toxicity, etc.).
The present training data set consists of 1909 compounds tested for
their ability to bind to a target site on thrombin, a key receptor in
blood clotting. The chemical structures of these compounds are not
necessary for our analysis and are not included. Of these compounds, 42
are active (bind well) and the others are inactive. Each compound is
described by a single feature vector comprised of a class value (A for
active, I for inactive) and 139,351 binary features, which describe
three-dimensional properties of the molecule. The definitions of the
individual bits are not included - we don't know what each individual
bit means, only that they are generated in an internally consistent
manner for all 1909 compounds. Biological activity in general, and
receptor binding affinity in particular, correlate with various
structural and physical properties of small organic molecules. The task
is to determine which of these properties are critical in this case and
to learn to accurately predict the class value. To simulate the
real-world drug design environment, the test set contains 636 additional
compounds that were in fact generated based on the assay results
recorded for the training set. In evaluating the accuracy, a
differential cost model will be used, so that the sum of the costs of
the actives will be equal to the sum of the costs of the inactives.
We thank DuPont Pharmaceuticals for graciously providing this data set
for the KDD Cup 2001 competition. All publications referring to
analysis of this data set should acknowledge DuPont Pharmaceuticals
Research Laboratories and KDD Cup 2001.
This diff could not be displayed because it is too large.
I
A
I
I
I
A
I
I
I
A
I
I
I
A
I
A
I
I
I
I
I
I
I
I
I
I
I
A
I
A
I
I
I
I
I
A
I
I
I
A
I
I
I
I
I
I
I
I
A
A
I
I
I
I
I
A
I
A
A
I
I
I
A
I
I
I
I
A
A
I
A
I
I
A
A
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
A
I
I
A
I
A
I
I
I
A
A
I
I
I
I
I
I
A
I
I
I
I
A
A
I
I
I
I
I
I
I
I
A
A
I
A
A
I
I
I
I
I
I
I
I
I
I
A
I
I
I
I
I
I
I
I
I
A
I
I
I
I
A
I
I
I
I
I
I
I
A
I
I
A
I
A
I
I
A
I
A
I
A
I
A
I
I
I
I
I
A
I
I
A
I
I
A
I
I
I
A
I
A
I
I
A
I
I
I
I
A
I
A
I
I
I
I
I
I
I
I
I
I
I
A
I
A
I
I
I
I
I
I
A
I
I
A
A
A
I
I
A
A
I
I
I
I
A
I
I
I
I
A
I
A
I
I
I
I
I
I
I
A
A
I
I
I
I
I
I
I
I
A
A
I
I
I
I
I
I
A
A
I
I
I
I
I
I
A
I
A
I
I
I
I
I
I
I
I
I
A
I
I
A
I
I
I
I
I
I
A
A
I
I
I
I
I
A
I
I
I
I
I
A
A
A
I
A
I
I
I
I
A
A
I
A
A
I
I
I
I
I
I
I
I
I
I
I
I
A
I
I
I
I
A
A
I
I
A
I
I
I
I
I
A
A
I
A
I
I
I
I
I
I
A
A
I
I
A
I
I
I
I
I
I
I
I
I
I
I
I
A
I
I
A
I
I
A
I
I
I
I
A
A
I
A
A
I
I
A
I
I
I
I
A
I
I
I
I
I
I
I
I
I
I
A
I
I
A
A
I
I
I
A
I
I
I
I
A
I
A
I
I
I
I
I
A
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
A
A
A
A
I
I
I
A
A
I
I
I
I
I
A
I
A
I
I
I
I
I
A
I
I
I
I
A
A
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
A
I
A
I
I
A
I
I
I
I
A
I
I
A
A
I
I
I
A
I
A
I
I
I
I
I
I
I
A
A
I
I
I
A
I
I
I
A
I
I
I
I
I
I
A
I
I
I
I
I
A
I
I
I
I
I
A
I
I
A
I
I
I
I
I
I
A
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
A
I
I
A
I
I
I
I
A
I
I
I
I
I
A
I
I
I
A
I
I
I
A
I
A
A
I
A
I
I
I
I
I
I
I
A
I
I
I
A
I
A
I
I
I
I
A
I
A
I
This diff could not be displayed because it is too large.
This diff could not be displayed because it is too large.
# -*- encoding: utf-8 -*-
import os
from time import time
import argparse
from sklearn.naive_bayes import BernoulliNB
from sklearn.svm import SVC
from sklearn.neighbors import NearestCentroid
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, \
classification_report
import sys
from scipy.sparse import csr_matrix
import numpy as np
__author__ = 'CMendezC'
# Goal: training and validation binding thrombin data set
# Parameters:
# 1) --inputPath Path to read input files.
# 2) --inputTrainingData File to read training data.
# 3) --inputTestingData File to read testing data.
# 4) --inputTestingClasses File to read testing classes.
# 5) --outputModelPath Path to place output model.
# 6) --outputModelFile File to place output model.
# 7) --outputReportPath Path to place evaluation report.
# 8) --outputReportFile File to place evaluation report.
# 9) --classifier Classifier: BernoulliNB, SVM, NearestCentroid.
# Ouput:
# 1) Classification model and evaluation report.
# Execution:
# python training-validation-binding-thrombin.py
# --inputPath /home/compu2/bionlp/lcg-bioinfoI-bionlp/binding-thrombin-dataset
# --inputTrainingData thrombin.data
# --inputTestingData Thrombin.testset
# --inputTestingClasses Thrombin.testset.class
# --outputModelPath /home/compu2/bionlp/lcg-bioinfoI-bionlp/binding-thrombin-dataset/models
# --outputModelFile SVM-model.mod
# --outputReportPath /home/compu2/bionlp/lcg-bioinfoI-bionlp/binding-thrombin-dataset/reports
# --outputReportFile SVM.txt
# --classifier SVM
# source activate python3
###########################################################
# MAIN PROGRAM #
###########################################################
if __name__ == "__main__":
# Parameter definition
parser = argparse.ArgumentParser(description='Training validation Binding Thrombin Dataset.')
parser.add_argument("--inputPath", dest="inputPath",
help="Path to read input files", metavar="PATH")
parser.add_argument("--inputTrainingData", dest="inputTrainingData",
help="File to read training data", metavar="FILE")
parser.add_argument("--inputTestingData", dest="inputTestingData",
help="File to read testing data", metavar="FILE")
parser.add_argument("--inputTestingClasses", dest="inputTestingClasses",
help="File to read testing classes", metavar="FILE")
parser.add_argument("--outputModelPath", dest="outputModelPath",
help="Path to place output model", metavar="PATH")
parser.add_argument("--outputModelFile", dest="outputModelFile",
help="File to place output model", metavar="FILE")
parser.add_argument("--outputReportPath", dest="outputReportPath",
help="Path to place evaluation report", metavar="PATH")
parser.add_argument("--outputReportFile", dest="outputReportFile",
help="File to place evaluation report", metavar="FILE")
parser.add_argument("--classifier", dest="classifier",
help="Classifier", metavar="NAME",
choices=('BernoulliNB', 'SVM', 'NearestCentroid'), default='SVM')
(options, args) = parser.parse_args()
if len(args) > 0:
parser.error("None parameters indicated.")
sys.exit(1)
# Printing parameter values
print('-------------------------------- PARAMETERS --------------------------------')
print("Path to read input files: " + str(options.inputPath))
print("File to read training data: " + str(options.inputTrainingData))
print("File to read testing data: " + str(options.inputTestingData))
print("File to read testing classes: " + str(options.inputTestingClasses))
print("Path to place output model: " + str(options.outputModelPath))
print("File to place output model: " + str(options.outputModelFile))
print("Path to place evaluation report: " + str(options.outputReportPath))
print("File to place evaluation report: " + str(options.outputReportFile))
print("Classifier: " + str(options.outputFile))
# Start time
t0 = time()
print(" Reading training data and true classes...")
trainingClasses = []
trainingData = []
with open(os.path.join(options.inputPath, options.inputTrainingData), encoding='utf8', mode='r') \
as iFile:
for line in iFile:
line = line.strip('\r\n')
listLine = line.split(',')
trainingClasses.append(listLine[0])
trainingData.append(listLine[1:])
# trainingMatrix = np.matrix(trainingData)
trainingMatrix = csr_matrix(trainingData, dtype='double')
print("Number of training classes: {}".format(len(trainingClasses)))
print("Number of training class A: {}".format(trainingClasses.count('A')))
print("Number of training class I: {}".format(trainingClasses.count('I')))
print("Shape of training matrix: {}".format(trainingMatrix.shape))
print(" Reading testing data and true classes...")
testingClasses = []
testingData = []
with open(os.path.join(options.inputPath, options.inputTestingData), encoding='utf8', mode='r') \
as iFile:
for line in iFile:
line = line.strip('\r\n')
listLine = line.split(',')
testingData.append(listLine)
testingMatrix = csr_matrix(testingData, dtype='double')
with open(os.path.join(options.inputPath, options.inputTestingClasses), encoding='utf8', mode='r') \
as iFile:
for line in iFile:
line = line.strip('\r\n')
testingClasses.append(line)
print("Number of testing classes: {}".format(len(testingClasses)))
print("Number of testing class A: {}".format(trainingClasses.count('A')))
print("Number of testing class I: {}".format(trainingClasses.count('I')))
print("Shape of testing matrix: {}".format(testingMatrix.shape))
if options.classifier == "MultinomialNB":
classifier = BernoulliNB()
elif options.classifier == "SVM":
classifier = SVC()
elif options.classifier == "NearestCentroid":
classifier = NearestCentroid()
print(" Training...")
classifier.fit(trainingMatrix, trainingClasses)
print(" Done!")
print(" Testing (prediction in new data)...")
y_pred = classifier.predict(testingMatrix)
print(" Done!")
print(" Saving report...")
with open(os.path.join(options.outputPath, options.outputFile), mode='w', encoding='utf8') as oFile:
oFile.write('********** EVALUATION REPORT **********\n')
oFile.write('Classifier: {}\n'.format(options.classifier))
oFile.write('Accuracy: {}\n'.format(accuracy_score(testingClasses, y_pred)))
oFile.write('Precision: {}\n'.format(precision_score(testingClasses, y_pred, average='weighted')))
oFile.write('Recall: {}\n'.format(recall_score(testingClasses, y_pred, average='weighted')))
oFile.write('F-score: {}\n'.format(f1_score(testingClasses, y_pred, average='weighted')))
oFile.write('Confusion matrix: \n')
oFile.write(str(confusion_matrix(testingClasses, y_pred)) + '\n')
oFile.write('Classification report: \n')
oFile.write(classification_report(testingClasses, y_pred) + '\n')
print(" Done!")
print("Training and testing done in: %fs" % (time() - t0))