Merge remote-tracking branch 'origin/master'

.gitignore

@@ -56,7 +56,7 @@ docs/_build/
 # PyBuilder
 target/
 .idea/
-data/
+.data/
 
 config.ini
 data.ini

LICENSE.md

@@ -0,0 +1,8 @@
+Copyright (c) 2016 Sebastian Proost and Marek Mutwil; Max Planck Institute for Molecular Plant Physiology
+
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

README.md

@@ -1,28 +1,32 @@
 # LSTrAP
+
 LSTrAP, shot for Large Scale Transcriptome Analysis Pipeline, greatly facilitates the construction of co-expression networks from
 RNA Seq data. The various tools involved are seamlessly connected and  CPU-intensive steps are submitted to a computer cluster 
 automatically. 
 
 ## Workflow
+
 LSTrAP wraps multiple existing tools into a single workflow. To use LSTrAP the following tools need to be installed
 
 ![LSTrAP Workflow](docs/images/LSTrAP_workflow.png "Steps automated by LSTrAP")
 
 Steps in bold are submitted to a cluster.
 
 ## Preparation
+
 LSTrAP is designed to run on an [Oracle Grid Engine](https://www.oracle.com/sun/index.html) computer cluster system and requires 
 all external tools to be installed on the compute nodes. The "module load" system is supported. A comprehensive list of all tools 
 necessary can be found  [here](docs/preparation.md)
 
 ## Installation
+
 Use git to obtain a copy of the LSTrAP code
 
-    git clone https://github.molgen.mpg.de/proost/RSTrAP
+    git clone https://github.molgen.mpg.de/proost/LSTrAP
 
 Next, move into the directory and copy **config.template.ini** and **data.template.ini**
 
-    cd RSTrAP
+    cd LSTrAP
     cp config.template.ini config.ini
     cp data.template.ini data.ini
 
@@ -38,6 +42,7 @@ The location of the transcriptome data, the refrence genome and a few per-specie
 Detailed instruction how to set up both configuration files can be found [here](docs/configuration.md)
 
 ## Running LSTrAP
+
 Once properly configured for your system and data, LSTrAP can be run using a single simple command (that should be executed on the head node)
 
     ./run.py config.ini data.ini
@@ -47,15 +52,38 @@ Options to skip certain steps of the pipeline are included, use the command belo
     ./run.py -h
 
 ## Quality report
+
 After running LSTrAP a log file (*lstrap.log*) is written, in which samples which failed a quality measure
 are reported. Note that no samples are excluded from the final network. In case certain samples need to be excluded
 from the final network remove the htseq file for the sample you which to exclude and re-run the pipeline skipping all
 steps prior to building the network.
 
     ./run.py config.ini data.ini --skip-interpro --skip-orthology --skip-bowtie-build --skip-trim-fastq --skip-tophat --skip-htseq --skip-qc
+
+More information on how the quality of samples is determined can be found [here](docs/quality.md).
+
+## Helper Scripts
+
+LSTrAP comes with a few additional scripts to assist users to download and process data from the [Sequence Read Archive](http://www.ncbi.nlm.nih.gov/sra),
+repeat analyses and the case study reported in the manuscript (Proost et al., *under preparation*).
+
+Details for each script can be found [here](docs/helper.md)
+
+## Running LSTrAP on transcriptome data
+
+To use LSTrAP on a *de novo* assembled transcriptome a little pre-processing is required. Instead of the genome a fasta 
+file containing **coding** sequences can be used (remove UTRs). Using the helper script fasta_to_gtf.py a gtf file suited
+for LSTrAP can be generated.
+
+    python3 fasta_to_gtf.py /path/to/transcript.cds.fasta > output.gtf
 
+
 ## Contact
 
 LSTrAP was developped by [Sebastian Proost](mailto:proost@mpimp-golm.mpg.de) and [Marek Mutwil](mailto:mutwil@mpimp-golm.mpg.de) at the [Max-Planck Institute for Molecular Plant Physiology](http://www.mpimp-golm.mpg.de/2168/en)
 
-This work is supported by [ERA-CAPS](http://www.eracaps.org/) though the [EVOREPRO](http://www.evorepro.org/) project.
+This work is supported by [ERA-CAPS](http://www.eracaps.org/) though the [EVOREPRO](http://www.evorepro.org/) project.
+
+## License
+
+LSTrAP is freely available under the [MIT License](LICENSE.md)

docs/helper.md

@@ -0,0 +1,32 @@
+# Additional scripts
+
+Scripts used to perform analyses reported in the LSTrAP manuscript (Proost et al., *under preparation*) are found in 
+*./helper*
+
+## htseq_count_stats.py and tophat_stats.py
+
+These scripts will extract the statistics used to assess the quality of samples. 
+
+    python3 htseq_count_stats.py ./path/to/htseq/files > output.txt
+    python3 tophat_stats.py ./path/to/tophat/output > output.txt
+
+## pca_powerlaw.py
+
+Script to perform a PCA analysis on the *Sorghum bicolor* data (case study) and draw the node degree distribution. The
+required data is included here as well. Note that this script requires sklearn and seaborn.
+
+    python3 pca_powerlaw.py ./data/sbi.expression.matrix.tpm.txt ./data/sbi_annotation.txt ./data/sbi.power_law.R07.txt
+
+## get_sra_ip.py
+
+Script to download runs from [Sequence Read Archive](http://www.ncbi.nlm.nih.gov/sra), requires the Aspera connect 
+client to be installed and a open ssh key is required (can be obtained from the Apera connect package)
+
+    python3 get_sra_ip.py runs.list.txt ./output/directory /absolute/path/to/opensshkey
+     
+## sra_to_fastq.py
+
+Script to convert sra files into fastq. Sratools is required.
+
+    python3 sra_to_fastq.py /sra/files/directory /fastq/output/directory
+

docs/preparation.md

@@ -5,13 +5,19 @@ tools need to be installed. A full list is provided below, tools can be installe
 When opting for the latter, the configuration file needs to contain the exact names of the modules containing the tools.
 
 
-  * [bowtie2](http://bowtie-bio.sourceforge.net/bowtie2/index.shtml)
-  * [tophat](https://ccb.jhu.edu/software/tophat/manual.shtml)
-  * [samtools](http://www.htslib.org/)
-  * [sratools](http://ncbi.github.io/sra-tools/)
-  * [python 2.7](https://www.python.org/download/releases/2.7/) + [HTSeq](http://www-huber.embl.de/users/anders/HTSeq/doc/index.html) + all dependencies (including [PySam](https://github.com/pysam-developers/pysam))
-  * [python 3.5](https://www.python.org/download/releases/3.5.1/) + SciPy + [Numpy](http://www.numpy.org/)
-  * [interproscan](https://www.ebi.ac.uk/interpro/)
+  * [Bowtie2](http://bowtie-bio.sourceforge.net/bowtie2/index.shtml)
+  * [TopHat](https://ccb.jhu.edu/software/tophat/manual.shtml)
+  * [Samtools](http://www.htslib.org/)
+  * [SRAtools](http://ncbi.github.io/sra-tools/)
+  * [Python 2.7](https://www.python.org/download/releases/2.7/) + [HTSeq](http://www-huber.embl.de/users/anders/HTSeq/doc/index.html) + all dependencies (including [PySam](https://github.com/pysam-developers/pysam))
+  * [Python 3.5](https://www.python.org/download/releases/3.5.1/) + SciPy + [Numpy](http://www.numpy.org/)
+  * [InterProScan](https://www.ebi.ac.uk/interpro/)
   * [OrthoFinder](https://github.com/davidemms/OrthoFinder)
   * [MCL](http://www.micans.org/mcl/index.html?sec_software)
-  * [Trimmomatic](http://www.usadellab.org/cms/?page=trimmomatic)
+  * [Trimmomatic](http://www.usadellab.org/cms/?page=trimmomatic)
+
+Optional tools
+
+  * [scikit-learn](http://scikit-learn.org/) for Python 3, required for PCA analysis (helper script)
+  * [seaborn](https://stanford.edu/~mwaskom/software/seaborn/) for Python 3, required for PCA analysis (helper script)
+  * [Aspera connect client](), required for the *get_sra_ip.py* (helper script)

docs/quality.md

@@ -0,0 +1,32 @@
+# Quality Control
+
+
+## Overview
+LSTrAP is designed to process large volumes of RNA Seq data to ultimately construct co-expression networks. However, 
+while composing a sufficiently large dataset sample not suited to construct such networks might be erroneously included.
+To detect those samples and warn users LSTrAP checks two parameters for each sample. The percentage of reads that map
+to the genome (reported by TopHat) can be indicative for issues. The figure below shows proper samples (gray circles), 
+and the majority have very high mapping percentages (y-axis). Samples from species that are not *Arabidopsis thaliana*
+have fewer reads mapping. 
+
+![LSTrAP Quality](images/LSTrAP_quality.png "LSTrAP quality test")
+
+Also samples from experiments not designed to determine expression levels might be included, while some of these map 
+very good to the genome (e.g. whole genome sequencing, non-coding RNA or small RNA studies), the reads don't correspond 
+to coding genes. This can be seen in the percentage of mapped reads that align with coding genes (reported by 
+HTSeq-Count). For good samples the median value is roughly 60% while some types of unsuited samples have less.
+
+Based on this comparison LSTrAP will warn users if samples where less than 65% of the reads map to the genome are 
+included or samples were less than 40% of the mapped reads are located in coding genes. Samples are *not* automatically
+excluded from further analysis as there are (albeit a limited number) of good samples (e.g. single-cell transcriptomics
+tend to have fewer reads mapping onto the genome due to mutations introduced during amplificiation steps) that have low
+scores for one of these criteria.
+
+An overview of samples that could have potential problems can be found in *lstrap.log*
+
+## How to remove spureous samples
+
+In case *lstrap.log* contains samples that need to be removed, remove the corresponding .htseq file for that sample and
+restart the pipeline skipping all previous steps like this:
+
+    ./run.py config.ini data.ini --skip-interpro --skip-orthology --skip-bowtie-build --skip-trim-fastq --skip-tophat --skip-htseq --skip-qc