The results evaluation of RNAseq aligners on synthetic and real datasets

Evaluation synthetic datasets

The evaluation of synthetic dataset was inspired by RNAseqEval project (https://github.com/kkrizanovic/RNAseqEval). deSALT compares the alignment files (SAM or BAM) to the simulation data generation by PBSIM which have ground truth. In order to reveal the performance of aligners, we take the potential structure of simulation data into consideration. We evaluate the results from four aspects:

1. the overview of evaluation;
2. seperate the results into two groups, one for genes with short exons (< 30bp) and one for genes without short exons. Then evaluate the two groups respectively.
3. seperate the results into two groups, one for gene with single splicing isoform and one for genes with alternative splicing isoforms. Then evaluate the two groups respectively.
4. seperate the results into three groups, one for genes contian less than 6 exons, one for genes contain 6-9 exons and one for genes contain more than 9 exons. Then evaluate the three groups respectively.

The script Eval_PBSim_data.py and Eval_NanoSim.py was specially designed for benchmarking.

Usage example:

python Eval_PBSim_data.py simulation_folder alignment.sam annotations.gtf group_list All_SS_iso.txt out.csv

[1] simulation_folder: the folder containing simulation datasets generated by PBSIM.

[2] alignment.sam: the alignment results (SAM) of simulation datasets by aligners.

[3] annotations.gtf: the annotations of reference genomes.

[4] group_list: the list of names of each group where simulation data were simulated with different sequencing depth by PBSIM.

[5] All_SS_iso.txt: The transcript id of all single splicing isoforms.

[6] out.csv: the results of evaluation.

python Eval_NanoSim_data.py input_read.fa alignment.sam annotations.gtf All_SS_iso.txt out.csv

[1] input_read.fa: the original RNA-seq reads for alignment.

[2] alignment.sam: the alignment results (SAM) of simulation datasets by aligners.

[3] annotations.gtf: the annotations of reference genomes.

[4] All_SS_iso.txt: The transcript id of all single splicing isoforms.

[5] out.csv: the results of evaluation.

The All_SS_iso.txt can be obtained using script GetSingleSpliceTranID.py:

python GetSingleSpliceTranID.py annotation.gtf

Evaluation real datasets

As for the evaluation of real datasets, we compare the alignment files to corresponding annotations. For each alignment, we find the most overlapped transcript in annotations as aligned transcript, then detect the overlapped exons and calculate the boundaries to make a decision whether the alignment is a good alignment. The evaluation was down by Eval_real_data.py script.

Usage example:

python Eval_real_data.py alignment.sam annoatations.gtf thread > eval_result.txt

[1] alignment.sam: the alignment results (SAM) of simulation datasets by aligners.

[2] annotations.gtf: the annotations of reference genomes.

[3] thread: the evaluation program supports multithreading.

The python library pysam is needed for evaluation.