┌─┐┌─┐┌┐┌┌─┐┌─┐┬─┐┌┬┐┌─┐┌┐┌┌─┐┌─┐ ┌┐┌┌─┐ │ │ │││││ │ │├┬┘ ││├─┤││││ ├┤───│││├┤ └─┘└─┘┘└┘└─┘└─┘┴└──┴┘┴ ┴┘└┘└─┘└─┘ ┘└┘└ parameters description Set/Default ========== =========== ======= --debug Set to 'true' to test false --cores Regular job cores 4 --out Directory to output results concordance-<todays date> --sample_sheet fastq file (see help) sample_sheet.tsv --fq_prefix fastq file (see help) null --reference Reference Genome WS245/WS245.fa.gz --bamdir Location for bams bam --tmpdir A temporary directory tmp/ --email Email to be sent results HELP: http://andersenlab.org/dry-guide/pipeline-concordance/
The concordance pipeline is used to detect sample swaps, identify samples with quality issues, and determine which wild isolate strains should be grouped together as an isotype. When performing sequencing, we often sequence the same DNA library or strain multiple times in order to attain adequate coverage. To ensure that samples are labeled properly we examine whether they contain discordant variant calls based on what strain they are labeled as.
concordance-nf pipeline will proceed to group FASTQs labeled as a strain regardless of the quality of the data. Therefore, if issues are suspected the problemetic data needs to be removed and the pipeline rerun. More details are available below.
To determine which strains belong to the same isotype we use two criteria. First we look at the strains that group together with a concordance threshold of 99.9%. Generally this will group most isotypes without issue. However, it is possible that you will run into cases where the grouping is not clean. For example, strain A groups with B, B groups with C, but C does not group with A. In these cases you must examine the data closely to identify why strains are incompletely grouping. Our second criteria we use to group isotypes may address these types of groupings.
The second criteria that we use to group isotypes regards looking for regional differences among strains. If two strains are similar but possess a region of their genome (binned at 1 Mb) that differs by more than 2% then we will separate them out into their own isotypes.
The process of grouping isotypes is very hand-on. This pipeline will help process the data but you must carefully review the output and investigate closely.
There is at least one exception to the 99.9% cutoff rule we use to determine isotypes.
We classify LSJ1 and N2 as separate isotypes despite the fact that they are greater than 99.9% identical. The strains are known to have diverged from one another, but work has demonstrated significant genetic and phenotypic differences.
nextflow.config file sets most of the defaults you need to get the pipeline running locally for debugging purposes or on Quest.
Debugging the pipeline locally¶
The pipeline comes with a test dataset that you can use to make changes or fix problems. When running locally, you should install the andersen-lab-env which will install all the required dependencies.
Fetching the reference
You will need a reference genome to align to. You can fetch one by running the following command:
curl https://storage.googleapis.com/elegansvariation.org/genome/WS245/WS245.tar.gz > WS245.tar.gz tar -xvzf WS245.tar.gz
Run the pipeline locally by setting
nextflow run main.nf -profile debug -resume
Debugging the pipeline on Quest¶
When running on Quest, you should first run the quest debug profile. The Quest debug profile will use the test dataset and sample sheet which runs much faster and will encounter errors much sooner should they need to be fixed. If the debug dataset runs to completion it is likely that the full dataset will as well.
nextflow run main.nf -profile quest_debug -resume --email <your email>
Running the pipeline on Quest¶
The pipeline can be run on Quest using the following command:
nextflow run main.nf -profile quest -resume --email <your email>
The nextflow profiles configured in
nextflow.config are designed to make it so that you don't need to change the parameters. However, the pipeline offers this flexibility if it is ever called for.
The number of cores to use during alignments and variant calling.
A directory in which to output results. By default it will be
concordance-YYYY-MM-DD where YYYY-MM-DD is todays date.
The sample sheet to use. Normally you can use the
sample_sheet.tsv located in the base of the repo. This sample sheet is constructed usign
When running with
-debug the sample sheet located in
test_data/sample_sheet.tsv will be used.
More information on the sample sheet and adding new sequence data on the Sample sheet page section.
A prefix path for FASTQs defined in the sample sheet. The sample sheet designed for usage on Quest (
sample_sheet.tsv) uses absolute paths so no FASTQ prefix is necessary. It is set to
null by default.
A fasta reference indexed with BWA. On Quest, the reference is available here:
A directory for storing temporary data.
A directory to output strain-level BAM files to. On Quest this is set by default to
Set an email to get notified when the pipeline succeeds or fails.
An image showing the distribution of pairwise concordances across all strains. The cutoff is at 99.9% above which pairs are considered to be in the same isotype unless issues arise.
A close up view of the concordances showing more detail.
This is the most important output file. It illustrates the isotypes identified for each strain and identifies potential issues.
A file with the following structure:
- group - A number used to group strains (in each row) into an isotype automatically. This number should be unique with the isotype column (e.g. 1--> AB1, 112 --> CB4858, BRC20067 --> 175). The number can change between analyses.
- strain - the strain
- isotype - the currently assigned isotype for a strain taken from the
WI Strain Infospreadsheet. When new strains are added this is blank.
- coverage - Depth of coverage for strain.
- unique_isotypes_per_group - Number of unique isotypes when grouping by the group column. This should be 1. If it is more than 1, it indicates that multiple isotypes were assigned to a grouping and that a previously assigned isotype is now being called into question.
- unique_groups_per_isotype Number of unique groups assigned to an isotype. This should be 1. If it is higher than 1, it indicates that a strain is concordant with strains in two different isotypes (including blanks). If it is equal to 2 and contains blanks in the isotype column it likely means that an isotype should be assigned to that strain.
- strain_in_multiple_isotypes - Indicates that a strain is falling into multiple isotypes (a problem!).
- location_issue - Indicates a location issue. This occurs when strains fall into an isotype but are located far away from one another. Some are known issues and can be ignored.
- strain_conflict -
TRUEif any issue is present that should be investigated.
File produced using
bcftools gtcheck; Raw genotype differences between strains.
Gives a count of the number of isotypes identified.
Number of heterozygous sites/strain. Can be an indicator of mixed samples or other issues.
Intra-strain FASTQ-pair concordances. The format is:
Contains images showing locations where regional discordance occurs among strains classified as being the isotype. You must look through all these images to ensure there are no strains being grouped that have regions with significant differences (> 2%). The image below illustrates an example of this. ED3049 and ED3046 are highly similar (> 99.9%). However, they differ in a region on the right arm of chromosome II. We believe this was enough reason to consider them separate isotypes.
Summary of duplicate reads (determined by picard).
The following files are output in the fastq directory
fq_bam_idxstats.tsv # Stats generated with `samtools idxstats` fq_bam_stats.tsv # Stats generated with `samtools stats` fq_coverage.full.tsv # Detailed coverage numbers fq_coverage.tsv # Summary coverage of individual fastqs.
The following files are output in the strain directory
strain_bam_idxstats.tsv # Stats generated with `samtools idxstats` strain_bam_stats.tsv # Stats generated with `samtools stats` strain_coverage.full.tsv # Detailed coverage numbers strain_coverage.tsv # Summary coverage for strains.
The union-variant sitelist from all strains.
Locations of the union VCFs.
The raw VCF. The following filters are applied before calculating concordance:
min_depth>3 # Minimum Depth qual>30 # Quality (VCF=QUAL) mq>40 # Mapping Quality dv_dp=0.5 # DV/DP > 0.5 (high-quality allelic ALT bases over total depth) max_missing<0.05 # Max number of missing sites cannot exceed 5%
Filtered VCF, filtered for true SNPs (no monomorphic sites)
Stats from concordance vcf. Contains the number of SNPs.