Updated S3 links and silenced installs

Author: EC2 Default User

Submodule1.ipynb

@@ -175,8 +175,8 @@
    "outputs": [],
    "source": [
     "#First let's install mamba to configure our environment\n",
-    "! curl -L -O \"https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh\"\n",
-    "! bash Miniforge3-$(uname)-$(uname -m).sh -u -b -p $HOME/mambaforge\n",
+    "! curl -L -O -s \"https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh\"\n",
+    "! bash Miniforge3-$(uname)-$(uname -m).sh -u -b -p $HOME/mambaforge > /dev/null 2>&1\n",
     "print(\"done\")"
    ]
   },
@@ -200,8 +200,8 @@
    "outputs": [],
    "source": [
     "#Now let's install several required packages. You will likely see \"warning libmamba\" messages. That is okay, please ignore these.\n",
-    "!mamba install -c bioconda fastqc bowtie2 multiqc trimmomatic cutadapt -y -q\n",
-    "!pip install --upgrade --force-reinstall jupyterquiz\n",
+    "!mamba install -q -c bioconda fastqc bowtie2 multiqc trimmomatic cutadapt -y -q\n",
+    "!pip install --upgrade --force-reinstall -q  jupyterquiz\n",
     "print(\"done\")"
    ]
   },
@@ -281,7 +281,7 @@
     "!mkdir -p $wd/Submodule1/Mapped\n",
     "\n",
     "#Let's copy and extract our tutorial files\n",
-    "!wget https://chromatinoccupancytutorial.s3.us-east-2.amazonaws.com/Submodule1.2.zip\n",
+    "!wget https://nigms-sandbox.s3.us-east-1.amazonaws.com/chromatinoccupancytutorial/Submodule1.2.zip\n",
     "!unzip Submodule1.2.zip\n",
     "print(\"done\")"
    ]
@@ -1020,7 +1020,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.16"
+   "version": "3.10.17"
   }
  },
  "nbformat": 4,

Submodule2.ipynb

@@ -151,8 +151,8 @@
    "outputs": [],
    "source": [
     "#First let's install mamba to configure our environment\n",
-    "! curl -L -O \"https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh\"\n",
-    "! bash Miniforge3-$(uname)-$(uname -m).sh -u -b -p $HOME/mambaforge\n",
+    "! curl -L -O -s \"https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh\"\n",
+    "! bash Miniforge3-$(uname)-$(uname -m).sh -u -b -p $HOME/mambaforge > /dev/null 2>&1\n",
     "print(\"done\")"
    ]
   },
@@ -164,9 +164,9 @@
    "outputs": [],
    "source": [
     "#now let's install several required packages\n",
-    "!mamba install -c bioconda picard samtools deeptools bedtools==2.27 ucsc-bedgraphtobigwig macs3 seacr meme manorm -y\n",
-    "!pip install jupyterquiz==2.0.7 jupytercards\n",
-    "!pip install --user igv-notebook\n",
+    "!mamba install -q -c bioconda picard samtools deeptools bedtools==2.27 ucsc-bedgraphtobigwig macs3 seacr meme manorm -y\n",
+    "!pip install -q jupyterquiz==2.0.7 jupytercards\n",
+    "!pip install -q --user igv-notebook\n",
     "print(\"done\")"
    ]
   },
@@ -254,7 +254,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# These commands move into our Tutorial 2 directory and create our subdirectory structure.\n",
+    "# These commands move into our Tutorial 1 directory and create our subdirectory structure.\n",
     "!mkdir -p $wd/Submodule2/\n",
     "%cd $wd/Submodule2/\n",
     "!mkdir -p $wd/Submodule2/Filtering\n",
@@ -271,7 +271,7 @@
    "outputs": [],
    "source": [
     "#Let's copy and extract our tutorial files\n",
-    "!wget https://chromatinoccupancytutorial.s3.us-east-2.amazonaws.com/Submodule2.zip\n",
+    "!wget https://nigms-sandbox.s3.us-east-1.amazonaws.com/chromatinoccupancytutorial/Submodule2.zip\n",
     "!unzip Submodule2.zip\n",
     "print(\"done\")"
    ]
@@ -362,7 +362,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "#list the files in CUTnRUNSam which we just downloaded.\n",
+    "#list the files in ChIPseqSam which we just downloaded.\n",
     "!ls $wd/Submodule2/Submodule2/CUTnRUNsam"
    ]
   },
@@ -371,7 +371,7 @@
    "id": "1df8945d",
    "metadata": {},
    "source": [
-    " You will notice that in addition to the mm39 sam files (mapped to mouse genome), we have additional sam files mapped to R64-1-1 (Saccharomyces cereviseae a.k.a the budding yeast) genome. These are for spike-in reads. In CUT&RUN and CUT&Tag, users add a constant and tiny amount of pre-fragmented heterogous DNA as \"spike-in\" (e.g., yeast spike-in for experiments targeting protein in mouse cells) to quantitatively compare signals or reads from the target genome. This helps to account for differences in cell number, DNA purification, library preparation, and sequencing efficiencies between controls and treatment sets. See PMID: PMID: [34117481](https://pubmed.ncbi.nlm.nih.gov/34117481/) "
+    " You will notice that in addition to the mm39 sam files (mapped to mouse genome), we have additional sam files mapped to R64-1-1 (Saccharomyces cereviseae a.k.a the budding yeast) genome. These are for spike-in reads. In CUT&RUN and CUT&Tag, users add a constant and tiny amount of pre-fragmented heterogous DNA as \"spike-in\" (e.g., yeast spike-in for experiments targeting protein in mouse cells) to quantitatively compare signals or reads from the target genome. This helps to account for differences in cell number, DNA purification, library preparation, and sequencing efficiencies between controls and treatment sets. See PMID: [31232687](https://pubmed.ncbi.nlm.nih.gov/31232687/)"
    ]
   },
   {
@@ -395,7 +395,7 @@
     "</div>In the cell below, let's view the first few lines of the bed file to see how they look.\n",
     "</div>\n",
     "\n",
-    "</div>Note: the first column (\"chrom\") denotes the chromosome number to which the reads mapped, the second and the third columns show the first and the last base pair positions of mapped fragment, \"chromStart\" and \"chromEnd\", respectively. Remember, we used paired-end sequencing, so we can obtain the coordinates of both the beginining an the end of each mapped fragment. This is the minimum requirement for a bed file. There can be up to 9 more columns. What are information can be there in a bed file? Learn more: [UCSC_BED_format] (https://genome.ucsc.edu/FAQ/FAQformat.html#format1)"
+    "</div>Note: the first column denotes the chromosome number to which the reads mapped, the second and the third columns show the first and the last base pair positions of mapped fragment. Remember, we used paired-end sequencing, so we can obtain the coordinates of both the beginining an the end of each mapped fragment."
    ]
   },
   {
@@ -415,9 +415,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "#Now let's make the bam and bed files for the treatment and IgG data.\n",
-    "#This may take a few minutes: 🕘\n",
-    "\n",
+    "#Now let's do the same thing for the treatment data and IgG files.\n",
     "!samtools view -b -h $wd/Submodule2/Submodule2/CUTnRUNsam/BRG1_CnR_FLV.mm39_chr4.sam -o $wd/Submodule2/Filtering/BRG1_CnR_FLV.mm39_chr4.bam\n",
     "!bedtools bamtobed -bedpe -i $wd/Submodule2/Filtering/BRG1_CnR_FLV.mm39_chr4.bam | cut -f 1,2,6 | sort -k1,1 -k2,2n -k3,3n > $wd/Submodule2/Filtering/BRG1_CnR_FLV.mm39_chr4.bed\n",
     "print(\"done with BRG1_CnR_FLV\")\n",
@@ -434,7 +432,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "#Let's make the spike-in bed files.\n",
+    "#Now let's make the spike-in bed files.\n",
     "!samtools view -b -h $wd/Submodule2/Submodule2/CUTnRUNsam/BRG1_CnR_con.R64-1-1.sam -o $wd/Submodule2/Filtering/BRG1_CnR_con.R64-1-1.bam\n",
     "!bedtools bamtobed -bedpe -i $wd/Submodule2/Filtering/BRG1_CnR_con.R64-1-1.bam | cut -f 1,2,6 | sort -k1,1 -k2,2n -k3,3n > $wd/Submodule2/Filtering/BRG1_CnR_con.R64-1-1.bed\n",
     "print(\"done with BRG1_CnR_con\")\n",
@@ -463,7 +461,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "#list the files in CUTnTagSam which we just downloaded.\n",
+    "#list the files in ChIPseqSam which we just downloaded.\n",
     "!ls $wd/Submodule2/Submodule2/CUTnTagSam"
    ]
   },
@@ -495,7 +493,8 @@
    "source": [
     "</div>In the cell below, let's view the first few lines of the bed file to see how they look.\n",
     "</div>\n",
-    "</div>Note: the first column (\"chrom\") denotes the chromosome number to which the reads mapped, the second and the third columns show the first and the last base pair positions of mapped fragment, \"chromStart\" and \"chromEnd\", respectively. Remember, we used paired-end sequencing, so we can obtain the coordinates of both the beginining an the end of each mapped fragment. This is the minimum requirement for a bed file. There can be up to 9 more columns. What are information can be there in a bed file? Learn more: [UCSC_BED_format] (https://genome.ucsc.edu/FAQ/FAQformat.html#format1)"
+    "\n",
+    "</div>Note: the first column denotes the chromosome number to which the reads mapped, the second and the third columns show the first and the last base pair positions of mapped fragment. Remember, we used paired-end sequencing, so we can obtain the coordinates of both the beginining an the end of each mapped fragment."
    ]
   },
   {
@@ -515,9 +514,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "#Let's make the bam and bed files for the treatment and IgG data.\n",
-    "#This may take a few minutes: 🕘\n",
-    "\n",
+    "#Now let's do the same thing for the treatment data and IgG files.\n",
     "!samtools view -b -h $wd/Submodule2/Submodule2/CUTnTagSam/RNAPII-S5P_CnT_FLV.mm39_chr4.sam -o $wd/Submodule2/Filtering/RNAPII-S5P_CnT_FLV.mm39_chr4.bam\n",
     "!bedtools bamtobed -bedpe -i $wd/Submodule2/Filtering/RNAPII-S5P_CnT_FLV.mm39_chr4.bam | cut -f 1,2,6 | sort -k1,1 -k2,2n -k3,3n > $wd/Submodule2/Filtering/RNAPII-S5P_CnT_FLV.mm39_chr4.bed\n",
     "print(\"done with RNAPII-S5P_CnT_FLV\")\n",
@@ -581,12 +578,12 @@
     "\n",
     "<img src=\"images/peaksofsignal.jpg\" alt=\"Drawing\" style=\"width: 500px;\"/>\n",
     "\n",
-    "<div class=\"alert-warning\" style=\"font-size:200%\">\n",
+    "<div class=\"alert-info\" style=\"font-size:200%\">\n",
     "    \n",
     "    \n",
     "<img src=\"images/CUT&RUNLogo.jpg\" alt=\"Drawing\" style=\"width: 120px;\" align=\"left\"/>\n",
     "    \n",
-    "<div class=\"alert-warning\" style=\"font-size:50%\">\n",
+    "<div class=\"alert-info\" style=\"font-size:50%\">\n",
     "</div>\n",
     "NOTE for CUT&RUN and CUT&Tag\n",
     "</div>\n",
@@ -685,7 +682,7 @@
    "metadata": {},
    "source": [
     "### We have previously learnt that we can see the coordinates of each read in the bam or bed files. However, this can be tideous and not practical for datasets with millions of reads. We need a better way of visualizing the results.\n",
-    "### In this step we will create files that summarize the pileup of reads at each base-pair along our genome, in [bedgraph](https://genome.ucsc.edu/goldenpath/help/bedgraph.html) or [bigwig](https://genome.ucsc.edu/goldenpath/help/bigWig.html) formats. \n",
+    "### In this step we will create a binary file that summarizes the pileup of reads at each base-pair along our genome, in [bedgraph](https://genome.ucsc.edu/goldenpath/help/bedgraph.html) or [bigwig](https://genome.ucsc.edu/goldenpath/help/bigWig.html) formats. \n",
     "\n",
     "For ChIP-seq datasets, we will create bigwig files using the command bamCoverage, part of the [deeptools](https://deeptools.readthedocs.io/en/develop/) package.\n",
     "\n",
@@ -761,33 +758,13 @@
     "display_quiz(\"../questions/BPMnorm.json\")"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "d4ca9524",
-   "metadata": {},
-   "source": [
-    "<div class=\"alert-info\" style=\"font-size:150%\">\n",
-    "How does spike-in calibration work?\n",
-    "</div>\n",
-    "In the example below, although sample A and sample B have about the same amounts of experimental DNA, sample B has more spike-in DNA which results in lower enrichment of expermental reads.\n",
-    "For spike-in calibration, we need to first count the number of spike-in reads in each sample."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "019e4e91",
-   "metadata": {},
-   "source": [
-    "<img src=\"images/spike-in.jpg\" alt=\"Drawing\" style=\"width: 300px;\"/>"
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "ecc66087",
    "metadata": {},
    "source": [
     "<img src=\"images/CUT&RUNLogo.jpg\" alt=\"Drawing\" style=\"width: 120px;\" align=\"left\"/> Run the following command for CUT&RUN datasets.\n",
-    "<div>We will first need to calculate the calibration factor for each sample by dividing a large constant number, such as 10,000, by the count of spike-in reads"
+    "<div>We will first need to calculate the calibration factor for each sample"
    ]
   },
   {
@@ -834,7 +811,7 @@
    "metadata": {},
    "source": [
     "<img src=\"images/CUT&TagLogo.jpg\" alt=\"Drawing\" style=\"width: 120px;\" align=\"left\"/> Run the following command for CUT&Tag datasets.\n",
-    "<div>We will first need to calculate the calibration factor for each sample by dividing a large constant number, such as 10,000, by the count of spike-in reads"
+    "<div>We will first need to calculate the calibration factor for each sample"
    ]
   },
   {
@@ -963,14 +940,14 @@
     "igv_notebook.init()\n",
     "myigv = igv_notebook.Browser(\n",
     "    {\n",
-    "        \"genome\": \"mm39\",\n",
+    "        \"genome\": \"mm10\",\n",
     "        \"locus\": \"chr4:10,000,000-150,000,000\"\n",
     "    }\n",
     ")\n",
     "myigv.load_track(\n",
     "{\n",
     "        \"name\": \"BRG1_CnR_con\",\n",
-    "        \"url\": \"Submodule2/Visualization/BRG1_CnR_con.mm39_chr4.bw\",\n",
+    "        \"url\": \"Submodule2/Submodule2/Viz/BRG1_CnR_con.bw\",\n",
     "        \"format\": \"bigwig\",\n",
     "        \"type\": \"wig\"\n",
     "    }\n",
@@ -979,7 +956,7 @@
     "myigv.load_track(\n",
     "{\n",
     "        \"name\": \"BRG1_CnR_FLV\",\n",
-    "        \"url\": \"Submodule2/Visualization/BRG1_CnR_FLV.mm39_chr4.bw\",\n",
+    "        \"url\": \"Submodule2/Submodule2/Viz/BRG1_CnR_FLV.bw\",\n",
     "        \"format\": \"bigwig\",\n",
     "        \"type\": \"wig\"\n",
     "    }\n",
@@ -988,7 +965,7 @@
     "myigv.load_track(\n",
     "{\n",
     "        \"name\": \"IgG_CnR\",\n",
-    "        \"url\": \"Submodule2/Visualization/IgG_CnR.mm39_chr4.bw\",\n",
+    "        \"url\": \"Submodule2/Submodule2/Viz/IgG_CnR.bw\",\n",
     "        \"format\": \"bigwig\",\n",
     "        \"type\": \"wig\"\n",
     "    }\n",
@@ -1015,14 +992,14 @@
     "igv_notebook.init()\n",
     "myigv = igv_notebook.Browser(\n",
     "    {\n",
-    "        \"genome\": \"mm39\",\n",
+    "        \"genome\": \"mm10\",\n",
     "        \"locus\": \"chr4:10,000,000-150,000,000\"\n",
     "    }\n",
     ")\n",
     "myigv.load_track(\n",
     "{\n",
     "        \"name\": \"RNAPII-S5P_CnT_con\",\n",
-    "        \"url\": \"Submodule2/Visualization/RNAPII-S5P_CnT_con.mm39_chr4.bw\",\n",
+    "        \"url\": \"Submodule2/Submodule2/Viz/RNAPII-S5P_CnT_con.bw\",\n",
     "        \"format\": \"bigwig\",\n",
     "        \"type\": \"wig\"\n",
     "    }\n",
@@ -1031,7 +1008,7 @@
     "myigv.load_track(\n",
     "{\n",
     "        \"name\": \"RNAPII-S5P_CnT_FLV\",\n",
-    "        \"url\": \"Submodule2/Visualization/RNAPII-S5P_CnT_FLV.mm39_chr4.bw\",\n",
+    "        \"url\": \"Submodule2/Submodule2/Viz/RNAPII-S5P_CnT_FLV.bw\",\n",
     "        \"format\": \"bigwig\",\n",
     "        \"type\": \"wig\"\n",
     "    }\n",
@@ -1040,7 +1017,7 @@
     "myigv.load_track(\n",
     "{\n",
     "        \"name\": \"IgG_CnT\",\n",
-    "        \"url\": \"Submodule2/Visualization/IgG_CnT.mm39_chr4.bw\",\n",
+    "        \"url\": \"Submodule2/Submodule2/Viz/IgG_CnT.bw\",\n",
     "        \"format\": \"bigwigh\",\n",
     "        \"type\": \"wig\"\n",
     "    }\n",
@@ -1053,7 +1030,7 @@
    "id": "6fa8ee5e-024c-44d6-9f35-c223bb049f26",
    "metadata": {},
    "source": [
-    "<div class=\"alert-info\" style=\"font-size:150%\">\n",
+    "<div class=\"alert-info\" style=\"font-size:120%\">\n",
     "Average Profiles\n",
     "</div>\n",
     "\n",
@@ -1127,7 +1104,7 @@
    "source": [
     "%cd $wd\n",
     "# -S option specifies the bigwig signal file, where we can specify multiple separated by spaces. -R option specifies the genome annotation bed file. -a and -b specify how many bp to plot on either side. \n",
-    "!computeMatrix reference-point -S Submodule2/Visualization/BRG1_CnR_con.mm39_chr4.bw Submodule2/Visualization/BRG1_CnR_FLV.mm39_chr4.bw -R Submodule2/Submodule2/GenomeAnnotations/mm39v36_chr4_genes.bed -o Submodule2/Visualization/CUTnRUN_GeneprofileMatrix -a 5000 -b 5000\n",
+    "!computeMatrix reference-point -S Submodule2/Submodule2/Viz/BRG1_CnR_con.bw Submodule2/Submodule2/Viz/BRG1_CnR_FLV.bw -R Submodule2/Submodule2/GenomeAnnotations/mm39v36_chr4_genes.bed -o Submodule2/Visualization/CUTnRUN_GeneprofileMatrix -a 5000 -b 5000\n",
     "print(\"done creating the matrix... plotting....\")\n",
     "!plotProfile -m Submodule2/Visualization/CUTnRUN_GeneprofileMatrix -o Submodule2/Visualization/CUTnRUN_Geneprofile.png --perGroup\n",
     "print(\"done\")"
@@ -1163,7 +1140,7 @@
    "source": [
     "%cd $wd\n",
     "# -S option specifies the bigwig signal file, where we can specify multiple separated by spaces. -R option specifies the genome annotation bed file. -a and -b specify how many bp to plot on either side. \n",
-    "!computeMatrix reference-point -S Submodule2/Visualization/RNAPII-S5P_CnT_con.mm39_chr4.bw Submodule2/Visualization/RNAPII-S5P_CnT_FLV.mm39_chr4.bw -R Submodule2/Submodule2/GenomeAnnotations/mm39v36_chr4_genes.bed -o Submodule2/Visualization/CUTnTag_GeneprofileMatrix -a 5000 -b 5000\n",
+    "!computeMatrix reference-point -S Submodule2/Submodule2/Viz/RNAPII-S5P_CnT_con.bw Submodule2/Submodule2/Viz/RNAPII-S5P_CnT_FLV.bw -R Submodule2/Submodule2/GenomeAnnotations/mm39v36_chr4_genes.bed -o Submodule2/Visualization/CUTnTag_GeneprofileMatrix -a 5000 -b 5000\n",
     "print(\"done creating the matrix... plotting....\")\n",
     "!plotProfile -m Submodule2/Visualization/CUTnTag_GeneprofileMatrix -o Submodule2/Visualization/CUTnTag_Geneprofile.png --perGroup\n",
     "print(\"done\")"
@@ -1216,9 +1193,6 @@
     "</div>\n",
     "In ChIP-seq, CUT&RUN, and CUT&Tag it is important to control for non-specific signal enrichment. The processing steps up until this point are exactly the same for those control samples, but we'll use the control samples during peak calling to provide a background level.\n",
     "\n",
-    "### Note that control samples are of different kinds for ChIP-seq (input) and CUT&RUN or CUT&Tag (IgG/non-specific antibody).\n",
-    "\n",
-    "\n",
     "<img src=\"images/ChIPseqLogo.jpg\" alt=\"Drawing\" style=\"width: 100px;\" align=\"left\" /> Run the following command for ChIP-seq datasets.\n",
     "</div>\n",
     "</div>We've prepared the bam files for the Input control samples for you, and specify them in the following command.  "
@@ -1377,7 +1351,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "a9172dde",
+   "id": "891987fa-70a6-4ff4-bef8-06fc08b2ff78",
    "metadata": {},
    "outputs": [],
    "source": []
@@ -1435,7 +1409,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "e0ee26ca",
+   "id": "75945418-183f-4acb-a048-b5c83d7926a5",
    "metadata": {},
    "outputs": [],
    "source": []
@@ -1480,14 +1454,6 @@
     "</div>"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "9da9a098",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  },
   {
    "cell_type": "markdown",
    "id": "2a5bf9f4-6814-4889-86ad-02c9088cbecd",