Processing of Tomograms using the FSU package

fiducial-free system
new scripts automate much hand-editing of files

list of scripts

setup-tilts.com OBSOLETE
makefiles.com OBSOLETE
tomo-step1.perl OBSOLETE
tomo-step1.com OBSOLETE
read-header.perl OBSOLETE
tomo-step2.perl OBSOLETE
make_refine_script.pl OBSOLETE
makeparam.pl OBSOLETE
setup-dir.com OBSOLETE
plot_all_shifts.pl OBSOLETE
plot_shifts.pl OBSOLETE
setup_dirs.pl -- setup directories for all .st files in current dir
tomo-step1t.pl
tomo-step2t.pl
protomo_setup.pl -- extract files from stack, make param, tlt, and ali.tlt files
fake_tlt_ali.pl -- makes a starting file for protomo with no initial alignment
imod_to_protomo_tlt_ali.pl -- makes a starting protomo alt.tlt file from .xg imod alignment file
renum_tlt.pl -- renumber tlt file so numbers are sequential (may be needed after nand editing of tlt file)
tomo-map.pl -- submit a script to make a tomogram to the cluster
findbigshifts.pl - list any images with shift in x or y >0.5 pixels (or as specified on command line), makes graphs for each iteration
protomo_deletebigshifts.pl -- delete images in stack with pixel shifts >0.5 pixels (or as specified)
protomo_refine.pl -- run iterative protomo refinement. stop and make tomogram when fully converged

Processing many tomograms on the cluster

change to your /usr/data/ directory
make a subdirectory for the files you want to process : eg. mkdir todays_date
change to that subdirectory: cd todays_date
transfer all tomograms to this directory, usually using sftp eg. sftp.nysbc.org
cd to folder eg. cd /mnt/cemdata/tecnai/username/temdata/...
get files eg. get *.st
when all files are present, run the following command to make subdirectory structure for each one:
- setup_dirs.pl
- for each .st file in this directory, the script will make the required directory structure for protomo, and move it to TOMONAME/raw/tmp/TOMONAME.st
to use same params for all tomograms, run the script protomo_make_answers.pl
- outputs the file answers.doc
For each tomogram, do the following procedure:
- cd TOMONAME/align
- protomo_setup.pl (REPLACES tomo-setp1.perl)
- answer all questions
- optionally, run tomo-ali.sh to do a manual alignment (needed for cryo, not usually needed for stained samples)
- qsub refine.script -- submits the refinement script to the cluster

a quick script to do it:

      for f in *; do
         cd $f/align
         protomo_setup.pl < ../../answers.doc
         qsub refine.script
         cd ../../
       done

Refinement script will now be running. Script make take many hours to run. It uses one CPU and 2GB RAM. Qsub takes care of all scheduling of this job

Processing with EmIP

EmIP has a tomogram tab, which gives a GUI for all setup and running of a refinement job
Can use this instead of above shell commands on a local workstation
Cluster job submission not yet implemented

Pre-processing with eTomo

Sometimes etomo works better for getting rid of bad pixels, doing an initial alignment
To do this, before running protomo_setup.pl, change to ../raw/tmp and run etomo
Choose the initial stack as input, scan header and then click "make com scripts"
Run the pre-processing, and after making corrected stack, choose to "use corrected stack"
Do the Coarse alignment step
- Under advanced parameters, change binning of output prealigned stack to 1
Calculate cross correlation, make coarse aligned stack, and then view it to verify
At this point, quit etomo
File which wil be used is TOMONAME.prexg
Can delete the prealigned stack (rm *.preali)
Need to rename original stack so protomo_setup.pl will only find one .st file in this directory
- mv TOMONAME_orig.st TOMONAME_orig.stack
cd ../../align
run protomo_setup.pl. It will find the .prexg file and use it to make initial alignments.

Analyzing results

Each run outputs the following files:
- refine-nn.log -- logfile from tomo-refine.sh
- TOMOGRAM-nn-xx-cof.ps -- coefficients of fit from run nn. xx is middle-tilt number
- TOMOGRAM-nn-fit-cof.ps -- fitted coefficients after running tomo-fit.sh
- TOMOGRAM-nn-xx-rot.ps -- plots of in-plane rotations
- TOMOGRAM-nn-fit-rot.ps -- fitted plot of in-plane rotations
- TOMOGRAM-nn-xx-coa.ps
- TOMOGRAM-nn-fit-coa.ps
Run the script findbigshifts.pl
- outputs refine-nn.ps for each run, a postscript file showing pixel shifts from previous iteration
- calculates the maximum and average pixel shift for each run
- also lists the images in each run which moved more than 0.5 pixel
  - limit can be specified on command line -- eg
```
findbigshifts.pl 1.0
```
Ideal refinement convergence will have resulted in minimal pixel shifts (<0.5 pixel)
examine .ps files with ghostscript (eg. gs TOMOGRAM-nn-fit-cof.ps)
cof -- ideally, should be between 0.99 and 1.01 for a good refinement (gives a measure of error)
rot -- should be small (<1 degree usually), and follow a pattern, not jump all over the place
Try to pick the iteration with minimal shifts, cof values closest to 1, rot values <1 (or at least following a pattern)
Usually, the worst values are found at the ends (high tilt or low tilt)
To automatically delete images with shifts >0.5, run protomo_deletebugshifts.pl
- for different shift cutoff, specify on command line -- protomo_deletebugshifts.pl 1.0
- saves original tiltfile as -orig
Can manually delete bad images as follows:
- Manually edit the TOMOGRAM-nn-fitted.tlt file
- Delete the bad images. For each image, there are three lines to be deleted, as the tlt file has three sections:
  - IMAGE 0 FILE N637Gc411500xpt25ta_000
  - IMAGE 0 ORIGIN [ 1016.527 1032.698 ]
  - IMAGE 0 TILT ANGLE -68.030 ROTATION 0.112
- Save the edited file
- If the only images which were deleted were at extremes (image numbers still consecutive, but perhaps not starting at 0), you are done
- If the images were in the middle, tlt file needs to be renumbered to make it consecutive again
  - /usr/apps/script.renum_tlt.pl will do this
  - Should consider manually running another iteration after this, as the "bad" image may have affected the good images which came after it.
Once the best iteration is found, and TOMOGRAM-nn-fitted.tlt edited as needed, use these parameters to generate the tomogram
On the cluster, run tomo-map.pl TOMOGRAM-nn.param
- This will use the fitted.tlt file to make the tomogram, submit the job via qsub, and at the end, convert the tomogram to MRC (16-bit integer) format
If manually running, tomo-map.sh TOMOGRAM-nn.param TOMOGRAM-nn-fitted.tlt
- Output will be in FSU format. To change to mrc:
  - cd ../out
  - cutimage -fmt MRC TOMOGRAM-nn-bck.img TOMOGRAM-nn-bck

Merging orthogonal tilt pairs

To fill in the missing wedge (ie have a missing cone) it is best to collect pairs of tomograms, rotated 90-degrees in plane
Use etomo to merge
probably better to NOT do this on the cluster, as imod is only installed on the head node
if files are named according to KD's convention (TOMONAMEt.st and TOMONAMEd90t.st) the following scripts will automate the procedure:
- /cryoem/script/make_merge_dirs.pl -- makes new directory TOMONAME, moves TOMONAMEt and TOMONAMEd90t directories into it, and also makes TOMONAME/merge
- cd TOMONAME/merge
- setup_mergedir.pl -- makes links to original stack files, tomograms, renames them according to imod protocols, and makes .tlt files required by imod
if files are named according to Imod/Serialem convention (TOMONAMEa.st and TOMONAMEb.st), then the appropriate scripts are
- make_merg_dirs_ab.pl
- setup_mergedir_ab.pl
once appropriate links are made in merge directory, run prepare_merge.pl. This rotates the tomograms, as required for imod processing
run etomo, and start at the "merge" step

Using Protomo Alignment Parameters in eTomo

File size of protomo volumes is limited to 2 GB, which can be a problem for large tomograms (EDIT - this is no longer a problem with 64-bit version of 1.14)
Following procedure will allow reconstruction in eTomo (will combine into program later)

Applying Protomo Alignment

Decide which iteration will be used for alignment parameters
Edit corresponding param file as follows
- change alibox_x and alibox_y to size you will want to reconstruct
- change alismp variable to 1.0
Run protomo_check.pl to view the final aligned stack created by a given tltfile. Outputs aligned image stack ../out/{filename}.box
This can be viewed with imod or displayimage
Run img2mrc16.pl. Input is ../out/{filename}.box. Change output name to {filename}.mrc
Run label. Choose option 2, then choose to add enough to make all image values positive. Make output {filename}.st
Now have an aligned stack

Setting up files

Set up a directory containing the following files:
- adjusted aligned stack, or stacks from image pairs {filename}a.st and {filename}b.st
- .param and fitted.tlt files from the best protomo alignment
Run /cryoem/script/extract_tlt.pl, choosing itr.tlt as input and {filename}.rawtlt as output
Run /cryoem/script/make_xf.pl -- will make imod ".xf" files for rotating the images so that tilt axis is vertical (gets azimuth from fitted.tlt files)

For each stack run

 newstack -input {filename}a.st -output {filename}a.ali -xform {filename}a.xf

For each stack, run
```
 mrctaper {filename}a.ali 
```
copy rawtlt files to tlt files:
```
 cp {filename}.rawtlt {filename}.tlt 
```
Now have rotated aligned stacks and lists of tilt angles, enough for tomogram reconstruction

Run eTomo

start eTomo
Choose .st files as input. Manually enter pixel size to correct size, fiducial size to "10" or whatever, tilt axis angle to 0. Choose "Tilt angles in existing rawtlt file".
Click "Make scripts"
Jump directly to Generate tomogram step.Change tomogram depth from 100 to the proper z-depth, and enter compyters to be used if running in parallel
Click "Generate Tomogram"
Repeat above for tomogram B if doing an orthogonal pair
Go on to merging step if doing an orthogonal pair

Using Protomo Alignment Parameters in Spider

Spider does not have a size limitation for the reprojected volume, and also has iterative back-projection methods (eg BP S2) which may work better

Setup

Follow instructions for etomo conversion to the point of getting an aligned, non-negative stack and a .rawtlt file
Run script rawtlt_to_spider.pl to make a spider "angles" file for reconstruction

Convert aligned stack to spider format:

/cryoem/script/extract.pl {filename}.ali 
      foreach F {filename}*.mrc 
      >proc2d $F {filename}.spd spider 
      >rm -f $F

Application

Now have aligned spider stack and angles file
start spider, and used desired back projection algorithm (bp s2, bp w2,

More detailed instructions, mostly dealing with doing everything more manually

Initial Setup for images collected with SerialEM software

Set up directories Command -- /cryoem/script setup-dir.com [tomoname] --
Change to [tomoname]/align subdirectory Command -- cd [tomoname]/align
Copy (or link) negative series into ../raw/tmp/[tomoname]a.st Command -- cp /dl380/home/tomo/xxx_0_to_-70.st ../raw//tmp/[tomoname]a.st --
Copy (or link) positive series into ../raw/tmp2/[tomoname]b.st Command -- cp /dl380/home/tomo/xxx_2_to_70.st ../raw/tmp2/[tomoname]b.st --
Extract and clean files. Outputs name of 0 degree (reference) image Command -- /cryoem/script/tomo-step1.perl
Examine 0-tilt image using displayimage to Trash.findDFdf center of interest. Command --displayimage ../raw/[tomoname]_XXX.mrc
Finish initial file preparation. Command: /cryoem/script/tomo-step2.perl

Initial setup for images collected with Tietz software.

Set up directories Command -- /cryoem/script setup-dir.com [tomoname] --
Change to [tomoname]/align subdirectory Command -- cd [tomoname]/align
Copy negative series of .dat files into ../raw/tmp/
Copy positive series of files into ../raw/tmp2/ . NOTE: basenames do not have to be same as negative series. Negative series basename will be final basename
Clean files, do part 1 of tomo preparation. Command -- tomo-step1t.pl
Examine 0-tilt image using displayimage to Trash.findDFdf center of interest. Command --displayimage ../raw/[tomoname]_XXX.mrc
Finish initial file preparation. Command: /cryoem/script/tomo-step2t.pl

FSU Software Alignment procedure

Copy a previous parameter file to current directory. Command -- cp ../../[tomo-old]/align/[tomo-old].param [tomoname]. param
Edit [tomoname].param to change reference section to actual number of 0-degree tilt image (examine [tomoname].tlt)
Perform rough alignment. Zoom out 3X, then choose "Align All". Command: tomo-ali.sh [tomoname].param [tomoname].tlt

Iteratative Alignment Procedure

Set up for iteration 1: cp [tomoname].param [tomoname]-1.param
cp [tomoname]-ali.tlt [tomoname]-1-itr.tlt
refine [tomoname]-1.param >& refine-01.log -- refinement. Takes hours.
tomo-fit.sh [tomoname]-1.param -- curve fitting of tilt
Examine fitting parameters: gs [tomoname]-1-??-cof.ps
Set up for iteration 2: cp [tomoname]-1.param [tomoname]-2.param
cp [tomoname]-1-fitted.tlt [tomoname]-2-itr.tlt
Re-run refine, but add 1 to all indices
Keep repeating until no more improvement seen.

Things to change: Correlation mode - xcf standard cross correlation
mcf - mutual correlation
pcf - phase-only correlation
guess mode: try "true" for first few iterations

To check correlation results, cd ../out
montageimage [name].xcf [name].mon
displayimage [name].mon

To view boxed images of Nth iteration, tomo-map.sh [tomoname]-N.param
Calculate tomogram of best iteration: tomo-map.sh [tomoname]-N.param
Maps and box files are in ../out/, in IMG format
View map with displayimage.
Convert tomogram to MRC: cutimage -fmt MRC [tomoname]-n-bck.img [tomoname]-n-bck.mrc
Fix header: fixheader -mrc2000 [tomoname]-n-bck.mrc

Typical Script for refinement

Save the following in a file called "refine.script".
In a text editor, replace "cdesmo33" with your tomogram basename.
Run it using the unit at command:
> at -f refine.script 16:30 (runs the script in the textfile "refine.script" at 4:30 pm)

----------------------cut here-------------------------

cat cdesmo33.param | sed 's/guess=false/guess=true/' > cdesmo33-01.param
cp cdesmo33-ali.tlt cdesmo33-01-itr.tlt
tomo-refine.sh cdesmo33-01.param >& refine-01.log
tomo-fit.sh cdesmo33-01.param
cp cdesmo33-01.param cdesmo33-02.param
cp cdesmo33-01-fitted.tlt cdesmo33-02-itr.tlt
tomo-refine.sh cdesmo33-02.param >& refine-02.log
tomo-fit.sh cdesmo33-02.param
cat cdesmo33-02.param | sed 's/guess=true/guess=false/' > cdesmo33-03.param
cp cdesmo33-02-fitted.tlt cdesmo33-03-itr.tlt
tomo-refine.sh cdesmo33-03.param >& refine-03.log
tomo-fit.sh cdesmo33-03.param
cp cdesmo33-03.param cdesmo33-04.param
cp cdesmo33-03-fitted.tlt cdesmo33-04-itr.tlt
tomo-refine.sh cdesmo33-04.param >& refine-04.log
tomo-fit.sh cdesmo33-04.param
cp cdesmo33-04.param cdesmo33-05.param
cp cdesmo33-04-fitted.tlt cdesmo33-05-itr.tlt
tomo-refine.sh cdesmo33-05.param >& refine-05.log
tomo-fit.sh cdesmo33-05.param
cat cdesmo33-05.param | sed 's/cormod=xcf/cormod=mcf/' > cdesmo33-06.param
cp cdesmo33-05-fitted.tlt cdesmo33-06-itr.tlt
tomo-refine.sh cdesmo33-06.param >& refine-06.log
tomo-fit.sh cdesmo33-06.param
cp cdesmo33-06.param cdesmo33-07.param
cp cdesmo33-06-fitted.tlt cdesmo33-07-itr.tlt
tomo-refine.sh cdesmo33-07.param >& refine-07.log
tomo-fit.sh cdesmo33-07.param
cp cdesmo33-07.param cdesmo33-08.param
cp cdesmo33-07-fitted.tlt cdesmo33-08-itr.tlt
tomo-refine.sh cdesmo33-08.param >& refine-08.log
cat cdesmo33-08.param | sed 's/cormod=mcf/cormod=pcf/' > cdesmo33-09.param
tomo-fit.sh cdesmo33-08.param
cp cdesmo33-08-fitted.tlt cdesmo33-09-itr.tlt
tomo-refine.sh cdesmo33-09.param >& refine-09.log
tomo-fit.sh cdesmo33-09.param
cp cdesmo33-09.param cdesmo33-10.param
cp cdesmo33-09-fitted.tlt cdesmo33-10-itr.tlt
tomo-refine.sh cdesmo33-10.param >& refine-10.log
tomo-fit.sh cdesmo33-10.param
cp cdesmo33-10.param cdesmo33-11.param
cp cdesmo33-10-fitted.tlt cdesmo33-11-itr.tlt
tomo-refine.sh cdesmo33-11.param >& refine-11.log
tomo-fit.sh cdesmo33-11.param
cp cdesmo33-11.param cdesmo33-12.param
cp cdesmo33-11-fitted.tlt cdesmo33-12-itr.tlt
tomo-refine.sh cdesmo33-12.param >& refine-12.log
tomo-fit.sh cdesmo33-12.param
cp cdesmo33-12.param cdesmo33-13.param
cp cdesmo33-12-fitted.tlt cdesmo33-13-itr.tlt
tomo-refine.sh cdesmo33-13.param >& refine-13.log
tomo-fit.sh cdesmo33-13.param
cp cdesmo33-13.param cdesmo33-14.param
cp cdesmo33-13-fitted.tlt cdesmo33-14-itr.tlt
tomo-refine.sh cdesmo33-14.param >& refine-14.log

----------------------cut here-------------------------

-- BillRice - 30 Mar 2006 * tomo-scripts.tgz: scripts described on this page

* protomo_scripts_dec_2010.zip: protomo scripts dec 2010