SkylineRunner not importing RAW files schen19  2019-06-07 14:00
 

Hi Skyline Team,

I was trying to use the latest version of SkylineRunner to import peptide search results from Proteome Discoverer version 2.3. Everything goes smoothly and a Skyline file was generated. However, the chromatogram information is missing in the Skyline file, so I had to re-import all the RAW file again in order to get the chromatogram displayed. Although things look fine after the reimporting, I'm still hoping to solve the problem because importing a large number of raw files takes a really long time and having to import RAW files manually sort of defeats the point of using an automated pipeline...

Here is the structure of the script:
SkylineRunner.exe --in= --dir= --import-search-file= --import-fasta= --import-threads=4 --import-process-count=4 --import-search-cutoff-score=0 --out=

I didn't have the script for importing the raw file because the location of the raw files are in the folder specified by "--dir". I also didn't see any error message after running the script.

Please let me know if more information is needed in order to understand/solve the problem.

Thanks!
Shimin

 
 
Brendan MacLean responded:  2019-06-07 14:31

Hi Shimin,
This same issue came up for someone else very recently:

https://skyline.ms/announcements/home/support/thread.view?rowId=40426

First, we don't recommend that you use --import-threads= with --import-process-count=. I am not totally sure what this will do. I guess it might just be that --import-process-count=4 will entirely override the --import-threads=4, because importing will be limited to 4 separate processes at a time and even if the --import-threads=4 is passed to those separate processes, each one is given just a single file to import, so those threads would have no impact.

I would recommend you choose parallel import by threads or processes and use on or the other. In our experience --import-process-count=4 scales better than --import-threads=4 because importing 4 files in a single process bottlenecks on garbage collection.

Assuming you do choose --import-process-count=4 as I would recommend, then until we fix this issue (raised in the support request above), you will need to do your SkylineRunner operations in either 2 runs of SkylineRunner:

  1. Set up and save your targets and settings.
  2. Import results into the saved targets list.

Or you could achieve the same thing with a single SkylineRunner call, but using the ‑‑batch‑commands=path/to/file argument, where each line in the batch-commands file is a separate set of operations, but reloading the document is not required between lines.

I will add code to save the document between the operations that change it and the results import when --import-process-count is used. The issue is that the separate import processes read the document from disk. So, if you make changes to the document in memory but those changes don't get saved before the import processes start, they will read from disk the document as it was at the start of your SkylineRunner command.

The workaround is not so difficult. Hope this works for you until we can get a fix into a Skyline-daily coming soon.

--Brendan

 
schen19 responded:  2019-06-10 15:21

Hi Brendan,

Thanks for the quick response and the solution! I tried the workaround and it works perfectly!

Thanks for the hard work. I will be looking forward to the new update on Skyline-Daily.

Shimin