Error in using Skyline command line minliecn  2020-02-18

Hi Skyline team,

I am trying to practice Skyline command line with data downloaded from Webinar 15. I modified the MultiAnalysis.bat and Skyline template files to reflect programs/file path. The rest files are used as downloaded. When I try to run the cmd line, it worked a few times. Then when I tried to change mass accuracy and RT window in the Skyline template file, it stopped working. After I restarted cmd program, it worked again. Then I changed the mass accuracy and RT window and restart the cmd program to retest the program, it stopped working. I am using Windows 7 and running Skyline v20.1.0.28. I am wondering what could I do differently so that the program will work better for me?

Thanks very much for your help in advance!

Nick Shulman responded:  2020-02-18

What exactly do you mean by "it stopped working"? Did you see any sort of error message?

Can you tell us the exact command line and arguments that you were using to run your batch file?

Can you send us your Skyline template file? In Skyline you can use the menu item:
File > Share
to create a .zip file containing the Skyline document and supporting files including extracted chromatograms, spectral libraries, etc.
-- Nick
minliecn responded:  2020-02-19
Hi Nick,

Thanks for your response. When it is not working, the program does not give error messages but gets stuck after extracting 95% of the triplicate data. Please see the attached log files for when it gets stuck and when it finishes data extraction, combination and R script calculation.

The exact command line that I used are MutlAnalysis_webnar15. I have attached a screen shot from one of the run with data extraction, combination and R script calculation finished. The scripts that I used are attached in my first support request.

The Skyline template file is attached. When I was saving the files, I had to unclick the library file. I am not sure whether having the library clicked will have any impact on the process.

Thanks very much for looking into this!
Brendan MacLean responded:  2020-02-19
Hi Leah,
Why don't you try removing --import-process-count=%PROCESSES% from your SkylineAnalysis.bat command line. The --import-process-count argument should still work, but we also improved Skyline in 20.1 so that we are not using it ourselves anymore, which may mean it is less well tested.

If you are finding the Skyline hangs reliably during your processing, then we would really like to find a way of reproducing that here so that we can look at it and fix the issue.

We are seeing some infrequent hangs during command-line imports ourselves in our tests right now, which I am trying to understand and fix, but they are pretty infrequent and may not be related to what you are seeing.

minliecn responded:  2020-02-19
Hi Brendan,

Thanks for your help! After removing --import-process-count=%PROCESSES% from the SkylineAnalysis.bat command line, Skyline was able to run successfully without hangs. The only thing is the run time has significantly increased. The run time with --import-process-count=%PROCESSES% varies from 2 to 10 mins; the run time without --import-process-count=%PROCESSES% increased to 40 to 45 min.

Brendan MacLean responded:  2020-02-22
That is very strange. Does your output still look the same with multiple files importing in parallel, like this?

[2020/02/18 16:35:14]    [1] 14% [2] 14% [3] 15%

The log you sent completed the 3 file import in 11 minutes 8 seconds.

[2020/02/18 16:34:53]    Adding results...
[2020/02/18 16:34:53]    1. C:\Users\lee lab\Documents\DIAtutorial_Skyline\Webinar15\mzml-demux-24\200ng_HeLa+10%rt_DIA_middle-bandCID_24Da_overlap_nB_RA1_01_2162.mzML
[2020/02/18 16:34:53]    Adding results...
[2020/02/18 16:34:53]    2. C:\Users\lee lab\Documents\DIAtutorial_Skyline\Webinar15\mzml-demux-24\200ng_HeLa+10%rt_DIA_middle-bandCID_24Da_overlap_nB_RA1_01_2163.mzML
[2020/02/18 16:34:53]    Adding results...
[2020/02/18 16:34:53]    3. C:\Users\lee lab\Documents\DIAtutorial_Skyline\Webinar15\mzml-demux-24\200ng_HeLa+10%rt_DIA_middle-bandCID_24Da_overlap_nB_RA1_01_2164.mzML
[2020/02/18 16:34:53]   
[2020/02/18 16:35:05]    [3] 11%
[2020/02/18 16:35:07]    [1] 11% [2] 11% [3] 13%


[2020/02/18 16:45:41]    [1] 95% [2] 95% [3] 95%
[2020/02/18 16:45:47]    [1] 95% [2] 95%
[2020/02/18 16:45:50]    [1] 95%
[2020/02/18 16:46:01]   

Then it hung waiting until you ran it again, which caused the 3 separate imports to get joined and the import process to complete.

Can you post the log without --import-process-count? Skyline 20.1 should use multi-threaded parallel import based on the number of cores on your computer. Any modern 4-core or greater i7 processor should be importing the 3 files in parallel by default, unless you explicitly specify otherwise with --import-threads. If you are not seeing this, then you might consider explicitly specifying --import-threads=3, which should again force Skyline to import the 3 files in parallel, but using threads instead of separate processes.

Thanks for your help with investigating this issue. I am also working on fixing the possibility of a hang in this case.

Brendan MacLean responded:  2020-02-22
Could you also just run --version to prove what version of Skyline is getting run?
minliecn responded:  2020-03-02
Hi Brendan,

Sorry for the delay in my response as we were in the process of moving to a brand new building and we did not have access to the computers.

After I remove the --import-process-count, multiple files were imported in parallel like before but at a much slower speed. I tried to run the program twice so I have two different log files. Please find the attached log files.

Do you know what will be the exact command to figure out which version of Skyline is being run?

Maybe this will help answer whether the Skyline version could be a problem. Initially, I was running Skyline v19. Then I installed Skyline v20 without updating the Skyline runner. With a Skyline runner from v19 and Skyline v20, sometimes it runs smoothly, sometimes it hangs. After updating the Skyline runner from Skyline v20, it is about the same. So I am not sure that Skyline version caused the hang problem.

Thanks for your help!
Brendan MacLean responded:  2020-03-02
Hmmm... --version should give you the version number. So, you are saying that Skyline 20.1 is also slow when you use SkylineRunner.exe?

I wonder if you might try using the Administrative Installer:

This will put Skyline in a known location (C:\Program Files\Skyline). Then you can try using SkylineCmd.exe which has the same interface as SkylineRunner, but which may end up using different configuration parameters which we expect to make the performance between --import-process-count and using threads but not separate processes similar in performance.

Also, you might just try running the Skyline UI and importing these 3 files to see if this also takes 45 minutes or whether the import performance is more similar to what you see without --import-process-count or with it.

I performed pretty extensive tests with the current SkylineCmd.exe and got slightly better results without --import-process-count than with it. So, it is surprising that you are seeing over 3-fold worse import performance between the two modes.

Thanks for your help in understanding why you are seeing such a big difference.

minliecn responded:  2020-03-03
It took about 16 min to import the three files and 2 min to join the files running Skyline UI. I will try SkylineCmd.exe later.

Thanks for your help.
Brendan MacLean responded:  2020-03-03
Interesting. A key part of the performance is in the .config files for Skyline.exe and SkylineCmd.exe. I would expect that to apply to Skyline.exe started by SkylineRunner.exe, but perhaps we now need a .config file for SkylineRunner.exe in order to get this to work.

Your results with SkylineCmd.exe should be informative. Thanks for your help in isolating this issue.