Slow importing of raw files

support
Slow importing of raw files Josh Baeza  2018-06-18 06:26
 

Dear Skyline team,

I am trying to import 4 thermo raw files into skyline. The skyline document has about 7000 proteins and the raw files being imported were collected using an overlapping window (24 m/z total, 12m/z overlapping). I began importing the data on Friday (6/15/18) and I could immediately see the chromatograms as they were being imported. After about 5 minutes, the full chromatogram for 1 raw file was displayed and the import had reached 40%. I returned on monday (6/18/18) and the progress for the same raw file has only reached 41%.

I am using Sklyine Daily (64 bit) 4.1.1.18151 on a windows server 2016 Datacenter v1607. The processor is an intel(R) Xeon(R) Silver 4114 CPU @ 2.2GHz 32 processors and 110 Gb of Ram. I also copied the raw files to the virtual desktop on the server.

I read a previous post that mentioned the issue stemming from unmapped network drives. All of the networks mapped to the server seem to be in working order. can you provide some guidance on how to fix this issue?

Many thanks,

Josh

 
 
Nick Shulman responded:  2018-06-19 10:44

That might be a bug in Skyline. It looks like Skyline is getting stuck right at the point where Skyline has finished importing your iRT peptides, and is trying to figure out what slope and intercept to use for mapping from iRT to retention time.

Is this a problem with all of your .raw files, or just that first one? Have you tried this on other computers and does it work?

Can you send us your files?

In Skyline, you can use the menu item:
File > Share > (Complete)
to create a .zip file containing your Skyline document and supporting files, including iRT database.

You can upload that .zip file here:
https://skyline.ms/files.url

You should also send us at least one of the raw files that you are trying to extract chromatograms from.

 
Brendan MacLean responded:  2018-06-19 11:16

Hi Josh,
Can you right click on CPU graphs and choose Change graphs to > Logical processors?

Not that it explains such a long pause at this point, but there will be some kind of pause after 40% while, as Nick explains, Skyline recalculates all of the extraction parameters for the nearly 1 million transitions in your document. The first 40% is extracting full-gradient chromatograms for your iRT standards. Though, if it is showing 41%, I suspect it is already past this calculation, which the progress UI also indicates with the gray vertical bar at 5 minutes. Though, I have myself recently been importing files with over 2 million transitions on a desktop i7 system with 16 GB of RAM. So, there is definitely something strange here.

Can you provide a screenshot of your Transition Settings - Full-Scan tab?

Sending us the files as Nick suggests would also help tremendously.

--Brendan

 
Josh Baeza responded:  2018-06-19 13:38

Hi Nick and Brendan,

I uploaded my skyline document along with a raw file to test. Thanks for the support.
I have tried to import data into skyline on my laptop (Dell XPS 15, same Skyline daily version) and it was also very slow. I don't think it even reached the 40% mark after 8 hours.
I went ahead and attached the screen shots of the CPU processors and there doesn't seem to be anything abnormal.

Thanks again,
Josh

 
Nick Shulman responded:  2018-06-19 14:07
Thanks for sending us your files.
The problem is that you have Overlap Deconvolution turned on.
Overlap deconvolution is an algorithm for dealing with overlapping DIA isolation windows. We have been meaning to remove this feature from Skyline, because the recommended way of dealing with such data is to use MSConvert to create .mzML file that has been deconvoluted.

The way that we implemented in Skyline really cannot handle a large number of peptides.

Anyway, you probably never intended to turn this on.

You should go to:
Settings > Transition Settings > Full Scan > MS / MS filtering Isolation Scheme > Edit Current
and then change "Deconvolution" to "None"
 
Josh Baeza responded:  2018-06-20 05:27
Thanks Nick.
I acquired my data using overlapping windows and I thought the raw files would be demultiplexed in Skyline. This was my approach because I couldn't get MSConvert to demultiplex my files. I actually met with you at ASMS and I mentioned the problems I was having with MSConvert. Well I tried to run MSConvert again today on a different system and it also fails. I've attached the screen shot and the failed mzML file.

Sorry to change the topic of this conversation. Let me know if you would prefer I begin a different thread instead.

Thanks,

Josh
 
Nick Shulman responded:  2018-06-20 14:29
I will try to figure out what is going wrong with MSconvert.

The error I am seeing is:
Failed - System.Exception: SpectrumToIndices() Number of demultiplexing windows changed. Window boundary tolerance may be set too low.
   at pwiz.CLI.msdata.MSDataFile.write(MSData msd, String filename, WriteConfig config, IterationListenerRegistry iterationListenerRegistry)
   at MSConvertGUI.MainLogic.processFile(String filename, Config config, ReaderList readers)
   at MSConvertGUI.MainLogic.Go(Config config)

I will ask around and see if I can figure out who can fix this.
 
atkeller responded:  2018-06-22 18:23
Hi Josh,

It looks like within each DIA cycle, the window centers are offset by 23 m/z instead of 24 m/z. This is causing msconvert to fail during demultplexing because the windows won't align to a nice repeating grid. Unfortunately, we can't demultiplex the data but you can still search it without any demultiplexing by setting "Deconvolution" to "None", as Brendan described above. You won't get the improved selectivity benefits of demultiplexing, but hopefully this will still let you salvage the data.

There's actually a feature in Skyline that will generate an optimized set of DIA windows as a table of Start m/z and End m/z values with the option to apply overlapping. This is meant to help guarantee that a chosen windowing scheme can be demultiplexed.

In a blank Skyline document, go to
Settings > Transition Settings > Full Scan > MS / MS filtering
and then set "Acquisition method" to "DIA".
Next, within "MS / MS filtering" go to Isolation scheme > Add... > Prespecified isolation windows > Calculate...
This will open a "Calculate Isolation Scheme" dialog where you just need to specify "Start m/z", "End m/z", set "Deconvolution" to "Overlap", and check "Optimize window placement", then select "OK" in order to generate a valid overlapping scheme. The resulting table can be copy-pasted from Skyline in order to transfer it to your method editor.

Let me know if this is helpful, or if you were trying to do something different. Did you know about the feature for generating overlapping windows within Skyline? Maybe this is something we could make easier for users to find.

Thanks,
Austin