DIA-NN .speclib support

support
DIA-NN .speclib support Tobi  2020-05-27 03:03
 

Dear Skyline Team,

could you please consider implementing support for DIA-NNs .speclib spectral libraries? Its a highly convenient tool for predicted libraries and much faster than Prosit.

https://github.com/vdemichev/DiaNN

With best regards,
tobi

 
 
Brian Pratt responded:  2020-05-28 07:41

Hi Tobi,

I had a quick read through the DiaNN documentation, I don't see any information on what that .speclib format looks like. It sounds like they can emit various other formats, though, so perhaps the problem is already solved of Skyline already deals with one of those. If you can provide an example of a .speclib file that would be helpful in assessing this.

Thanks,

Brian Pratt

 
Tobi responded:  2020-05-28 11:16

Dear Brian,

thanks for the fast reply. Please find attached a small spectral library on Pierce Retention time standard peptides (unlabeled) in .speclib and .tsv.

I know Skyline can somehow import .tsv but its not an option for large libraries due to ram usage (forced target list creation as side process). For that reason support for .speclib the same way as .blib would be awesome. DIA-NN has great potential for DIA, especially with skyline on the side for visualization.

In terms of size, speed, and adjustability DIA-NN might be preferably over Prosit in terms of predicted libraries, but it can analyse only DIA raw data.

Feel free to let me know if I can provide you with anything else.

Best,
tobi

 
Brian Pratt responded:  2020-05-28 11:51

Hi Tobi,

I was hoping that .speclib would be implemented as SQLite, but it's some other db format, so no luck there.

I think I read that they can export .msp spectrall library format, have you tried that? Skyline reads that directly.

Best,
Brian

 
Brendan MacLean responded:  2020-05-28 12:18

You might also try either building a .blib directly from the .tsv file (which we implemented in 20.1) or using File > Import > Assay Library with that .tsv file, if you just want to target everything in it. File > Import > Assay Library has received a lot of testing and benefited from many iterations of bug fixing from working with the Aebersold lab and most recently the Guo lab (former Aebersold lab member).

Unless we can get more information on the .speclib format, it seems it must be some custom binary format we are not aware of. Probably best to use one of the available interop formats.

--Brendan

 
Tobi responded:  2020-05-28 12:30

Hi Brian,

DIA-NN has experimental support for reading .msp, but exporting libraries as .speclib and .tsv seem the only option based on documentation and developers comments.

It is also said it should be technically possible to convert from .speclib or .tsv to .msp , but implementation would require some work so I do not expect this to happen soon from DIA-NNs side.

The .speclib seems roughly 12-times more space efficient than a Prosit .msp (including fragment filtering). Plus its own format for raw file conversion (.raw, .mzmL, .wiff to .dia) can reduce centroided mzMLs to half their size. Just the file formats and related processing speeds are surprisingly good and worth to take a look at.

Best,
tobi

 
Brendan MacLean responded:  2020-05-28 15:59

It looks like Matt Chambers has been silently working on this since I emailed him about handling it in BlibBuild.exe yesterday.

He just submitted a pull request on GitHub:

https://github.com/ProteoWizard/pwiz/pull/1097

So, it seems likely that there will be support in a Skyline-daily coming soon. You will need to go through the library build interface to convert from .speclib to .blib, our own SQLite based binary format, which we believe is relatively fast and space-efficient, and more approachable (because of the SQLite use) to other developers, than an entirely custom binary format (which we use for our .skyd files and .slc - library cache - files).

Keep an eye out for it in the Skyline-daily release notes. Thanks for the feedback.

--Brendan

 
Tobi responded:  2020-05-29 00:02

Hi Brendan,

thanks for the feedback, sounds great. Its always nice and sometimes a big deal to have everything at least somehow convertible.

Thank you, looking forward to it.

Best,
tobi

 
Matt Chambers responded:  2020-05-29 09:07

DiaNN is, like you said, highly convenient to run, so it make sense to support it. Especially since I was able to just copy his C++ code into BiblioSpec. However, FYI, from my admittedly-not-very-thorough testing, it seems DIA-Umpire -> pseudo-DDA spectra -> DDA search produces a better library than DiaNN. Sometime this year we will have that pipeline implemented in a way that makes it even more highly convenient to use from within Skyline. :)

 
Tobi responded:  2020-06-01 00:51

Dear Matt,

thank you very much for your helpful comment. Please don't benchmark DIANN too seriously right know as libraries and search still contain extra nonsense-precursors. This will hopefully fixed soon, further increasing its speed and sensitivity.

Best,
tobi

 
Vadim Demichev responded:  2020-06-05 10:34

Matt, thank you so much for implementing .speclib support in ProteoWizard!
Brendan, would be very cool if .speclib could be imported in Skyline, thanks! I know a number of people are using Skyline for visualising things in conjunction with DIA-NN. SQLite-based input/output is definitely planned (I use SQLite to read diaPASEF data anyway), but not in the near future.

Matt, in terms of performance, I would think something must have been wrong with the settings (deep learning not used?). In LFQbench, for example, DIA-NN's library-free now yields 100k+ precursors per run (I think it's ~3x times higher than the original DIA-Umpire + DDA engine result). Mainly due to the fact that it is peptide-centric (PECAN idea) and the use of deep learning.

Tobi, working on that :) I am thinking about feeding the peptide characteristics into neural networks, so that they automatically take into account things like the fact that charge 1 peptides with lysines are very unlikely to be detected. If that does not work, will probably add an option to just eliminate them completely, as you suggested. This would definitely be an improvement, but I would not expect any drastic effects (FDR might decrease by ~10%-20%).

Vadim

 
Tobi responded:  2020-06-08 07:37

Thank you for implementing .speclib in Skyline daily,

however, it does not seem to work, at least not via Settings / Peptide Settings / both add and built library do not accept and list .speclib files.

There is a small test file in the third post from the top.

Could you please have a quick look at that or let me know if I do sth. wrong?

Best,
tobi

 
Brian Pratt responded:  2020-06-08 08:37

Hi Tobi,

You'll need to use Matt's work on BlibBuild to convert speclib to BiblioSpec .blib format, then give that to Skyline. There are only a few formats that Skyline reads directly.

I can see where it would be good if Skyline had logic to try BlibBuild automatically when asked to read a spectral library whose format is not recognized. I'll add a feature request for that.

Best Regards,

Brian Pratt

 
Tobi responded:  2020-06-08 09:08

Dear Brian,

it seems I was mislead by the recent patch notes Skyline-daily 20.1.1.155 "Spectral library building for the DiaNN specLib format."

Thank you for adding a request for direct support. I would assume the current way is not really accessible for non-coders? Direct support would be great, especially since both skyline and DIA-NN are software with GUIs specifically for uncomplicated and easy access.

Thank you and with best wishes,
tobi

 
Brian Pratt responded:  2020-06-08 09:31

You should be able to proceed with the tools you have., it's not very complicated.

https://skyline.ms/wiki/home/software/BiblioSpec/page.view?name=BlibBuild

 
Matt Chambers responded:  2020-06-08 09:46

Normally, instead of adding it as a library, you'd import it as a "Peptide Search" (File -> Import -> Peptide Search). But because I forgot to add ".speclib" to the valid file extensions for library search result files within Skyline itself, at the moment you'd have to convert from speclib to blib by running BlibBuild from the command-line.

Brian meant that if opening a library directly failed because it's not a supported format, it would fallback to trying to import it with BlibBuild (as a "peptide search").

Brian/Brendan, shouldn't setting the BuildLibraryDlg to "All Files" disable the IsValidInputFile() validation of the file extensions? The "All Files" option seems completely superfluous otherwise.

 
Brendan MacLean responded:  2020-06-08 12:25

We don't really want to just call BlibBuild with whatever file the user wants to supply, but All Files allows the user to see the files in a folder, and maybe understand better why something they expect to see isn't showing up.

Probably, Matt, you should add a big comment in BlibBuild that reminds a developer to add a newly support file format to Skyline, or figure out a way to either test this is the case or build the list Skyline uses during the Bibliospec build... or something.

For the time being we don't add new formats that often, and we remain vulnerable to the mistake you made. It will be fixed in the next Skyline-daily, which can be relatively soon.

 
Tobi responded:  2020-06-09 09:09

Dear all,

thank you very much for the extensive effort. Looking forward to the direct support of .speclib since Skyline and DIA-NN are a really sweet combination and will be used as such by lots of people.

With best wishes,
tobi

 
f capuano responded:  2021-05-18 07:51

Hi All,

Very useful thread.

As a follow up to this post I have a question about the library generated by DIA-NN 1.7.15 and its use in combination to Skyline.

I want to use a library generated using DIA-NN to confirm the IDs of my target peptides. In order to do that I have searched 4 data files using DIA-NN and imported the library tsv file on Skyline as an assay library. Then, I have imported as results the same files used for generating the library to extract target peptide peaks.

I observe that the RT for my target peptides on the library differ from the RT of the peaks selected by Skyline. By comparing the transitions ranking and ppm the peaks selected by Skyline seem to be correct. How do I reconcile the difference in RT? Is it possible I am using a library that is a predicted one and not experimental?

Unfortunately I have no HL that I can use to pin down RT for my target peptides.

Any advice highly appreciated.

Floriana

 
Vadim Demichev responded:  2021-05-18 08:03

Hi Floriana,

DIA-NN saves 'predicted iRT' retention times in the library. That is not the experimental times, but experimental times translated to some reference scale. To save experimental times, please use the '--out-measured-rt' option. Another workaround is to map library entries to the entries in DIA-NN's main report, and get RT values from there. Another option is to look at peptide spectra using the --vis command in DIA-NN (please see the readme file), which basically exports chromatograms for the selected peptides and their fragments.

Best,
Vadim

 
Brendan MacLean responded:  2021-05-18 10:06

Hi Vadim,
Great to have the lead author of the DIA-NN paper on this thread. One thing we have done with other tools like EncyclopeDIA and OpenSWATH is to use the actual integration boundaries they came up with and not just the detection RT (presumably the apex or central retention time of the integrated peak). If there were a way to get these from DIA-NN, we could do the same for it. Skyline now has a checkbox in its library setup form for whether integration boundaries in the library should be applied to the data in Skyline.

It is obviously extremely helpful to have the measured RT at the center or apex of the peak, but having the integrated range can give a researcher an even clearer idea of how the tool came up with the peak areas it did.

Thanks for considering the ideal integration between DIA-NN and Skyline for visualization of your results.

--Brendan

 
Vadim Demichev responded:  2021-05-18 10:15

Hi Brendan,

Yes, in recent versions there are RT.Start and RT.Stop columns in the main report which correspond to the boundaries DIA-NN uses for quant.

Best,
Vadim

 
Brendan MacLean responded:  2021-05-18 10:35

Great news. I will make sure Matt starts storing those in the appropriate location in our BLIB format libraries so that they can be used by Skyline.

Thanks for posting to this support board.

--Brendan

 
Vadim Demichev responded:  2021-05-18 10:42

Oh, sorry, I misunderstood, these are only saved in the main report, not in the spectral library. Can also save in the library? I guess it might be a good idea anyway.

Vadim

 
Brendan MacLean responded:  2021-05-18 10:55

Well, let us know when and where to find them, and we will make sure DIA-NN users seeking Skyline visualization support benefit from their availability.

Thanks again.

 
Matt Chambers responded:  2021-05-18 11:34

Yes I was just about to ask about it being in the library. We do currently read the .speclib file. Is the main report actually suitable to use as a replacement? Is that different than the TSV version of the speclib file?

 
Vadim Demichev responded:  2021-05-18 15:38

Many thanks Brendan! I will most likely implement saving the RT boundaries in the next version.

 
Vadim Demichev responded:  2021-05-18 15:40

Hi Matt, no, the main report does not contain spectra. But it can be used to supplement the information stored in the .tsv library: just need to match the pairs [file name - precursor name] between the report and the library (easy to do in R).

 
f capuano responded:  2021-05-20 06:29

Hi Vadim,

Thank you for suggesting the experimental RT export it works as a temporary fix. Exciting to see the ID mark aligning with the peak selected by Skyline :)

Thanks to all of you for looking into this. It will be very useful to have all the library info available in a file to upload directly in Skyline.

 
Juan C. Rojas E. responded:  2021-09-24 08:44

Hi all,

I would like to revive this thread with the following issue: I was trying to create a spectral library within Skyline (Settings -> Peptide Settings -> Library -> Build...) with the .speclib output of DIANN and I got an error about the .speclib version that is supported (image attached).

From the support threads my understanding is that creating a spectral library in .blib Skyline format from a DIANN .speclib format should be possible now? I can try to find a workaround with other outputs of DIANN, but I was wondering if this is a version incompatibility issue or am I doing something wrong?

I am using Skyline-daily 21.1.1.223 and DIANN 1.8

Sincerely,
JC

 
Brian Pratt responded:  2021-09-24 09:24

That sounds like DIANN has changed something in their .speclib format, and BiblioSpec is being appropriately cautious about it. Can you provide an example file?

Best regards,
Brian Pratt

 
Juan C. Rojas E. responded:  2021-09-24 09:32

Sure thing. I just uploaded it to the File Sharing dropbox with the name of "ubi_ptm_fig4_spectral_library.speclib".

Would you need anything else or is that enough?

JC

 
Brian Pratt responded:  2021-09-24 09:43

That's probably sufficient - we'll look into this, thanks.

  • Brian
 
Matt Chambers responded:  2021-09-24 09:56

Please share the speclib.tsv as well. From what I can tell (https://github.com/vdemichev/DiaNN/commit/f2c8a78f46520b763c99d8be6cb98504726310f6#commitcomment-55711150) DiaNN has gone closed-source since I added support for reading its proprietary speclib binary format. Since I just copied from DiaNN's C++ source code to read that format, this may be a problem for maintaining that reader with a reasonable amount of effort. IIRC I chose the speclib format because it had spectral peaks and the tsv did not, but I'm not sure that's still the case. If it is, then I may need support from Vadim to tell me which fields are new/changed in the new speclib version(s) so I don't have to invest a lot of time reverse engineering it myself.

 
Juan C. Rojas E. responded:  2021-09-24 10:49

Hi Matt,

I just uploaded the corresponding "ubi_ptm_fig4_spectral_library.tsv". That is what I am currently using to create a parser to import and assay library.

Thanks a lot for looking into this.
JC

 
a schmidt responded:  2021-12-07 07:38

Dear All,

at the moment I am also facing the same problem. Skyline reports a speclib version error (speclib 3.0 from DIA-NN vs supported 2.0) when trying to built a library from the from the DIA-NN .tsv speclib file. Has there been any update to the issue?
Also, I have been trying to convert the .speclib into .blib with the blibbuild tool, but could not get it running. I might need a bit more information how to set it up correctly.

best regards
Andreas

 
Matt Chambers responded:  2021-12-07 14:54

Hi Andreas,

Skyline uses BlibBuild to create spectral libraries, so if Skyline doesn't work it's very unlikely BlibBuild will (in this case, speclib v3 definitely won't work yet with either).

I'll have a fix in today or tomorrow that should get released in the next version 21.2.

 
a schmidt responded:  2021-12-08 02:26

Dear Matt,

thank you for the update, I am looking forward to test it.

best wishes
Andreas

 
Matt Chambers responded:  2021-12-15 07:24

In the next release and current daily this should be fixed. However, you will need the report.tsv file from DIA-NN as well as the speclib (not the lib.tsv). The speclib and report.tsv must be in the same directory.

 
gshlayen responded:  2024-08-02 08:16

Hello,

I know this thread is old but I am trying to upload an entire .speclib file into skyline for visualization. I am not uploading any raw data at all, just want to visualize the whole .speclib. Is this possible?

thanks

 
gshlayen responded:  2024-08-02 08:23

Also, I can't seem to use blibbuild to convert the .speclib file into a readable format for skyline. Does anyone have a step by step tutorial on how to do this? I created a .tsv file from the .speclib but the size seems like it is too large for skyline to accept.

Thanks

 
Matt Chambers responded:  2024-08-02 08:28

Yes, you can build a library from speclib in Skyline:
Settings -> Peptide Settings -> Library -> Build

The build happens in the background, so it might initially appear empty and you'll see a popup at the bottom left when it finishes. Then you can click "Explore" in that settings page or go to View -> Spectral Libraries.

 
Nick Shulman responded:  2024-08-02 08:28
gshlayen,

The way to get a .speclib file into Skyline is to build a spectral library from it.
One way to do this is by using the "Build" button on the "Libraries" tab at "Settings > Peptide Settings".
After you have built the library you can use the "View > Spectral Libraries" to look at the entries in that spectral library.

When you build a library in Skyline, you are asked for the peptide search results files to be included in the library.
The list of file types that you can use to build a library is here:
https://skyline.ms/wiki/home/software/BiblioSpec/page.view?name=BlibBuild

For DIA-NN results, you are supposed to point Skyline at the .speclib file.
Skyline will then look at all of the .tsv files in the same folder as the .speclib and try to find the report tsv file based on the column names in the first line of the tsv file.
-- Nick
 
gshlayen responded:  2024-08-02 08:42
Hi,

Thank you for the replies. I have succesfully been able to upload raw data into skyline before from DIA-NN, but I'm trying to upload the entire .speclib into skyline with no raw data, so there is no corresponding report.tsv file. I tried building the library, but skyline says that it doesn't accept .speclib format. I also used DIA-NN to convert the .speclib into a tsv and uploading it that way to build the library but it didn't work either. I am not doing an assay library import with that same .tsv file and it seems to be working so maybe that is the solution I was looking for. Strange though that skyline doesn't accept the .speclib files without the corresponding .tsv, and that theres no easy way to convert it to a .blib.
 
Matt Chambers responded:  2024-08-02 09:06
Is that just the pure predicted library then? I think the latest DIANN version for that uses a new format (v8?) that BlibBuild can't read. I guess the TSV you created with DIANN that imports as an assay library is the TSV representation of the library. The TSV file that BlibBuild imports along with the speclib is the "report" TSV, not the "library" TSV. It has information about what experimental spectra were used to build the library that the speclib does not have. Which I guess makes sense in your case if the speclib is purely predicted spectra. I think I had previously suggested to Brendan that we allow building blib files from just speclib files if no report TSV can be found, but he didn't like the idea because it would mean the user could get an incomplete library (speclib only) when they expected a full one (speclib+TSV), just because files were named wrong or in the wrong place. He wanted users to just get an error in that case.

Technically if the user wants a "full library" Skyline should have them select the TSV report and Skyline should use that and get peak info from the speclib. That's more or less impossible to do with any certainty though because the DIANN TSV format does not specify what library was used. Do you have a manifest for your DIANN run that created the speclib? A text file that contains lines like:
{"Library in .speclib format for Skyline":"C:\Out\mann_evosep-lib.parquet.skyline.speclib"}
{"Main .parquet report":"C:\Out\mann_evosep.parquet"}
{"Main .tsv report":"C:\Out\mann_evosep.tsv"}
 
gshlayen responded:  2024-08-02 11:01
Yes exactly it is a pure predicted library. The tsv is a representation of the library(it's a massive 7GB file of the whole human proteome). I'm not sure if I have a manifest, but I have the log file from the DIA-NN run, I have attached it below. I also received an error when trying to upload the assay library from the .tsv file of the spectral library. The error message states "index was outside the bounds of the array". Maybe what I am trying to do just ins't possible on skyline, will keep trying to troubleshoot.
 
Matt Chambers responded:  2024-08-02 11:27
Yes, for now there is no way to import a pure predicted speclib into Skyline.