“Compare Peak Picking” option

“Compare Peak Picking” option peter r mosen  2021-11-15

Hello Brendan and Team,

my post is linked to a previous issue of mine (Comparing Peak Scoring, 2021-06-17). I am still struggling with the “Compare Peak Picking” option. I acquired my MRM-data (400 peptides light and corresponding 400 heavy standards), built customized peak scoring model(s) using the 2nd best peak option and applied the models to my data. Peak scoring models (p- and q-value distribution) look good and q-value based refusal of peptides corresponds well with results from manual inspection. For comparison of the built models I want to use the ROC plots. And here my doubts start:

a) If I load my peak scoring model specific to the dataset (here CIL, which are isolated organelles), no ROC curve is visualized at first (slide1)
b) as soon as I load other models (which are actually not specific to the sample loaded in Skyline, here cell line) curves which look like ROC-curve are visualized and also for the CIL peak scoring model a trace is visualized (faint blue line directly on the y-axis)- at slide2. But these curves don’t really look like the ROC curves I know. I see that the shape of the curves look distorted (I see a retrograde trend for the curves (allCellLines and MEF)) as well as the vertical traces for the CIL.
c) Furthermore the ROC curves never reach 0.01 FP, which is the 1% FDR cutoff, right ? From my data I know that there are peptides with q-values higher than 0.01, so I would expect that the curve goes beyond this 0.01 FP point (slide 3)

To doublecheck I loaded DIA sample data from webinar 15 (slide4) and re-used the model you built. Also here the ROC curves for your model have this unusual shape. Maybe you guys can comment on this. What am I doing wrong? I didn’t find any documentation/tutorial on this.
Thank you for suggestions and your help,

FYI: I am using Skyline alpha (64-bit)

Nick Shulman responded:  2021-11-15
I do not know the answer to your question but if you send us your Skyline document we can probably figure out what is going on.

In Skyline you can use the menu item:
File > Share
to create a .zip file containing your Skyline document and supporting files including extracted chromatograms.
If that .zip file is less than 50MB you can attach it to this support request. Otherwise, you can upload it here:

The Skyline document will only contain the one peak picking model that it is currently using. If you would like us to see your other peak picking model, you should go to:
Refine > Reintegrate
and choose another model, and Share the document again and send us other .zip file too.
-- Nick
Nick Shulman responded:  2021-11-16

Thank you for sending me your Skyline documents in email.
It sounds like you are asking about why your ROC plots look jagged like that.
I believe I have seen ROC plots do that before, but I do not know why they might do that.
I hope someone else on the Skyline support board can explain why this can happen.
-- Nick