resolve as By Design
|Assigned To||Guest||»||Brendan MacLean|
That is a pretty messed up model. With a mProphet model like that I am not surprised that the default model does better. Your model has perfect separation of targets and decoys, but that clearly (especially from your example) is just an overtraining illusion. Since the default model has no ability to overtrain, as it is fixed, it does better.
You should not use mProphet with a model like this, and Skyline gives you all the tools you should need to see that the model has gone awry.
First step is to go back to the Advanced Peak Picking tutorial and understand how to figure out why certain scores are disabled. You really want to find a way to get the Library and Retention Time scores enabled. They are 2 of the most important scores, and I wouldn't recommend using a model without them.
If all you need is q values on your peaks, you could also try switching from "mProphet" to the "Default" score and clicking the "Train" button for that to calibrate the default score to your decoys and so achieve a model that can assign "Detection Q Value" and "Detection Z Score" to your peaks without actually altering the relative contributions of the scores in the default score (which helps to avoid overtraining). We are planning on making this more automatic in Skyline in the future so that the default score assigns q values and z scores in any document containing decoys.
So, up to you. You can continue working on your mProphet model until it seems more likely to be valid. I doubt a reviewer who understood mProphet models would let this one get by in a manuscript. Or you can try using Default score trained to give you q values.