Hi Matthias,
Thanks for the screenshots!
Both in Webinar 14 and the course DIA materials, the library is what is known as a "sample specific" library created without fractionation (as outlined in Bruderer, MCP 2015), but just running DDA on the same sample prep as the DIA. This technique then simply asks DIA and mProphet to find the same peptides that were found with DDA under the same sample prep conditions.
The library you are using, however, contains targets for 8,660 proteins with over 1 million fragment ion transitions. This is a deep proteome-wide library (possibly Rosenberger, Sci. Data 2014 - https://www.ncbi.nlm.nih.gov/pubmed/25977788 - PXD000954) likely created with fractionation, and possibly including sample preparation different from your own samples (as outlined in Selevsek, MCP 2015). You should expect a very different mProphet model in this case, because it is far less likely that you can detect all of the targets in DIA under your sample prep.
In fact, the large distribution of targets that form a shape very similar to your decoys should give you confidence that the model is working as expected, because this is what is expected of libraries like this one. You will not be able to detect a lot of the peptides in this kind of library, and those failures are expected to look a lot like your decoys.
If you used the Pan Human library (PXD000954), then the targets should already be limited to 6 transitions per precursor.
So, what was the source of your library? What type of sample are you searching against (e.g. HeLa cell lysate, plasma, or ...) And what instrument did you use with what isolation scheme? How many proteins and peptides are detected a q value < 0.01?
You should not expect this model to ever look like a model from a "sample specific" library, but there may still be things you could do to improve.
--Brendan