One thing that you could do is install Skyline version 3.7, and build your library using that older version of Skyline.
Skyline version 3.7 and earlier used only one decimal place to describe modifications in BiblioSpec library.
You can install older versions of Skyline from the "Unplugged" installer page here (click the "Archive" link):
https://skyline.ms/wiki/home/software/Skyline/page.view?name=install-64-disconnected_4_2
Also, if you know how to run BlibBuild from the command line yourself, you can use a current version of BlibBuild, and omit the "-H" command-line argument that tells BlibBuild to use more than one decimal.
You can tell Skyline that you are interested in average masses instead of monoisotopic masses on the settings page:
Setting > Transition Settings > Prediction
I actually do not know whether that has any effect on the way that Skyline figures out what modification corresponds to the masses in a spectral library (I suspect that this setting will not fix your problem).
You could try sending us your peptide search results, and we might be able to figure out something else that would work.
If your files are less than 50MB, you can attach them to this support request.
Otherwise, you can upload them here:
https://skyline.ms/files.url
-- Nick