When you choose a "Max transition peak rank" at "Refine > Advanced > Results", the behavior depends on what you have chosen for "Replicate inclusion" ("All", or "Best") at the bottom of the form.
I believe that if you choose "All" then the rankings are based on the average transition peak areas across all the replicates, and if you choose "Best" then they are based on the replicate with the highest total peak area for all of the transitions belonging to the same peptide or molecule.

These rankings, therefore, are different than the ones that you see in the Targets tree, which are based on whatever is the currently selected replicate.

Do you think that's what's going on?
If not, then you should send us your Skyline document and we'll figure out what is going on.
In Skyline you can use the menu item:
File > Share
to create a .zip file containing your Skyline document and supporting files including extracted chromatograms.
If that .zip file is less than 50MB you can attach it to this support request. You can upload larger files here:
https://skyline.ms/files.url
-- Nick

Thanks, I tried using the All or Best but still see different ranks. But i realized that there are actually two numbers next to each transition- (rank #) [#]. Which number relates to the rank based on the spectral library? My aim is to filter for the top 3 ranked transitions based on the the spectral library entry. I attached pic of example

If you want Skyline to choose the best transitions in the spectral library, you should go to:
Settings > Transition Settings > Library
and change the number in the textbox in the middle of the "Pick <textbox> product ions".

The "Results" tab of the "Refine > Advanced" dialog is always going to be looking at your extracted chromatograms to decide which transitions to keep, not the spectral library.

In the text that you see in the Targets tree, the "rank X" part comes from the spectral library, and the number in square brackets comes from comparing transition peak areas in your extracted chromatograms.
-- Nick

Thanks, this is very helpful to understand the order of these transitions

If I want to only work with the top 3 most abundant transitions in the spectra, it would be correct to look at the number in [ ] as opposed to using the library rank #, is this correct? So when I set the Results filter to 3 for 'Max transition peak rank', I am limiting it to the top 3 most abundant transitions? This is actually what I would like to do and may have misunderstood using the library rank.

I'm not sure I understand what you mean by "in the spectra".
Do you mean the spectra that the spectral library was created from, or the spectra that the chromatograms were extracted from?

If you have questions about the Library tab of the "Transition Settings" dialog, you might find the answers in the Targeted Method Editing tutorial:
https://skyline.ms/wiki/home/software/Skyline/page.view?name=tutorial_method_edit

If you have questions about the "Refine > Advanced" dialog you might find the answers in the Targeted Method Refinement tutorial:
https://skyline.ms/wiki/home/software/Skyline/page.view?name=tutorial_method_refine

-- Nick