The implementation of TransitionGroup.GetTransitions() appears not to have been updated to support the advances in small molecule spectral library support. Specifically, this section of code:

            // For small molecules we can't generate new nodes, so just mz filter those we have
            foreach (var nodeTran in groupDocNode.Transitions.Where(tran => tran.Transition.IsNonPrecursorNonReporterCustomIon()))
            {
                if (minMz <= nodeTran.Mz && nodeTran.Mz <= maxMz)
                    yield return nodeTran;
            }

            if (!sequence.IsProteomic) // Completely custom CONSIDER(bspratt) can this be further extended for small mol libs?
                yield break;

Should be updated to pay attention to a spectral library, if present, and the Transition Settings - Library tab - Pick [ ] product ions - and I guess [ ] minimum product ions.

A good test case is provided by the new File > Import > Assay Library test for small molecules. Once you have the assay library imported:
1) Refine > Advanced > Auto-select all [x] Transitions, OK
Note: This alone should reduce the number of transitions, because the default Pick product ions count is 3 and many of the precursors have 4 library transitions.
2) Change Pick product ions to 2 and then 4 and verify that the transition counts in the precursors change to match the settings.
3) Change Pick minimum product ions to 4 and 5 and verify that the precursor counts change to match these expected values.
Note: You can also use the Document Grid to show transition counts for the precursors and verify that these numbers change as expected for the cases above.