Strange Fst distribution in BayeScan

[sacketlc]

I have used BayeScan (2.1) successfully several times on various subsets of a large SNP dataset that I am currently working on. Recently, I tried to use it to infer outliers between several population pairs using my whole dataset (57,000 SNPs), and came up with a peculiar result in all 9 population pairs. The plot of posterior density for Fst looks as expected, except with a mean Fst ~ 0.4 - 0.6, depending on the population pair (not ~0.05 as I know them to be). If I plot a frequency distribution of Fst values inferred by BayeScan, the distribution is the opposite of what I would expect -- the highest frequency is Fst at these high values around 0.5, with a tail on the left for low Fst values. These are bird populations that we know are differentiated but still experience some gene flow, and are geographically quite close. Calculating Fst in vcftools or Genepop gives a mean and distribution more in line with expectations -- mean ~0.05 with a tail to the right of higher Fst values.

Has anyone else experienced something similar? I have been picking my brain to try to figure out what is causing this to happen, but don't have any hypotheses yet. The current dataset has less missing data than subsets I have previously used, so I wouldn't expect that to be an explanation, and I used the same approach as previously. I used PGDSpider to convert from vcf format to BayeScan format.

I would appreciate any thoughts!

[GenoMax]

For reference

cross-posted on Biostars: https://www.biostars.org/p/160738/