The objective of this study was to identify quantitative trait loci (QTL) affecting fitness of hybrids between wild soybean (Glycine soja) and cultivated soybean (Glycine max). Seed dormancy and seed number, both of which are important for fitness, were evaluated by testing artificial hybrids of G. soja × G. max in a multiple-site field trial. Generally, the fitness of the F1 hybrids and hybrid derivatives from self-pollination was lower than that of G. soja due to loss of seed dormancy, whereas the fitness of hybrid derivatives with higher proportions of G. soja genetic background was comparable with that of G. soja. These differences were genetically dissected into QTL for each population. Three QTLs for seed dormancy and one QTL for total seed number were detected in the F2 progenies of two diverse cross combinations. At those four QTLs, the G. max alleles reduced seed number and severely reduced seed survival during the winter, suggesting that major genes acquired during soybean adaptation to cultivation have a selective disadvantage in natural habitats. In progenies with a higher proportion of G. soja genetic background, the genetic effects of the G. max alleles were not expressed as phenotypes because the G. soja alleles were dominant over the G. max alleles. Considering the highly inbreeding nature of these species, most hybrid derivatives would disappear quickly in early self-pollinating generations in natural habitats because of the low fitness of plants carrying G. max alleles.