4. Study 2: Semantic Feature Production Norms
Although the WordSimilarity-353 data set has been widely used, it does have one notable weakness
as a measure of human semantic behavior, namely its size. Given its limited size, it is possible that the
results from Study 1 might not generalize. In order to assess the ability of the W3C3 model to generalize
to new data sets, we applied the constituent models and W3C3 model from Study 1 to a large set of
semantic feature production norms[1]. For the sake of exposition we will refer to these norms using
the first initial of the last name of the authors presenting the norms, MCSM.
The MCSM norms consist of 541 nouns, or concepts. Participants were asked to list features for a
concept such as the physical, functional, or encyclopedic aspects of the concept. The generated features
were regularized, e.g., usually has a tail and tail would be coded as has tail. After regularization,
2,526 features remained. Thus the data can be represented as a 541 × 2,526 matrix whose cell values
υij are the number of production occurrences for a feature j given a concept i. From this matrix, a
541 × 541 matrix was created in which the value at each cell is the cosine of two 2,526 dimension
row vectors associated with their respective concepts. The 541 × 541 matrix has 95,437 non-zero pairs
representing similarities between concepts. Although the collection methodology for the feature norms
is an associative production task, this 541 × 541 matrix represents the feature overlap between concepts,
which is more comparative in nature.
Table 8 presents the Pearson correlations between the similarities from the 541 × 541 similarity
matrix and both the predictions from the constituent models and the W3C3 model. The overall pattern
of correlations in Table 8 has a striking similarity to those of Table 3. First, the correlations of ESA and
WLM are almost identical in both cases. Secondly, the correlation for COALS is significantly greater
than that of both ESA and WLM. And finally, the correlation of the W3C3 model is significantly greater
than all of the constituent models. That the pattern of correlations is the same in both cases, especially
when the MCSM set is so large, suggests that the properties of the models observed in Study 1 are
generalizing in a systematic way.
Table 8. Pearson correlations with MCSM (N = 95,437).
| Model | Correlation |
| W3C3 | 0.67 |
| COALS | 0.63 |
| ESA | 0.52 |
| WLM | 0.53 |
In order the assess the relative contributions of each constituent model to MCSM performance, a
linear regression was conducted. The regression used COALS, ESA, and WLM raw scores to predict
the MCSM similarity scores. The results of the linear regression are presented in Table 9. Tolerance
analyses were conducted to test for multicollinearity of COALS, ESA, and WLM by regressing each on
the other two. The obtained tolerances, all between 0.55 and 0.68, suggest that the three models are not
collinear. In contrast to the WordSimilarity-353 regression presented in Table 6, the constituent models
- ↑ McRae, K.; Cree, G.S.; Seidenberg, M.S.; McNorgan, C. Semantic feature production norms for a large set of living and nonliving things. Behav. Res. Methods 2005, 37, 547–559; PMID: 16629288.
