How numerical quantity is processed is a central issue for cognition. On the one hand the "number sense theory" claims that numerosity is perceived directly, and may represent an early precursor for acquisition of mathematical skills. On the other, the "theory of magnitude" notes that numerosity correlates with many continuous properties such as size and density, and may therefore not exist as an independent feature, but be part of a more general system of magnitude. In this study we examined interactions in sensitivity between numerosity and size perception. In a group of children, we measured psychophysically two sensory parameters: perceptual adaptation and discrimination thresholds for both size and numerosity. Neither discrimination thresholds nor adaptation strength for numerosity and size correlated across participants. This clear lack of correlation (confirmed by Bayesian analyses) suggests that numerosity and size interference effects are unlikely to reflect a shared sensory representation. We suggest these small interference effects may rather result from top-down phenomena occurring at late decisional levels rather than a primary "sense of magnitude".