This study aims to understand why some children from poor households do not experience stunting, with a particular focus on the geographical and gender dimensions of child nutrition. The research was conducted in two villages on Timor Island—Lakamola (inland) and Nusakdale (coastal)—representing different geographic contexts within persistent poverty. A qualitative approach was employed through in-depth interviews, participatory observation, and focus group discussions with mothers, health workers, and community leaders. Findings reveal that the determinants of stunting and non-stunting among poor households go beyond food practices—such as the quality, diversity, and quantity of food provided to infants and young children—to include the caregiver’s emotional responsiveness and affection during feeding episodes. Several challenges were identified, including difficult topographical conditions, heavy maternal workloads, limited financial support from partners, and seasonal fluctuations in food availability. Despite these, families demonstrated adaptive strategies by leveraging family and social support, including health facility staff, informal labor income, women’s savings groups, and home gardens. This study highlights the importance of incorporating both geographic and gendered contexts into child nutrition interventions. Community-based approaches and the empowerment of women are essential to enhancing the resilience of poor households in mitigating the risk of stunting in a sustainable and culturally grounded manner.