Caldwell is located northwest of the center of Burleson County at 30°31′43″N 96°42′1″W (30.528580, -96.700350). Texas State Highway 21 passes through the center of the city, leading northeast 23 miles (37 km) to Bryan and southwest 50 miles (80 km) to Bastrop. Texas State Highway 36 bypasses the center of the city to the northeast, leading northwest 16 miles (26 km) to Milano and southeast 32 miles (51 km) to Brenham. According to the United States Census Bureau, Caldwell has a total area of 3.9 square miles (10.2 km2), of which 0.03 square miles (0.07 km2), or 0.66%, is water.
As of the census of 2000, there were 3,449 people, 1,322 households, and 938 families residing in the city. The population density was 1,021.4 people per square mile (394.0/km²). There were 1,485 housing units at an average density of 439.8 per square mile (169.6/km²). The racial makeup of the city was 71.24% White, 12.64% African American, 0.17% Native American, 0.09% Asian, 13.71% from other races, and 2.15% from two or more races. Hispanic or Latino of any race were 22.96% of the population. There were 1,322 households out of which 36.5% had children under the age of 18 living with them, 52.5% were married couples living together, 14.1% had a female householder with no husband present, and 29.0% were non-families. 26.3% of all households were made up of individuals and 15.8% had someone living alone who was 65 years of age or older. The average household size was 2.61 and the average family size was 3.16.
In the city, the population was spread out with 28.8% under the age of 18, 9.5% from 18 to 24, 26.7% from 25 to 44, 20.5% from 45 to 64, and 14.5% who were 65 years of age or older. The median age was 34 years. For every 100 females, there were 87.0 males. For every 100 females age 18 and over, there were 83.2 males. The median income for a household in the city was $29,936, and the median income for a family was $37,658. Males had a median income of $25,745 versus $20,306 for females. The per capita income for the city was $14,141. About 14.2% of families and 17.9% of the population were below the poverty line, including 21.0% of those under age 18 and 12.2% of those age 65 or over.