Chalav stam is at worst a din d'rabanan.
For what purpose? The "Issur" was already done, what did that accomplish?
See Y"D 81,7
RaM"A
(re: nursing a baby)
"for milk from an idolater is metamtem haleiv (of the baby consuming the milk) and will give the child a bad nature, therefore a nursing (mother) should not eat anything prohibited, nor should the child, for all this will harm him in his older age."
Ta"Z - s"k 12
"all this (particularity about not having a child nursed by a lady that has eaten issue) applies even in circumstances where the lady was permitted to eat (normally) prohibited food for reasons of pikuach nefesh (in which case she was
halachically obligated to eat it ) nevertheless she should not nurse the child"
Sha"Ch- s"k 25
"if she is sick in a fashion that we are obligated to feed her (normally) prohibited foods, the father of the child should not give the child to be nursed by her"
Sha"Ch - s"k 26
"even though when a child eats
an issur d'Rabbonnon the father is not obligated to separate him (from the transgression).....this is regarding the Halacha (of chinuch), however he should separate him for it will harm him in his older age for it is metamtem the heart and will cause him (to have) a negative disposition"