Where had "hate thine enemy" been said?

Ye have heard that it hath been said, Thou shalt love thy neighbour, and hate thine enemy.

But I say unto you, Love your enemies, bless them that curse you, do good to them that hate you, and pray for them which despitefully use you, and persecute you;

-- Matthew 5:43f.

Hate thine enemy was said at Qumran: 

... to love all that He has chosen and to hate all that He has rejected ...

to love all the Sons of Light, each according to his lot in the council of God,

to hate all the Sons of Darkness, each according to his guilt in the vengeance of God ...

 

-- 1QS, The Community Rule 1.4f, 10f.

 

But hate thine enemy was also said in old Athens, reflecting a widespread societal value: 

Justice benefits friends and harms enemies.

-- Plato, Republic 1.334b.

Evidently the Greek leaven had leavened the Jewish lump, and Jesus was nothing if not a reactionary Jew opposing such degeneracy.

Jesus' appeal to good works in response to evil kept to something hardly novel within Judaism, even if the eschatological moment was:

If thine enemy be hungry, give him bread to eat; and if he be thirsty, give him water to drink: For thou shalt heap coals of fire upon his head, and the LORD shall reward thee.

-- Proverbs 25:21f.

What was truly novel was Paul's later application of this beyond the boundaries of Judaism: 

Therefore if thine enemy hunger, feed him; if he thirst, give him drink: for in so doing thou shalt heap coals of fire on his head.

-- Romans 12:20.