I'm starting (in a rather ad-hoc fashion) to compile texts which may support the case that God will renew and not replace this creation when he comes to settle the scores and set up home with his people forever. First, here's some verses from the psalms.
Some of the texts merely allow (with varying degrees of strength) for a continuationist reading (by, for e.g., implying that judgment day will be the removal of wickedness/the wicked from the earth, not the removal of the earth, or of the righteous from the earth).
In others, it seems to me that the conclusion is inescapable.
The face of the LORD is against those who do evil,
to cut off the memory of them from the earth.
Say among the nations, "The LORD reigns!
Yes, the world is established; it shall never be moved;
he will judge the peoples with equity."
Let the heavens be glad, and let the earth rejoice;
let the sea roar, and all that fills it;
let the field exult, and everything in it!
Then shall all the trees of the forest sing for joy before the LORD,
for he comes, for he comes to judge the earth.
He will judge the world in righteousness,
and the peoples in his faithfulness.
He set the earth on its foundations, so that it should never be moved.
Let sinners be consumed from the earth,
and let the wicked be no more!
Your faithfulness endures to all generations;
you have established the earth, and it stands fast.