Reading Exodus 20:11 to mean an insistence on six literal 24 hour days of Earth time is (a) taking it out of context, (b) missing the point, and (c) placing ourselves rather than God at the centre of the universe. The point of Exodus 20:11 merely uses the days of creation in Genesis 1 are a schematic, or a blueprint, for the working week. Nothing more, nothing less.
In any case, the days of Genesis 1 can’t refer to literal 24 hour days, because Genesis 1 talks about things on a cosmological scale rather than on a geocentric scale. Even within the Solar System there is no fixed concept of how long a day is. It may be 24 hours on Earth, but it’s 28 days on the moon, 10½ hours on Saturn, and 243 Earth days on Venus. Even on Earth, the idea of “evening and morning” breaks down when you start to consider time zones.
And then of course we have 2 Peter 3:8 and Psalm 90:4. Nothing to lose sleep over here.