By default a continuous aggregate is a realtime one, which means it’ll query raw data for all time ranges not being materialized yet. This is all the data in your case since you haven’t refreshed anything yet.
If you only want materialized data (hence nothing as of now), you need to set materialized_only to true. See parameters: Timescale Docs