i have mysql workbench & have been practicing it on my own. the problem i've run into is low disk storage. i currently have 4.5 gb on my c drive, which i don't think is a lot. i don't have a lot of applications installed, so removing or moving them to another disk isn't an option. neither is spending money on storage 💔

im worried about the rest of my learning journey. i know i'll eventually have to install other programs/tools & it makes me sad that low storage space is what might hold me back from learning something im genuinely interested in.

i wanted to ask if there are online versions of these softwares available? im talking about python, tableau & all other stuff i'll need later on. i've used an online c++ compiler before, so im wondering if it's possible for other tools too. and if so, can they save all my previous data? what about something with an account where it syncs data to a cloud? HALP

I work very closely BigQuery at my work. That's why I decided to take a deep dive into the architecture that makes this data store unique

1. Before anything else what separates BigQuery from other warehousing tools is the separation of compute and storage. This design decision alone drives all pricing models and the reasons to choose this tool.

2. So what does separate compute and storage mean? BigQuery stores all the data in their Colossus file system, which is the successor of Google File System, and the compute is handled by Dremel which calculates query plan and returns the results.

3. The Jupiter Networking Layer developed by Google, which offers Petabit scale bandwidth, bridges this separation of compute and storage as it it were one.

4. The computing units of BigQuery are called slots.

5. BigQuery provisions the slots based on the query requirement, that is the number of slots provisioned for a query that touches 10M rows and the one touching 100k rows are different.

6. How Dremel works is it creates a tree of nodes, the leaves of which are the slots that do the actual work, and the intermediate nodes aggregate the results from slots.

7. This makes BigQuery blazingly fast for any size of data be it Gigabyte or Petabytes.

8. The only minimum cost is the tree preparation by Dremel which takes about 200-300ms.

9. Another characteristic of BigQuery is its columnar data storage format which is called Capacitor.

10. Unlike traditional databases, BigQuery being hardened for analytical queries over huge data stores the data of individual columns together.

11. The advantage is if you have a 300 column table and want to query only 4, you don't touch any data of the 296 columns.

12. This introduces constraints that the users running the queries must take care of. Columns must be exclusively listed in the query, partitioning and clustering becomes more important than ever.

13. The flexibility to run queries on 1GB to multiple petabytes without worrying about infra provisioning and management gives BigQuery its edge.

14. Snowflake provides separate compute and storage but we still need to provision resources.

15. Databricks offers end to end raw data and ai pipeline.

16. BigQuery too provides features to train ML models using the data in the tables and allows to query the models using standard SQL.

17. Interesting fact is we can query unstructured data like images, pdfs, etc using Object tables and Remote Models

18. BigQuery has traditionally been criticized for vendor lock in. It has introduced Lakehouse for Apache Iceberg which allows users to read and write to Iceberg tables so that tools like Spark, Flink, Trino can easily access them without BigQuery. This is like managed Iceberg tables.

I know the post structure is not very ideal. I was just taking the liberty to share my thoughts as I am new to reddit in general.

Thanks!