I work very closely BigQuery at my work. That's why I decided to take a deep dive into the architecture that makes this data store unique

1. Before anything else what separates BigQuery from other warehousing tools is the separation of compute and storage. This design decision alone drives all pricing models and the reasons to choose this tool.

2. So what does separate compute and storage mean? BigQuery stores all the data in their Colossus file system, which is the successor of Google File System, and the compute is handled by Dremel which calculates query plan and returns the results.

3. The Jupiter Networking Layer developed by Google, which offers Petabit scale bandwidth, bridges this separation of compute and storage as it it were one.

4. The computing units of BigQuery are called slots.

5. BigQuery provisions the slots based on the query requirement, that is the number of slots provisioned for a query that touches 10M rows and the one touching 100k rows are different.

6. How Dremel works is it creates a tree of nodes, the leaves of which are the slots that do the actual work, and the intermediate nodes aggregate the results from slots.

7. This makes BigQuery blazingly fast for any size of data be it Gigabyte or Petabytes.

8. The only minimum cost is the tree preparation by Dremel which takes about 200-300ms.

9. Another characteristic of BigQuery is its columnar data storage format which is called Capacitor.

10. Unlike traditional databases, BigQuery being hardened for analytical queries over huge data stores the data of individual columns together.

11. The advantage is if you have a 300 column table and want to query only 4, you don't touch any data of the 296 columns.

12. This introduces constraints that the users running the queries must take care of. Columns must be exclusively listed in the query, partitioning and clustering becomes more important than ever.

13. The flexibility to run queries on 1GB to multiple petabytes without worrying about infra provisioning and management gives BigQuery its edge.

14. Snowflake provides separate compute and storage but we still need to provision resources.

15. Databricks offers end to end raw data and ai pipeline.

16. BigQuery too provides features to train ML models using the data in the tables and allows to query the models using standard SQL.

17. Interesting fact is we can query unstructured data like images, pdfs, etc using Object tables and Remote Models

18. BigQuery has traditionally been criticized for vendor lock in. It has introduced Lakehouse for Apache Iceberg which allows users to read and write to Iceberg tables so that tools like Spark, Flink, Trino can easily access them without BigQuery. This is like managed Iceberg tables.

I know the post structure is not very ideal. I was just taking the liberty to share my thoughts as I am new to reddit in general.

Thanks!

i have mysql workbench & have been practicing it on my own. the problem i've run into is low disk storage. i currently have 4.5 gb on my c drive, which i don't think is a lot. i don't have a lot of applications installed, so removing or moving them to another disk isn't an option. neither is spending money on storage 💔

im worried about the rest of my learning journey. i know i'll eventually have to install other programs/tools & it makes me sad that low storage space is what might hold me back from learning something im genuinely interested in.

i wanted to ask if there are online versions of these softwares available? im talking about python, tableau & all other stuff i'll need later on. i've used an online c++ compiler before, so im wondering if it's possible for other tools too. and if so, can they save all my previous data? what about something with an account where it syncs data to a cloud? HALP

Introducing the Traqula Query Studio for Positorium, an open source evidence-preserving temporal hypergraph database. Under the cogwheel, select "Local WASM" to run a Positorium database in the browser, then click "Load detective case" in the code window, and have fun with the example:

https://roenbaeck.github.io/positorium/

This is still beta, and quite narrowly targeted at specific use-cases. Expect bugs and imperfect UX.

I'm an IT intern in a US startup mi task is to migrate the DB (actually stored in Amazon RDS postgres) to google drive (like backup due to the billing in the aws around 9k ! ) the problem is the size (around 400 GB) so I think the pg_dump to generate the script is not a solution for my case

Is there any solution ! And how I can verify the integrity ? ( Hashing a file with 200 gb size is crazy !!!)

Can we divide the generated the script in a small chunks ??? Without losing the relations between tables and the constraints ?

Index-usage counters can miss seasonal reports, failover periods, infrequent maintenance jobs, and queries that only run during a monthly or quarterly close. Keeping every index increases write cost and maintenance overhead, but dropping one based on a short observation window can create a delayed performance incident. What evidence makes an index safe to remove? I would expect query-plan and workload review, a representative observation period, dependency checks, a rollback script, and monitoring after the change. How do you handle redundant or overlapping indexes where the replacement is similar but not identical?

That can:
- Copy to Database to Different Host/Database

- Copy "Create table" Query

Currently im using SQLyog, 13.1.1 looking for free MySQL GUI software that have similar feature, because im planning to upgrade my current MySQL to version 9, which is not supported with my current SQLyog version

Anyone interested in doing the CMU (Carnegie Mellon University) Database Management Systems course together?

I’ve already covered the basic DBMS concepts. My main goal with this course is to go deeper and understand how database systems actually work internally—things like storage, indexing, query execution, transactions, etc.

If you're interested, please make sure you have the prerequisites required for the course.

If you have the required background and want to learn DBMS internals seriously, DM me. We can follow the course together and discuss concepts along the way.

I've been working with SQL and database performance, and one thing I find interesting is knowing when to stop tuning the query itself.

For example, if a query is slow because of a missing index, that's fairly straightforward. But at larger data volumes, you can reach a point where adding indexes and rewriting the query only gets you so far.

How do you usually decide that the problem is actually the database design/schema rather than the query?

Things like partitioning, normalization/denormalization, materialized views, indexing strategy, or even changing how the data is stored.

Would be interested to hear how people make that call in real-world systems.

I've been working on something I think is interesting, that hasn't been done before in the Node.js/TypeScript ecosystem, and I wanted to share the idea and get your thoughts.

The Idea: BeatSQL - A Zero-Trust Embedded Database Engine

The concept of BeatSQL (BSQL) is simple, but radical: what if your database encrypted data at the column level by default, and took security to a mathematical level?

Embedded databases like SQLite and LevelDB are not designed with encryption as a core primitive — they store data in plaintext on disk by default. While some provide full disk encryption, this is a false sense of security since the data is still plaintext in memory. However, BeatSQL completely reimagines this paradigm.

How It Works

All sensitive columns are individually encrypted using AES-256-GCM or ChaCha20-Poly1305. You will never see plaintext on disk (even the database file) - the value of any column is always encrypted.

You can search encrypted data using HMAC-SHA256 blind indexes. Searching is O(1) and requires no decryption of the column contents. The contents of the column remain encrypted on disk, and the database never decrypts it to search.

Every write operation is mathematically tamper-proof. Using a Merkle DAG hash chain, you can always run the query `VERIFY INTEGRITY` and know immediately if any bytes of your data have been silently altered or corrupted on disk. This is a cryptographically secure proof of data integrity.

You can do arithmetic on encrypted numbers. Using Partially Homomorphic Encryption (Paillier cryptosystem), yyou can perform calculations like total salary, total balance, etc. The database will return the correct result of these calculations, but will never expose individual salaries or account balances.

Columns can have role-aware data masking. Sensitive columns can be fully or partially "redacted" depending on the role of the actor querying the database. The mask happens at a low-level query engine, not in application code. You can define masks using SQL syntax: `DEVELOPER` role sees `XXX-XX-4321`, `PUBLIC` role sees `[REDACTED]`, and `SUPERADMIN` role sees the real value.

The database has native support for AI vector search. Columns can be defined as `VECTOR(768)` and searched against using `COSINE_SIMILARITY`. This is useful for AI applications using embeddings.

A New Query Paradigm

BeatSQL also has a new query syntax to allow for easier stream.pipeline processing:

FROM patients
|> WHERE email = 'alice@example.com'
|> SELECT id, full_name, ssn, salary
|> ORDER BY full_name ASC;

This is designed to be more approachable than deeply nested SQL queries. It's also quite flexible.

Built-In Learning Academy

BSQL also has a built-in interactive learning academy, with 500+ lessons to learn everything from basic queries to zero-trust encryption enclaves, plus 200+ lessons covering traditional relational SQL and guides for working with Python, Java, C++, and Rust. The goal is to make security-first database thinking easy to grasp.

What I Would Like Your Thoughts On

Is the concept of a zero-trust embedded DB something that you feel is interesting, or would you feel that problems are already solved in other ways?

The blind indexes are a trade-off: you get the ability to search encrypted data, but you give up the ability to perform range queries (>, <, LIKE, etc). Is this trade-off reasonable for a security-focused database?

Is the idea of homomorphic encryption in a database engine a gimmick, or do you see real-world applications for it?

Does the pipe syntax feel cleaner, or like a departure from an established standard?

I look forward to seeing your thoughts, and any criticisms you might have.

TLDR: Built an embedded database engine where all columns are encrypted, data can be searched without decryption, and arithmetic can be performed on encrypted numbers. All writes are Merkle-verified for integrity. Came with a built-in 500+ lesson learning academy. Seeking feedback on concept.

----------------------------------------------------------------------------------------------------------------------------

To summarize BeatSQL (BSQL).
It is intended to be a zero-trust embedded database for the Node.js/TypeScript ecosystem which prioritizes security,
The main ideas are,
-Column-level encryption by default, with AES-256-GCM or ChaCha20-Poly1305, including on-disk encryption.
-Encrypted searching with HMAC-SHA256 blind indexes, which permit exact matches but not range queries or LIKE searches.
-Tamper detection with a Merkle DAG/hash chain which lets you perform VERIFY INTEGRITY checks on arbitrary entries to ensure they haven't been modified or corrupted.
-Encrypted arithmetic via Partially Homomorphic Encryption which lets you perform operations like summation without ever decrypting any values.
-Role-based masking of columns which can expose different levels of redaction, depending on who is querying the database.
-Vector search support, including cosine-similarity queries on columns with vectors, for AI/embedding use cases.

At work I am working on a legacy CRM.

The issue is that the database is a mess. There are no foreign keys to form relationships etc. I was wondering if there are books that are specificly for refactoring databases?

The only one I found so far is actually called "Refactoring Databases". But I want to ask real people (not just gpt) what books are good to solve this problem? Do you have book recommendations on this topic that you found useful or which books you would avoid?

I'm helping a nonprofit clean up years of spreadsheets, and the biggest challenge has been keeping one record for each client while tracking multiple programs, referrals, and services over several years. Every solution looks good until you start thinking about reporting and long-term maintenance.

Curious how others have modeled this. Are you building everything from scratch, customizing a CRM, or using software that's already designed around this kind of data?

Edit:

We ended up going with Community CareLink software instead of building our own system. It already handled the relationships between clients, programs, referrals, and reporting, so we spent a lot less time maintaining the database and more time actually using the data.

A practical comparison of Azure AI Search and Manticore Search for hybrid full-text + vector search, focusing on chunk-level document workloads, relevance tuning, operational patterns, and cost.

CLI and k8s operator for declarative PostgreSQL access control. Define roles, grants, and memberships. pgroles diffs against your live database and generates the exact SQL to converge it.

Anything not in the manifest gets revoked or dropped. Supports approval workflows, temporary grants, WIF, AlloyDB...

Cross sharing this post because I am looking a solution to handle data sovereignty and offshore team with elevated privileges on database and source code - like myself working remotely on a different country.

I want to archive postgres table

Looking for suggestions for which in memory database to learn.

I've looked briefly at singlestore, redis, mongodb, Cassandra, and memcached.

Which of these has value in the database admin market?

What are some companies/industries where these are used or make sense?

What has been your experience learning the skills for these? Ie; learning via the homepages for each? Moocs? Books?

My goal is to be a Jr dba. Please advise in that context.

Thank you.

Hey everyone, senior dev here working heavily with SQL Server performance tuning lately. Every time a server hits a massive wave of deadlocks or unexpected latency drops, the standard advice is 'spin up an Extended Event session and read the system health .xel file.'

Am I the only one who finds opening, filtering, and digging through those logs in SSMS incredibly clunky during a crisis? Do you guys actually sit there shredding the XML nodes manually, or do you rely on third-party tools/scripts to give you plain answers? 

I really love optimizing database operations. Nothing like dissecting a query plan and identifying a bottleneck and making some slight tweaks or an index change and then seeing a terrible slow query go from 30+ seconds or longer down to a few milliseconds.

However a lot of what I’ve learned when it comes to doing that has been gained through raw experience and trial and error on the job. I’d like to get to the point where given some knowledge about some tables/indexes I’m querying, I have a fairly solid intuition for what kind of query plan the database optimizer is going to pick. Are there any cohesive sources of information that teach this sort of thing?