Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)

[Otto]

Hello friends,
since there are currently very few technical problems, I think it's a good time to deal with future-oriented topics.
**I'll start with a provocative statement.**
I'm curious to hear how you see the future.

Best regards,
Otto

---

## Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)

1. SQL hides your data behind layers of magic

“You don’t own your data if you can’t see it.”

* Data is trapped in engines and formats
* You need specialized tools to export, debug, or back it up
* Indexes? Hidden. Logs? Hidden. Snapshots? Only if the DB supports them
* With files and DBF: Your data is visible and copyable as `.dbf`, `.json`, or `.txt`

---

2. Filesystems are more transparent, flexible, and AI-friendly

“AI loves structure. Not secrecy.”

* AI works best with plaintext, JSON, Markdown, CSV
* SQL? You have to export, transform, clean, and explain the schema
* With DBF + memos + JSON:

* Every record is ready for NLP or semantic processing
* Every field is directly searchable with `grep` or shell scripts

---

3. SQL is a monolith. Files are composable.

“You don’t need a cathedral to store customer addresses.”

* SQL locks you into transactions, schemas, and foreign key drama
* Every new tool needs a connector or query engine
* With DBF:

* You can output JSON, build views, or parse records with any scripting language
* Works offline, in the browser, in a microservice, or even from a USB stick

---

4. Query logic should live in your code – not buried in a database engine

* `SELECT * FROM sales WHERE status = 'paid'` is readable – but trapped
* As soon as logic lives in SQL views, triggers, or stored procedures:

* It’s no longer portable
* Not versionable
* Not reusable outside the SQL engine

With PHP, JS, or Python:

* Your filters, validations, and logic live in real code
* In Git, in your editor, right next to your UI

---

5. Want auditability? You won’t find it in SQL logs

“Dump files ≠ backups. Binary logs ≠ trust.”

* With files, auditing is easy:

* Use Git, rsync, `diff`, or timestamped snapshots
* With SQL:

* You have to trust the engine, hope logging was enabled, and pray your dump isn’t corrupted

File = Control.
SQL = Delegation.

---

6. Want AI or LLM integration? SQL is your bottleneck

* AI doesn’t query databases – it processes text
* In the end, you’re exporting SQL data as JSON anyway
* So why not store it as JSON, CSV, or DBF from the start?

Examples:

* Memo fields as `.md` → embeddings, summaries, Q\&A
* `ripgrep` across DBF memos = instant full-text search
* JSON APIs from `php4dbf_list()` → LLM-compatible feeds

---

7. SQL is good for scaling. But most systems don’t need Google scale.

“Most projects need 10,000 records. Not BigQuery.”

* SQL shines at massive scale – but brings setup, maintenance, and lock-in
* Flat files + microservices are:

* Easier to deploy
* Easier to secure
* Easier to migrate

---

### Summary: SQL isn’t bad – but for many modern workflows:

**Plain Files + Microservices + Grep > Databases**

Feature | SQL | Filesystem + DBF + Tools
---------------- | ---------------------------- | -----------------------------
Portability | Engine-bound | USB-ready
Auditability | Hidden logs | Git, diff, snapshots
Transparency | Tool-dependent | Text editor and shell
AI suitability | Requires export | Native formats
Learning curve | SQL syntax & logic split | Pure scripting
Web integration | Requires APIs | Direct access via PHP/JS
Complexity | Views, joins, keys, rights | Files, folders, grep


---

Want web, AI, and data sovereignty? Start with:

* DBF (or JSON/CSV)
* Plaintext memos
* Modular scripts
* APIs instead of queries
* Git instead of binary logs

**SQL is useful – but often oversized.
The future belongs to files, not engines.**

[Enrico Maria Giordano]

I don't know who or what has written that, but I agree.

[Otto]

Enrico, your opinion means a lot to me because I know you're a top expert.
Best regards,
Otto
PS: EMAG, as we call it, is one of the most used and productive tools in the company.

[Enrico Maria Giordano]

Thank you, but I am overrated too.

[Otto]

Enrico, I think we've known each other since 1994. I know what I'm talking about.
Best regards,
Otto

[Enrico Maria Giordano]

1994? Wow! Are you sure?

[Adolfo]

Hi..
I disagree. DBF is not good, at least for me

Have some supermarkets as clients.
Some invoice and sales details have 30.000.000 records a year, can´t make any summary, report on that table with 5 relations in a DBF format, nope, 15+ minutes against 1 second. Products table, 30.000 records in a mid size hardware store

6 GB of data in 45 tables, 22 POS, 10 admin pc's, no problem with simultaneous access, read and write in a MariaDb, It was a nightmare with DBF's, used to have temporary dbf's to make fast work

Really, DIRECT ACCESS TO DISK WITH DBF, NOPE

A cheap server , I5, 4 gb ram, and a 128 gb SSD for 15 pc's connected to it, only access thru port 3306, security granted.

Can´t relay on a DB structure you can copy a file (dbf) and stole your data. (So visible to everyone)

Wanna save images, docs, xlxs, json, pdf's in a record, no matter the size, there you have BLOB, try this in a DBF.

“AI loves structure. Not secrecy.” Try Deepseek and sql db. Runs like a charm

"Works offline" there you have embeded Mysql

WEB app, SQL (anyone, even NOSQL) is the way to go. PHP, Python, Javascrit, Angular, React....

Is soooo cheap to have a hosting with SQL (here in Chile U$ 30 a YEAR, 10 GB space, 3 db's), access from everywhere, no special server, daily backups

I used dbf for 25 years, not going back.

My 2 cents.

[Otto]

Hello Adolfo, thank you.

What always bothered me about SQL is this:

The data doesn’t feel like it belongs to me.
Especially in document-related workflows (DMS), the database engine starts acting more like a gatekeeper than a helper.
It slows me down where I need flexibility, transparency, and raw access.

SQL is fine for POS – but breaks down in DMS.

SQL is fine for POS – but breaks down in DMS.
Point-of-sale systems are mostly standardized:

Articles, prices, transactions, customers – structured, repetitive data

Few transaction types (sale, refund, payment)

SQL works well here:
Schema is fixed
Data volume is predictable
Performance is fine with caching

Conclusion: SQL is a solid choice –
But a clean DBF or JSON setup would often be more portable, transparent, and flexible.

Where SQL struggles: Document Management Systems (DMS)
In a DMS, things get messy:

Mixed file formats (PDF, DOCX, JPG, XLSX)

Unstructured metadata (OCR, comments, notes, tags)

Legal audit trails, file versions, attachments

User access vs. process automation vs. AI analysis

SQL schema becomes unstable:

BLOB storage is opaque, hard to search or audit

Full-text and AI needs require external tooling (Elasticsearch, LLM wrappers)

Versioning and diffing are nearly impossible

Schema breaks with every new edge case

Better for DMS: A file-based, index-driven system
Use:

JSON for metadata

Real files for content

.dbf as a blazing-fast index

Git or timestamp snapshots for versions

ripgrep, jq, or LLM for analysis

DBF = high-speed index – like a manifest.json, but grepable and scriptable

Example DBF index:
doc_id filename date type tags version deleted path_hint
10001 RE_1023.pdf 2025-05-12 invoice finance,2025 1 0 /docs/2025_05/
10002 leasingcontract.pdf 2023-01-10 contract leasing,legal 3 0 /docs/contracts/
10003 memo_MAY.txt 2025-05-01 note internal,team 1 1 /docs/memos/

Filesystem layout:
swift
Kopieren
Bearbeiten
/docs/
/2025_05/
RE_1023.pdf
RE_1023.meta.json
/contracts/
leasingcontract.pdf
leasingcontract.ocr.txt
leasingcontract.changes.json

Queries & AI Analysis
Search → php4dbf_search("finance AND 2025")

Embedding / NLP → read .ocr.txt or .meta.json
Version tracking → Git diff on .changes.json
Backups → rsync, borg, restic, etc.
What happens if you store PDFs or images in SQL BLOBs?
You lose the original:

No accessible file
No version history
No direct inspection
No compatibility with external tools
No AI-readiness

BLOBs live inside the database engine (e.g., .ibd in InnoDB).
You need SQL to extract them – no file, no transparency.

SQL is good for structured, known data.
Files + JSON + DBF are better for complex, unpredictable domains.

For AI, compliance, and future-proof data ownership,
Don’t store the document in the database – store it like a document.

Auditability and Forensic

I believe that if you're serious about compliance and integrity in a POS system, you have no choice but to log each transaction at the time of entry – either as a .json or .dbf record.
Especially when auditability and forensic traceability are part of the business requirement.

A tough but realistic test: the forensic live audit
Let’s imagine a real-world scenario:

Situation:
Management suspects fraud or manipulation

An external audit team is deployed

Immediate system lockdown: network cables pulled, server process halted

IT is not allowed to touch anything

The demand:

“Show us – from raw, immutable data – the last 3 receipts from cash register #3. Right now.”

What happens in a classic SQL-based system?
Problem Why it’s dangerous
Data stored in binary (InnoDB, etc) Human-readable inspection is impossible
Dump or SELECT needs the engine Engine is off → no access
Data is interpreted through queries Can’t prove the query wasn't manipulated
Timestamps/logs are often internal Hard to verify retrospectively
Recovery after crash is fragile Might be incomplete or corrupted

Without a prior export or write-time logging strategy, you're exposed.

Better: DBF- or JSON-based live logging
If every receipt is saved immediately as its own .json or .dbf record, you can say in the audit:

"These are the originals – not reconstructed, not interpreted. Stored exactly as they were created."

Example receipt file:
{
"timestamp": "2025-05-27T11:36:21",
"register": "3",
"operator": "Anna",
"items": [
{"product": "Coffee", "qty": 1, "price": 2.50},
{"product": "Cake", "qty": 1, "price": 3.50}
],
"total": 6.00,
"payment": "Cash"
}
File structure:
/sales/register3/
Q2025-05-27_11-36-21.json
Q2025-05-27_11-36-22.json
Q2025-05-27_11-36-23.json
No query required
No engine needed
No excuses possible

In SQL, you must defend the query.
In a log-based system, you just show the original.

True audit safety means:
Log it when it happens, not when someone asks.
Keep it readable, verifiable, and context-free.

Best regards,
Otto

[Lailton]

I totally agree with Adolfo.

DBF gives me a lot of headaches ( in special with the CDX ). It’s acceptable for some very specific tasks, but to be honest, I’m trying to eliminate anything I have that still uses it.
For situations where I need a large database, MariaDB has worked well for me. For smaller projects, SQLite (or SQLCipher for encryption) is the one that I am using.

[Enrico Maria Giordano]

DBF databases and SQL databases are both files, so they have the same advantages and the same problems. The "legend" that SQL is more secure is only a "legend", a false sense of security. The fact is that SQL is very much limited and slow if compared with DBF.

[Adolfo]

DBF databases and SQL databases are both files, so they have the same advantages and the same problems. The "legend" that SQL is more secure is only a "legend", a false sense of security. The fact is that SQL is very much limited and slow if compared with DBF.

---> DBF databases and SQL databases are both files, so they have the same advantages and the same problems.

In order to work with a DBF you need DISK ACCESS, read and write, while in a little Linux server, you CAN'T, so no way you can compare them.

---> The "legend" that SQL is more secure is only a "legend", a false sense of security.

Enrico, you only access data thru a PORT, there's no way you compare it with DISK ACCESS

---> The fact is that SQL is very much limited and slow if compared with DBF
Thats a lie.

[vilian]

I agree with you Adolfo. Moving from DBF to MySql was the best I have done.

[Otto]

Hi Adolfo, thanks for sharing your experience – and I respect your migration to MySQL.

But I’d like to clarify a few things – especially for others reading along:
"You need disk access for DBF – on Linux that’s impossible"
That’s not accurate. Disk access works perfectly on Linux – DBF is a plain file. What’s not safe is using DBF via network file shares (like SMB/NFS).
But if you use DBF server-side, behind Apache or via PHP/Python, it works reliably – just like any flat file or SQLite.

"DBF and SQL have the same advantages and same problems"
No – the difference is architectural:
DBF gives raw access, perfect for transparency, offline scenarios, or embedded systems.
SQL gives abstracted access, useful for multi-user concurrency and remote APIs.
They solve different problems.

"SQL is not more secure"
True. Security is about architecture, not engine.
If someone gains server access, they can extract .ibd just as easily as .dbf.
SQL just feels safer – until your mysqldump is corrupt.

"SQL is faster"
Sometimes. But:
DBF reads single rows in milliseconds
No JOINs = no overhead
You can grep, tail, hash, or version-control DBF files with system tools
DBF excels in transparent, modular, embedded workflows

They’re tools. Choose the one that matches your architecture, your risks, and your philosophy of control.

I prefer to own my data – and for that, files win.

Best regards,
Otto

[Enrico Maria Giordano]

---> The fact is that SQL is very much limited and slow if compared with DBF
Thats a lie.

As you wish, but a fact is a fact, sorry.

[Maurizio]

I agree with you Adolfo .
I've been working with MySQL for 10 years, it's a shame I didn't start earlier.
Maurizio