Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)

User avatar
Otto
Posts: 7020
Joined: Fri Oct 07, 2005 7:07 pm
Contact:

Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)

Post by Otto »

Marcelo, you're right.
But it has some limits regarding access, security, navigation, etc.
That's exactly why we should work on the DBF format and on standardization.
The truth is, no one is really taking care of its development as a whole anymore.
But in our group, there are many database experts — and you're now pointing out what's missing.
I believe we could solve this quickly and extend DBF if we all pulled in the same direction.

I had the issue with mod\_harbour where almost 200ms pass just for startup and data access — that’s too slow for paging.
Another mod\_harbour development, version 2, is supposed to be faster in this regard.
But this group is again focusing on SQL.
So I started looking into the format itself and created a driver for PHP.
And I see now that this whole data access topic isn’t that complicated.
As a group with so many experts, we could quickly solve the few missing parts.
As far as I know, ADS is currently in maintenance mode.

Best regards,
Otto
User avatar
Giovany Vecchi
Posts: 244
Joined: Mon Jun 05, 2006 9:39 pm
Location: Brasil

Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)

Post by Giovany Vecchi »

Hello everyone. I have been using Ads since 2004.
I started with ads+dbf and around 2008 I started using ads with a data dictionary with adt tables.
I work with both ISAN and Static and I mix the two methods without any problems.
The speed is much higher.
Here is the tAds class that I created to be able to work with a data dictionary using DataSet style objects and with an example in the .\Samples folder.
This example will access a computer in my house. The computer is a dual core 1.2mhz. The internet is 200mb.
Since my internet belongs to private companies, access according to the source will not have a good speed. For this I would have to have a dedicated server in the cloud.
The example does not include tables.

https://mega.nz/file/7bZSEJYJ#1EQa6ckGx ... 2wEaN55ctc
Jack
Posts: 307
Joined: Wed Jul 11, 2007 11:06 am

Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)

Post by Jack »

Hello,
I develop software with MS Sql Server sinse 10 years .

I 'll no more use DBF in network development.

Select with linked table is very easy .

Have a good day .

Philippe
User avatar
reinaldocrespo
Posts: 991
Joined: Thu Nov 17, 2005 5:49 pm
Location: Fort Lauderdale, FL

Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)

Post by reinaldocrespo »

Ok, I will try to chime in this subject:
#1-- SQL hides your data behind layers of magic “You don’t own your data if you can’t see it.”
#2. Filesystems are more transparent, flexible, and AI-friendly “AI loves structure. Not secrecy.”
SQL is "magic" or "mysterious" or "incomprehensible" --yes, but only to anyone who does not understands it. Just like DBFs are incomprehensible to my cyclists friends who are great cyclists but have never seen the command prompt. SQL is not more complicated. Just different.
4. Query logic should live in your code – not buried in a database engine
This sounds nice, but in reality it is a lot easier to have your database reject certain transactions or behave in a specified manner than worrying about any possible way that any given code here or there that I may forget exists and writes or reads from a table may behave under certain unforeseen conditions. Why not just have the db centrally enforce the rule?
5. Want auditability? You won’t find it in SQL logs. * With files, auditing is easy:
* Use Git, rsync, `diff`, or timestamped snapshots
* With SQL:
Uffff, nothing further from the truth. I can't imagine asking a user to git, svn, diff, or anything else looking for how data changed, by whom and when. With SQL I simply have a trigger that writes to the audit table anytime a record is added or changed or deleted. Show audittrail table records that match a unique record id to see the audit trail on any given record. That's it. User gets to see full auditing. All you had to do was write the trigger that logs changes and you only write that once.

I'm sure a dbf user does not miss, money fields, or blob fields, or timestamp fields, or VarChar fields, or TinyInt fields. A DBF user does not miss long field names like: PropertyPhotoID and the such. **BUT** once you start using these fields, break through dBF limitations, start using triggers, have db enforce data integrity where no child is left orphaned... it will be hard to go back.

I can understand smaller systems with 5 or even 10 users and 10k records working with dbf fairly well. But if you ever aspire to have your system be used by 25 concurrent users, store millions of records, with with extended field types... I would think past dbfs. Even for smaller systems with less than 50 tables and 5k records, I find it is safer, easier, faster to write using some SQL.

I see Mr. Rao has done a great job at making it easier to transition to MySQL. He even has a method to import data from dbfs to MySql. I had to write my own since I was porting from ADS, including stored procedures and functions. My advise is to take advantage of Mrs. Rao's work and jump right in. Eventually SQL will be second nature and you will have a hard time staying with just ISAM on dbfs.

BTW-- any hosted server (Amazon, Oracle, DreamHost...) have MySQL db already configured so your system will run on any 3rd party server.

Just something to think about.
shrifw
Posts: 76
Joined: Fri Aug 28, 2009 5:25 am

Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)

Post by shrifw »

Dear All ,

SQL has strong features:

a) Powerful Indexing almost no rebuild is needed
b) SQL does aggregation in very few lines
c) Strong transaction management for OLTP Apps
d) Concurrent Management

many more...!

Thanks
Shridhar
User avatar
Otto
Posts: 7020
Joined: Fri Oct 07, 2005 7:07 pm
Contact:

Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)

Post by Otto »

Hi Reinaldo,
Thanks for the input – from a technical point of view you're right: most hosted servers come preconfigured with MySQL and are ready to use.

However, things are quite different for us here in the EU, especially in Austria and Germany. We have to follow strict GDPR rules when working with personal data – and that makes using third-party hosting (like Amazon, Oracle, etc.) more complicated.

Key points we need to consider:

Data transfers to non-EU countries (like the US) are problematic due to legal access (e.g., the CLOUD Act).

A data processing agreement (DPA) is mandatory – and must be actively signed.

Technical and organizational measures (TOMs) must be clearly documented.

Full transparency is required about where and how data is stored and processed.

That’s why many of us prefer EU-based or even self-hosted solutions when dealing with sensitive user data.

Best regards from Austria!
Otto
User avatar
Otto
Posts: 7020
Joined: Fri Oct 07, 2005 7:07 pm
Contact:

Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)

Post by Otto »

Hello,
I fully agree that SQL brings powerful features, especially for classic business systems:

a) Powerful indexing with minimal rebuilds
b) Fast aggregations in very few lines
c) Reliable transaction management for OLTP
d) Great concurrency handling

These are well-proven advantages, and they make SQL a great choice for established use cases like financial systems, administrative records, and structured reporting.

But I’m currently working on something different — a highly dynamic, context-driven workflow for processing human communication (e.g. anonymized emails, JSON mappings, context-aware responses). Everything is in constant flux:

The data structure is semi-structured at best (e.g. [Guest:ID], [Date:ID])

Placeholder formats evolve frequently

Contextual logic (e.g. temporal flow, RAG-augmented reasoning) is essential

Workflows need to adapt quickly — not go through migration scripts and schema locks

In such a case, I feel SQL might get in the way rather than help — and that file-based + script-based approaches (e.g. JSON + Python) offer more agility.

That said:
Are there scenarios where SQL could still add value in such dynamic, non-relational contexts?
Or would you agree this falls outside of where SQL shines?

I’d love to hear your thoughts — especially if you’ve built hybrid approaches.

Thanks!
Otto
shrifw
Posts: 76
Joined: Fri Aug 28, 2009 5:25 am

Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)

Post by shrifw »

Dear All ,

UPI Indian Payment System is great example which is running on SQL database e.g. Oracle , PostgreSQL , MySQL approx. 1000 TPS in one bank.

Thanks
Shridhar
User avatar
Otto
Posts: 7020
Joined: Fri Oct 07, 2005 7:07 pm
Contact:

Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)

Post by Otto »

Thanks Shridhar – technically you're right, but here in Europe a system like UPI would never be built with something like Harbour. It lacks scalability, security certifications, and enterprise support. Harbour is better suited for small, local business applications.
By the way – are you really considering Harbour for such systems, or was it just an example?
Best regards,
Otto
shrifw
Posts: 76
Joined: Fri Aug 28, 2009 5:25 am

Re: Why SQL Is Overrated – and Why Files, DBF, and Microservices Are the Future (Especially for Web and AI)

Post by shrifw »

Dear Otto ,

As Harbour using C platform it should be used also to develop scalable apps. I am not expert in developing languages but I do not know what it is lagging HVM from JVM ( java virtual Machine ) ?

Thanks
Shridhar
Post Reply