At 50 Years Old, Is SQL Becoming a Niche Skill?

I think SQL is due for a renaissance, and I hope htmx is part of that. I know htmx has a reputation as being something only a backend developer could love, but I think that in a few years some of the most effective htmx developers are going to be front end developers who know the web stack (CSS, JS, HTML) well and who have figured out how to use SQL effectively.

If you are smart enough to figure out GraphQL, you are smart enough to figure out SQL, and SQL comes without the security/expressiveness tradeoff[1] inherent in front-end data access technologies. You have hypermedia-oriented tools like hotwire, unpoly & htmx pushing data access back onto the server, and at the same time things like React Server Components doing the same.

I'm very bullish on SQL.

1 - https://intercoolerjs.org/2016/02/17/api-churn-vs-security.h...

No, the number of tools JS developers have made to avoid learning SQL if it was going to die out it would have done by now.

All that collective effort to avoid learning one language and thinking about your data structures first is staggering.

Among software engineers it’s possible SQL is becoming a niche skill. I’m a data scientist and probably write SQL at least 4 hours a day, lately that’s been more like 10. SQL is an incredible language and I strongly prefer it to python for data processing.

Sort of, what I've been noticing is that it's considered a background skill. Until the database and your queries need to be optimized for higher load, then suddenly everyone wants a DBA.

To all junior SWEs: read a SQL book over the weekend. You'll end up being the respected DB expert on most calls after literally a few weeks. Most everyone never progresses above a SELECT with a couple JOINs and then complain the DB is slow and need to push work into the backend service where usually the opposite is true.

Ha, SQL is one of only a handful of computer languages that is both older than most of us and probably will outlive most of us.

I worked at an organization that ran an energy market and we had a massive amount of data stored in SQL databases and this was how you did large-scale data analysis. Most electrical engineers there knew the schema backwards and forwards and could write multi page queries (there are a LOT of tables and many many join operations) pretty quick to go through TB, probably at least a PB of data. It wasn't considered a niche skill...more like bread and butter (equally useful as Python). Every now and then we'd hire someone that had a "data science" background that wouldn't know SQL (just some stats and R or Python+Pandas) that would try to convince us we were doing things wrong, but it was generally because the "big data" they learned in school could fit into a dataframes. I'd say SQL is now more important than ever.

How do you do

    SELECT AVG(price) FROM cars
    WHERE brand='BMW'
    GROUP BY year
    ORDER BY year

without SQL?

So far, every alternative syntax I have encountered was less human readable and/or less concise.

I think SQL will never be replaced because it is not really something that was invented but rather the simple consequence of wanting to store and retreive data.

You could replace "SELECT" with "GET" or "FETCH". But in the end you need to define what action you want to perform. You could replace "WHERE" with "FILTER" or "CONDITION". But in the end you need to define what you want to retreive. Etc. So by just stating the neccessary information, you always end up with SQL. And every real alternative has the problem of being less concise, less easy to read and less easy to write.

I think relational databases as a technology are absolutely the bees knees. It still boggles my mind that devs are choosing NoSQL databases for their products right out of the gate and throwing away 50 years of relational alegbra database technology.

SQL as a technology did an admirable job of allowing practitioners to express relational queries, but let's be real - the syntax of SQL, especially for advanced queries, was always pretty awful. I think that syntax, more than the underlying DB technologies, was the number one factor driving people towards NoSQL unnecessarily.

What would be amazing is if the RDBMS community could create an industry standard querying language that supported relational algebra but actually had a pleasant development experience.

For my work I use SQL pretty extensively, the PostGreSQL flavor, and it's really powerful. I wonder how it is possible to do without except for very simple queries where the ORM might kind of work.

There is however a weakness of SQL is that it's purely declarative and it's difficult to make sure it does the right thing. I often find myself in front of queries that are badly optimized and I'd welcome a language that would be less declarative where you could tell the database engine which index you want to use and how explicitly. The optimizer has table metrics but with the domain specific knowledge the programmer has generally better insights.

A basic understanding of SQL is one of the most useful skills a software Engineer can have. Sooner or later, you will get to face a database system that requires it, either for actual application development or for data mining.

Even if you don't use it all that often, I think that every developer should get some exposure to Functional, Logic and Declarative languages - and SQL is a decent way to learn more about the "declarative way of thinking", even if it isn't used in a fully declarative fashion in many practical situations.

I’m getting pretty damned tired of NoSQL as a default. Picking it because it matches your use case exactly is one thing, picking it because you want to avoid modeling your domain, building a schema, and managing migrations is a fool’s errand. All you’re doing is trading problems and ass ache, but I think NoSQL is less flexible than SQL. Hell, DocumentDB is built on Postgres, allegedly (is that confirmed?).

I started at a new place that uses DynamoDB and I can’t articulate how much I loathe it. Currently rewriting it into Postgres as we speak.

When I started my dev career in the early aughts, by no particular plan, ended up down the route of a DBA after doing a few Access to MSSQL migrations. Wrote some really gnarly SQL sprocs back in those days.

In those early days, ORMs were really bad and didn't feel very ergonomic to use.

But the tooling has come such a long way. Today, ORMs are generally pretty competent for 80-90% of the use cases I would find in a typical business app. Microsoft's Entity Framework always seems to surprise me when I find that I can do fairly complicated queries and have it generate almost perfect SQL. It's migrations are pretty good to the extent that it's rare to have to manually amend its migrations. Even though I think I'm quite good at hand rolling SQL, EF Core just makes it far more productive to work at the application layer instead.

I think the biggest hurdle to SQL today is that the DBA is a dying breed and SQL hasn't really had a "revolution" to meet the (sometimes perceived) needs of younger teams. For example, scaling relational database still requires more forethought and each approach comes with some limitations and gotchas. On the other hand, the rise of NoSQL databases and JavaScript in the application stack means that document-oriented databases like Firestore and CosmosDB are perhaps easier for teams to adopt, even with the limitations, because infrastructure concerns (scaling, replication, etc.) disappear.

Good/serious SQL I think yes. It's gone except in legacy systems.

We see a lot of complexity moving into the application and the rdbms used mainly as a dumb table storage. The whole idea of normalised models is out the window, which kinda makes sense because that's a carryover from the 70s when storage costs were high and now they are negligible and development costs/time are instead the barrier. Really nobody gives a crap about their database size :P

I do see some stuff like stored procedures for security but the whole complex 10-table hyper optimised join, the kind you tweaked for a week so it could finally run at an acceptable speed and you'd feel like a magician, that stuff seems to be over.

These days if you need some kind of summary you just add it to the business logic and keep a separate table updated. Of course you have total duplicate information and a risk of mismatches but at agile breakneck speeds and basically zero storage costs this is not an issue these days.

I never ended up taking those database electives at uni

In practice though, SQL just seems like a lower-barrier to entry lingua franca than C (for its use)

Unless it’s absolutely essential to your performance -like GIS or data-driven stuff (at which point you might be looking beyond SQL) - I don’t really see it as a highly specialised niche skill

I think most people see it (perhaps rightfully) as a commonly understood but seldom directly used paradigm in isolation

Picking up an ORM is comparatively child’s play but not understanding the basic concepts of SQL is an uphill battle for even the simplest CRUD applications, even if SQL isn’t used

Lingua francas such as C/SQL/HTTP in computing are perhaps even more important as paradigms of thought rather than tools to be used

Especially as we seem to year on year move higher up the tech stack for the foundations to greenfield projects

SQL is bifurcating, becasue it's kind of two things.

Good SQL is becoming more specialized for dev. No longer are the DBAs tutoring the devs to avoid inefficiencies. Now, the DEs handle really complex pipelining, and get really good.

But SQL isn't just used to "build" - it's also used to "access." And that's less specialized. The PM, BA, etc can all use ChatGPT/Stack Overflow and get things done quickly. And it's a virtuous cycle: the more people access data, the more the DEs are asked to clean things up, the easier it gets to query, and the more people try/succeed.

So SQL-for-development is increasingly niche, while SQL-for-access is increasingly common.

SQL is the opposite of teenage sex - everybody does it, nobody talks about it.

I find it really weird that a discussion on the state of SQL right now doesn't include any mention of Snowflake or dbt. SQL is _everywhere_ in the data engineering world right now.

I think some of this is due to SQL frameworks becoming a must-have. Which is kind of fine, I guess, because it's certainly nicer to have your objects coming back to you in some known format rather than manually massaging raw row responses.

But at the same time, a fairly basic SQL query I could bang out in two minutes becomes an exercise in acrobatics & research in how to do it right in this version of this particular framework.

Within the decade it's likely the vast majority of programming work will be done by AI, being considered too complex and dangerous to trust to human beings. So all such skills and knowledge, including SQL, will become equally niche as job opportunities disappear and schools abandon CS for prompt engineering and AI driven education. We're not there yet but it's going to happen.

Yes. I’d consider myself an “intermediate” DBA but I can command an insane salary and I have people tripping over themseves to hire me.

Nobody cares about SQL until it takes 15 seconds to load your user facing login dashboard.

You can do a lot with MongoDB, and maybe it does everything you need but you can do a lot more with a full-fledged RDBMS such as PostgreSQL.

Since MongoDB is so cheap, quick, and easy to get going my team tends to make that the default option. In many cases we've been able to run for years on MongoDB. In other cases, the needs changed such that we need updated DB capabilities and so we migrated that data from MongoDB into PostgreSQL.

That strategy has been working out really well for us.

As far as SQL itself becoming a niche skill, optimized SQL has always been a niche skill, and many data scientists don't have this skill. It seems many applications have that one "killer query" that simply takes too long to execute. A DBA will be able to optimize that query for you. They may have to create new indices and who-knows-what to optimize the query and almost always they use engine-specific SQL to optimize the query. You need people on-hand who can optimize SQL queries for whatever RDBMS you're using.

Betteridge's law of headlines: "Any headline that ends in a question mark can be answered by the word no."

I could just throw in a mention of Betteridge's law, but the question being asked is rather nuanced and is worth actually answering:

No, it is not. SQL is a way of thinking about your data, and querying against those assumptions and mistakes. If you think SQL is just a dumb row storage engine that you write garbage SQL for (ex: every single MySQL use I've ever seen), then you failed SQL 101.

A good proper use of a real SQL database (I'm obviously a huge pg and sqlite fan, but no hate towards DBA wizards that fight Oracle, DB2, and MSSQL every day at work) can shepherd your data, maintain its consistency, and conquer some of the weirdest footgun anti-patterns that keep showing up in business code that use 'modern' nosql databases.

To be absolutely fair, all technologies are a set of tradeoffs, and you have to understand them; if you don't, it will be fatal. Not today, probably not tomorrow, but in 3 years when your startup runway ran away. As technology advances, the meaning of those tradeoffs change; how I would answer some questions for people 10-15 years ago isn't how I'd answer them today.

Scalable DBs that make other tradeoffs are slowly being eaten by the march of technology. When someone picked up a mishmash of parts that don't fit, a nosql row store here, a nosql columnar store there, a dedicated event logger somewhere, all glued together with some message passing bus: its all people who don't understand the tradeoffs because they never learned the thought process behind how to use SQL effectively.

If you're asking "is SQL becoming a niche skill", you're really asking, "is thinking a niche skill?". Half of HN seems to absolutely adore the fake AI scams and willingly threw themselves down that flight of stairs, so I'm worried some people here really do think the answer to "thinking is a niche skill" is yes, and a dying one at that.

No. Those who refuse to learn it will be niche.

No

NoSQL application databases still a thing?

No data warehouse?

From a language/interface standpoint, there is no alternative.

SQL is not going anywhere. If anything it’s becoming more and more popular.

Every developer I’ve worked with who touches server-side code has eventually said some version of ”Bah damn these ORMs can we just write SQL?”.

And now with local-first there’s a growing trend of putting SQL on the client as well. Working directly with an in-memory (ish) SQLite and syncing to the server occasionally is super nice it turns out.

s/SQL/advanced SQL/ FTFY - SELECT/FROM/WHERE/GROUP BY for int/float/string is not hard or niche.

2c as DBA & researcher: advanced SQL is becoming irrelevant because LLMs are better at writing complex SQL than 99% of the available experts, who routinely botch NULL handling, datatype and index selection, performance optimization, etc. I see no reason why anybody should memorize the arcane syntax of SQL features like sliding window functions, exception handling in stored procedures, etc.

If this seems radical, consider that experts rely on query optimizers and EXPLAIN and people rely on the planner/optimizer for 99+% of queries. That was a radical position in the early 1980s.

LOL...

And then there is Golang. SQLC ( https://sqlc.dev ) becomes a source of truth not a sink... mix in some yaml and you have your json tags and validation mixed in.

Candidly good engineers are still using SQL...