Car dealerships revert to pens and paper after cyberattacks on software provider

This is what happens when a single player (CDK Global) has massive market share (90% when measured by number of vehicles sold [0] --- EDIT: this is actually the combined market share of CDK + Reynolds, who have a non-compete agreement ---). The entire industry becomes fragile to product issues and/or malicious attacks. Anti-trust is important.

[0] https://casetext.com/case/loop-llc-v-cdk-global-llc-in-re-de...

The AP article is terrible. It's basically CDK's press release.

There's more useful stuff on Reddit.[1][2][3]

- Some dealerships can still sell cars. Some can't.

- Parts and service are in worse shape than new vehicle sales because their inventory info is in the CDK system.

- The process by which new cars and parts are ordered and delivered to dealers is down.

- Many dealerships can sell what's on the lot and do repairs with parts in inventory, but the supply chain has stopped.

[1] https://www.reddit.com/r/partscounter/comments/1dmbmy7/the_c...

[2] https://www.reddit.com/r/serviceadvisors/comments/1djisf5/cd...

[3] https://www.reddit.com/r/askcarsales/comments/1dkf0xv/how_pr...

I'm reminded of the opening line of the second Mistborn novel:

> I write these words in steel, for anything not set in metal cannot be trusted.

The digital world feels increasingly dangerous and ephemeral. If I have something written with ink on paper I have certain guarantees:

I know that it's accessible even if the power is completely out. I know that it won't randomly get deleted if my hard drive suddenly dies. I know that the only way someone can read its contents are if they get physical access to that piece of paper. I know that it's not being automatically scanned by a platform provider to comply with government surveillance laws. I know that any alteration to the document would require physical access to it and would most likely leave a visible trace.

There are technologies that if applied would provide some or most of these guarantees and even provide increased safety, but I don't trust any existing SaaS providers to give me any of them out of the box, and the average person doesn't have the skill to string tools together to get them.

And so here we are: in 2024, anything not printed on paper still cannot be fully trusted.

I bought a new car this weekend and some of the symptoms of this were:

* dealer didn't have a good handle on what was available in stock on the lot * anything related to a title for a tradein was massively hampered. * appears USAA's auto insurance online add/replace vehicle on your policy is broken * everything was literally hand written, down to the sales contract

I would imagine the impact of this is in the tens of millions of dollars

Oh boy, has this made issues and opportunities for my automotive B2B SaaS. My product's customer are car dealerships and most of my customers are on CDK. This is definitely creating a void of data for them though fortunately my product won't fail without the expected DMS data. In terms of opportunity, dealers had paperwork that would be generated for the F/I department from CDK which I was able to port into my SaaS and prefill with data we can capture at the time of sale. All the praise is due for FOSS like `pdf-lib` that I was able to incorporate in a matter of a few hours. All the disdain for Adobe Acrobat Pro which fought me tooth and nail when converting their paperwork to fillable PDF forms ( - if you're the Product Manager on that then please get a UX audit of it!) Currently, we are supporting our CDK dealers in this critical time by getting them paperwork filled and printed to keep the car buying experience going.

Also, we're hiring! I could use another solid dev and a solid designer that wants to create a delightful experience out of an awkward wait time in dealership. Challenges abound but persistence prevails. Primary stack includes React, Remix, Xstate, Remotion, Framer motion, Python, Django, Graphql. Contact devs@zipdeal.com if interested and we can go from there.

For those looking for close coverage of this issue, "GuyDealership" is a good source.

https://x.com/GuyDealership https://www.dealershipguy.com

*not affiliated

Crazy how we are talking about this like a weather event, like it's just an unfortunate outage. Cyberattacked by whom? How? What vulnerabilities allowed the intrusion and what organizational processes created those vulnerabilities?

Naturally the people who know these answers are very busy today but hopefully we will hear more soon.

I'm trying to buy two cars right now, and this has put a big wrench in the process. The dealership can't tell me what's on their lot compared to what's on their Website, except by going out and searching.

I think every process needs a paper backup. There’s just no telling when something will be down.

If you run a SAAS you could offer this as a feature. Even hire data entry people to input transactions later if needed.

I know some (former) CDK alumni and I'm not surprised. Notorious for paying below market in engineering and IT. A place to cut your teeth but not to stay.

This illustrates the difference between a Business Continuity Plan and Disaster Recovery.

One of the reports of this I heard was on Marketplace the other day... and the part that made me chuckle was on cursive.

Pen and paper, but please no cursive: Widespread cyberattack sends car dealers back to 20th century - https://www.marketplace.org/2024/06/21/cdk-global-hack-softw...

> Car dealers are a pretty big part of this economy, doing some $1.2 trillion in sales last year, according to the National Automobile Dealers Association. In the wake of the CDK hack, there’s a new policy among the salespeople and mechanics at the Willis Automotive dealership in Des Moines, Iowa: no cursive.

> “We have a lot of staff members that are younger than 30 that I’m not sure have seen cursive in their life. So we try not to go the cursive route just to make sure everybody understands what’s going on,” said Jason Willis, CEO of the dealership.

I was involved in the building of the 911 system for my country. When doing that, we had to design it with this in mind.

If the system ever fell down, the operators had standard templated papers they would fill, and once the system would be back up they would add the case data back in.

For about ten years, I've theorized that we will see the rise of 'boutique' doctor, dentist, and law offices that keep only paper records for the sake of the privacy of their clients.

There are a bunch of problems that connection and automation bring with them, the rise of digital monocultures seems to be one of them.

I worked at one of these companies catering to a large network of dealerships. Quit after 1 month. The one company I worked at was outsourcing all or most of the development work with an onshore mostly as management and a couple of devs in the US.

It was a massive shit show.

- Tests? Useless and garbage. Only used to pump their code coverage reports to show to management

- Design? Non existent and a hacked together code base across a half dozen low bid contractors, probably junior engineers at best

- CI? Dedicated test environments? Nah, costs too much money. Just “use docker to test on local and deploy directly to prod”

- Documentation? Besides very basic instructions on how to deploy locally … good luck

- backup and recovery? Nah who has time for that. Just work on the features that {big client} wants

- code quality? I wish there was a “god” class, but in this code base they managed to create a universal class or “macro service”

Management was a mess. Just a mouthpiece for sales or executives. No backbone. No vision. Just “alright we just need to get through the quarter” mentality.

Some more updates from bleeping computer last week: https://news.ycombinator.com/item?id=40751754

This explains so much.

I am in the market for a new vehicle and did multiple test drives this weekend. One salesperson casually mentioned "our software is down and things are a little crazy". But other than a lot of waiting, things proceeded as I expected.

I figured that by 1000 this (Monday) morning I'd be fending off follow up texts and calls. Yet its now 1430 and not one outreach has appeared on my phone. This is not a complaint but it is a surprise compared with past buying experiences.

I don't want to come off a loving Tesla too much, they're not perfect, but this is another big area where they are transforming the status quo. So much of the FUD being thrown at them is by this lobby, they still can't sell cars direct to consumer in Texas! Car dealers, especially used cars, in the US are one of the most hated consumer experiences. Pricing is unclear, consumers are talked into buying things they don't want or need, to the point that most women I know won't buy a car without a man joining them. Additionally, they're resistant to EVs because it is eating into their service margin. I saw a brand new Porsche dealer going up recently with only two Level 2 charges. Probably a 10M project, and they have only a token level of concern for planning for future infrastructure. They're out of touch, short sighted, businesses with little alignment to the corporate manufacturing brand.

I haven't bought a new car in 8 years. Do they still do that thing with the pen and paper where they write down a bunch of numbers while crossing other numbers out randomly for like 10 minutes?

In an age where digital transformation is the backbone of business operations, a cyberattack can bring entire industries to a standstill. And it's a little bit frightening

They upgraded to the Centurion package: https://youtu.be/gbyXfLSqveM

Seems weird that to me that it make a front page on HN (from an European pov ; it's quite common)

Scary. I made a short story based on a massive AI attack that takes this up a notch: https://www.tiktok.com/@likearollingbot/video/73841103452012...

What is the reality something like this could happen to all google accounts, apple accounts, etc?

Someone needs to get fired over this. Probably an incompetent, non technical Product VP

I like old doctors who still use pen and paper for medical history. I don't like the idea of my medical records being stored online. Unfortunately this is the exception and not the rule nowadays. The same goes for imagery, X-Rays aside I don't remember any exam that didn't go straight into an online system.

I would bet that a developer knew of the security issues, but was vetoed by Product.

I know it's not directly related to this, but I remember one time I went to a dealership in my area that was apparently known for being particularly skeezy (unbeknownst to me before I went in). They got sued for some sort of fraud.

Anyways, I remember going in, and the guy pulling up a car in their system when the (too good to be true) car listed wasn't in stock.

He turned the computer screen around, and there was one of those (I think) fake "you have a virus" pop ups in the IE window. It was either that or a real alert. Either way - clearly incompetent with tech, and once I saw that I noped out of there fast.

Got took back to the stone ages

Another commercial for eliminating cash and moving to 100% digital currency!

In studying voting systems I’ve come to respect paper-based logs along the same quality parameters as databases. Resiliency, scalability, backup, readability, schema-flexibility , onboarding, power-efficiency are all very good with paper.

So many of us were quick to move paper-based systems (voting, orders, kiosks, parking meters, journals, requests, etc) over to computers – without accounting for the e2e cost and flakiness of the software-based system.

Related note: car stealerships are a relic of the past. There is no reason why a middleman should exist when buying a car. Consumers should have the option to purchase directly from the manufacturer.

I am personally not a fan of Tesla as a vehicle, but I admire the business model of buying directly from Tesla. Stealerships are a drain on the consumer. All that land they own can be used for productive economic activity or housing.

Chinese retaliation for EV tariffs?