hwayne / awesome-cold-showers Goto Github PK

In the first half of Performance Matters (video) the speaker talks about how big effects memory layout can have on the results of performance benchmarks.

What's worse is that this is a variable that can fluctuate rather wildly for no obvious reason. I don't remember how well the research was made, so that would remain to be vetted.

"LATOZA, T. D., VENOLIA, G., AND DELINE, R. 2006.Maintaining mental models: a study of developer work habits. In Proc. of International Conference on Software Engineering. ACM, 492–501."

Found it as a cite in a different article, might potentially be a cold shower on documentation hype? Haven't looked to see if it's freely accessible.

https://blog.acolyer.org/2017/05/29/an-empirical-study-on-the-correctness-of-formally-verified-distributed-systems/

This is something I have domain knowledge in so can handle myself, mostly listing so I don't forget about it

Regarding the Cold Shower for "Scaling SQLite to 4M QPS on a Single Server", AWS is notoriously expensive compared to GCP, so I'm not very impressed with the claim and a bare metal VS GCP comparison would be much more relevant. (And I'm not sure bare metal would come out (significantly) ahead, then.)

http://sunnyday.mit.edu/papers.html#misc

Stochastic parrots is the obvious one, but there's others that are more pointed about specific flaws in, like, ChatGPT or GPT4

https://news.ycombinator.com/item?id=23739596

Fortunately it's not mainstream (so not hype?), but the amount of money that goes into teams to foster this initiative is not something to ignore.

https://people.well.com/user/doctorow/metacrap.htm

I won't write a PR, so this issue is just to drop a link in case someone steps up to do it.

This is just food for thought, sorry for creating an issue.

When reading the formatted markdown, the headers stick out. Since the headers are 1:1 to papers' names, they may not be "the" obvious representation of the hype.

There's also no room in the current format for associating multiple papers (with different approaches) to the single hype, if that's something that would be useful. I.e. possibly multiple sets of {shower,caveat,paper} per hype.

Would you name "integrating different runtimes" a fallacy?

I've taken a look into the paper (the chapter) and that's not great of a paper unfortunately: https://dev.to/gabrielfallen/a-cold-shower-for-a-cold-shower-237d

In my view, it didn't provide an "extensive literature review" even at the time, and still less than that now, 20 years later. And I don't see how it supports the claim that "formal methods are hard to learn, extremely expensive to apply, and often miss critical bugs".

Besides, I don't see anybody actually claiming "Formal Verification is a great way to write software. We should prove all of our code correct.", and it looks like nobody ever did. Thus it doesn't look like we need a cold shower on this one at all...

Internet going off for the night so getting this note iiiiiiiin

http://neverworkintheory.org/

~150 papers, most have limited scope and don't cover hype topics, but there might be some diamonds in there.

This book starts with guidance on how to be critical of software productivity research, and then goes on to present a bunch of studies where folks have tried to prove or disprove the benefits of different productivity enhancing ideas:

http://shop.oreilly.com/product/9780596808303.do

I read the Scalability entry, and it's a good post. I'd add a couple more caveats (discussed briefly in the article). Not all "big data" scalability problems are built around scaling out the number of CPU cores; I've worked in "big data" scaling on Spark before and often built out clusters for 10,000-100,000 times the dataset size of the one on McSherry's laptop. The calculus for these sorts of systems starts to tip back towards "the cluster's better" fairly quickly when you're also dealing with bus and memory bounds (do you have enough memory to hold the data you need in-memory, plus room to receive shuffles? Do you have a local network/NICs that are adequate to run those shuffles in reasonable time? Do you have enough striped fast storage?)

I'd add the 1G (still fairly large, sure) dataset size to the Shower part and explain that this is heavily a warning against overengineering and premature optimization.

Hype: Programming language X makes you more productive

Shower: An experiment with more than 600 professional programmers shows that (apart from assembly) programming language makes no difference.

Caveat: Was done in the 80s with Fortran, Cobol, C, Pascal.

Unfortunately, the book is not freely accessible. Maybe someone knows a paper version? Maybe even a more recent study?

https://dl.acm.org/citation.cfm?doid=3152284.3133876 (open access, PDF)

Archive.org Link instead: https://web.archive.org/web/20160410132241/http://www.cypherpunks.to/~peter/04_verif_techniques.pdf

https://blog.acolyer.org/

Several hundred papers along with extensive analysis. Across the entire spectrum of CS, so most are not cold showers.

Papers found should link to the actual paper in the title and have an additional note:

Further discussion at [The Morning Paper](link to acolyer's post)

I think the monorepo hype can result in bad architectural decisions if not properly evaluated.

This seems perfect to add to the list:

Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible?
A: Because Keynote Speakers Make Bad Life Decisions and Are Poor Role Models

https://www.youtube.com/watch?v=ajGX7odA87k

It's an excellent cold shower on the promises of generalised AI and Machine Learning and how they should not be let anywhere near either the internet or critical infrastructure, let alone both. It's also hilarious and really accessible.

I posted this comment with this article. Someone was getting too hyped up about a certain topic :) Let me know if I should submit a pull request or if you want to discuss further.

https://news.ycombinator.com/item?id=16036133

https://www.sigarch.org/the-unreasonable-ineffectiveness-of-machine-learning-in-computer-systems-research/