At a certain point in a solution everything you do to optimize (“push”) in one area will cause a negative effect in a different area (“bulge”).
But this is a nice concrete example.
salomon812 17 hours ago [-]
Oh thank you for sharing! I had tried describing this earlier in my career and had call it "The complexity shell game" as in attempts to hide complexity only serve to move it around and to trick yourself it's gone. I would much rather use an existing standardized term, so I'm switching to Waterbed Theory, thank you!
sscaryterry 15 hours ago [-]
Its amazing how humans come up with the same things independently.
sigmoid10 16 hours ago [-]
I call it the conservation of effort. If you are able to save mental energy in one place, it necessarily has been spent in another.
andrewflnr 9 hours ago [-]
That's not entirely true though. There are sometimes true wins in things like finding better perspectives, e.g. models that are both simpler and more accurate, good abstractions that separate concerns cleanly, stuff like that.
sigmoid10 3 hours ago [-]
You still have to find these things though. Even if you didn't do it, someone, somewhere did spend all that mental energy. There are no shortcuts to saving effort in a closed environment. Same way you can't reduce total energy in a closed system. In fact these two are closely related.
jsrcout 17 hours ago [-]
My friend calls it the lump under the carpet. You can move it around, but you can never get rid of it.
baq 18 hours ago [-]
I like minimal irreducible inherent complexity of a problem or a system (waterbed or zero-point would work I guess) and then you can derive some form of a law of conservation of complexity… if you can measure it. You can definitely feel it, though.
edit: see also ‘No Silver Bullet’ and its essential vs accidental complexity; but don’t disregard Kolmogorov, either, even if it isn’t strictly engineering.
dylan604 15 hours ago [-]
I had heard of it in reference to auto sales being a balloon. If they show you a squeezed price, fees bulge on the other end of the deal.
ttoinou 18 hours ago [-]
It’s called constraints and tradeoffs yeah
aleksiy123 18 hours ago [-]
Sure, but I like how this feels more tactile and how things may bulge in unexpected way.
Like pushing clay around into the right shape of the problem while it’s fighting you.
Constraints and tradeoff feels like the simplified textbook model.
ambicapter 17 hours ago [-]
You're getting downvoted but I had the same thought. This sounds like engineering. I do believe there are some cases where pushing the problem to a different place is the right thing to do, sometimes that different place is contextually better suited to that task, so you do actually gain performance or capability from moving the problem around.
16 hours ago [-]
shayonj 18 hours ago [-]
v interesting. Thanks for sharing. I kept seeing this phenomenon of "no free lunches". TIL about the waterbed theory.
api 15 hours ago [-]
Something that comes to mind right away is: the cloud will eliminate the complexity of managing OSes and hardware.
Instead it moved it to the complexity of devops and complicated cloud APIs.
cpard 19 hours ago [-]
When I saw the title the first thing I thought of was schema on read versus schema on right when in data platforms.
You can make writes faster and part of that is by not dealing with schema resolution but you do push the work somewhere else there too.
I guess the same principles apply on many different levels, from when you write to the file system up to how you deal with conflicted data types during data ingestion.
rtibbles 19 hours ago [-]
The first thing I thought when seeing the title was "writing" with LLMs - writing quickly can easily just move the work onto your readers!
dozerly 18 hours ago [-]
I like this a lot, it’s true for more than DBs!
xixixao 19 hours ago [-]
2 angles I think DB designers don’t often think about:
1. Durability extends to the client. Replicated db might ack a write to client, but what if that ack gets lost on the way back over network? If client talks to the DB over simple HTTP, the write might first look like a failure. Can the client retry?
2. Human perception times are biological and don’t change much. But everything in the tech stack has gotten so so much faster since the 80s. Throughput matters, sure, but latency (relatively speaking), is much less of a constraint now than it was.
cyberpunk 18 hours ago [-]
1) you ack the ack, then ack the ack ack ;)
2) depends, i have a database that farks up a pretty complex distributed system when clients write from another az, latency really can be an issue for some workloads
mkeeter 18 hours ago [-]
Another fun failure mode in the same vein: if you are emulating an NVMe device and declare that a write operation is successful once bytes are in memory, you will quickly find yourself buffering arbitrarily large amounts of data as "persist to disk" (SSD speed) falls behind "acknowledge writes" (RAM speed). If you do not add backpressure to your system intentionally, it will be added for you – and you may not like where it's placed!
(This also happens at the SSD level: burst writes can be very fast as data is buffered in the SSD's own RAM, then performance steady-states at the true write speed once that's saturated)
martheen 10 hours ago [-]
UFS WriteBooster not only have a buffer that eventually need to be flushed, since it uses flash cells with pseudo-SLC behavior, the write amplification is even higher, to the point that they give it a special flag because using it all the time will just kill the media faster.
throwaway_95283 15 hours ago [-]
if you use the linux kernel that amount of ram is tunable, and write will cause backpressure on its own when this is exceeded.
https://en.wikipedia.org/wiki/Waterbed_theory
At a certain point in a solution everything you do to optimize (“push”) in one area will cause a negative effect in a different area (“bulge”).
But this is a nice concrete example.
edit: see also ‘No Silver Bullet’ and its essential vs accidental complexity; but don’t disregard Kolmogorov, either, even if it isn’t strictly engineering.
Like pushing clay around into the right shape of the problem while it’s fighting you.
Constraints and tradeoff feels like the simplified textbook model.
Instead it moved it to the complexity of devops and complicated cloud APIs.
You can make writes faster and part of that is by not dealing with schema resolution but you do push the work somewhere else there too.
I guess the same principles apply on many different levels, from when you write to the file system up to how you deal with conflicted data types during data ingestion.
1. Durability extends to the client. Replicated db might ack a write to client, but what if that ack gets lost on the way back over network? If client talks to the DB over simple HTTP, the write might first look like a failure. Can the client retry?
2. Human perception times are biological and don’t change much. But everything in the tech stack has gotten so so much faster since the 80s. Throughput matters, sure, but latency (relatively speaking), is much less of a constraint now than it was.
2) depends, i have a database that farks up a pretty complex distributed system when clients write from another az, latency really can be an issue for some workloads
(This also happens at the SSD level: burst writes can be very fast as data is buffered in the SSD's own RAM, then performance steady-states at the true write speed once that's saturated)