Flutterby™! : Two decades ago I delicately and

Next unread comment / Catchup all unread comments User Account Info | Logout | XML/Pilot/etc versions | Long version (with comments) | Weblog archives | Site Map | | Browse Topics

Two decades ago I delicately and

2022-11-14 05:55:03.268322+01 by Dan Lyke 5 comments

Two decades ago I delicately and carefully wrote a high performance threaded database core using pthreads. Now I spend a lot of time debugging race conditions because people just throw queues and timers around without engineering in threads from the start. Sigh.

[ related topics: Theater & Plays Databases ]

comments in ascending chronological order (reverse):

#Comment Re: Two decades ago I delicately and made: 2022-11-14 19:48:16.865439+01 by: markd

The new Swift "Structured concurrency" async/await syntax helps out some with that. But I noticed that that makes it easy to add an extremely expensive operation in the middle of something. "it's really cool I can trivially add a round-trip to the dropbox API to get some missing metadata, but I'm trivially adding a round- trip to a slow API..."

#Comment Re: Two decades ago I delicately and made: 2022-11-17 00:06:26.687389+01 by: Dan Lyke

Yeah, the problems I seem to be running up against, and sometimes it's hard to debug, are "these things that take no time are thrown off into a background thread, but this thing that round-trips via JavaScript in a web view has to happen on main and kaboom.

The other place I've run into it is a C++ situation I haven't debugged yet where Qt is forking processes for audio decoding and there's some other deadlock.

In both cases, I think having a threading and process strategy that got religiously adhered to, rather than "woohoo, we've got Promises, let's rock!", would make this debugging less... everything.

#Comment Re: Two decades ago I delicately and made: 2022-11-19 01:08:51.071795+01 by: TheSHAD0W

You'd think it would be easy, do things asynchronously until you can't; and turns out all the important stuff is tied together in a way that doesn't allow for it, and coding to prevent that is ridiculous.

In a couple of things I wound up doing multiprocessing instead of threading, which was a lot more resource intensive but didn't result in things locking up.

#Comment Re: Two decades ago I delicately and made: 2022-11-21 23:51:14.853924+01 by: Dan Lyke

Yeah, one of the big lessons back in the days when I was writing in pthreads was that at that point sockets were roughly the speed of memcpy, so processes were a smarter way to go.

Unfortunately, everyone wants to use "promises" and similar abstractions these days, and rather than letting us ponder good architectures it seems to be a lot of "hey, let's throw some shit at this wall and see what sticks", and...

Anyway, two separate projects, lots of pain.

#Comment Re: Two decades ago I delicately and made: 2022-11-22 00:13:08.179477+01 by: TheSHAD0W

Oh yeah, on the Bittorrent fork I managed, I wound up ripping out threading completely and running everything in a single process. Admittedly, python threads aren't real threads so there was no chance of a performance loss.

Add your own comment:

(If anyone ever actually uses Webmention/indie-action to post here, please email me)

Format with:

(You should probably use "Text" mode: URLs will be mostly recognized and linked, _underscore quoted_ text is looked up in a glossary, _underscore quoted_ (http://xyz.pdq) becomes a link, without the link in the parenthesis it becomes a <cite> tag. All <cite>ed text will point to the Flutterby knowledge base. Two enters (ie: a blank line) gets you a new paragraph, special treatment for paragraphs that are manually indented or start with "#" (as in "#include" or "#!/usr/bin/perl"), "/* " or ">" (as in a quoted message) or look like lists, or within a paragraph you can use a number of HTML tags:

p, img, br, hr, a, sub, sup, tt, i, b, h1, h2, h3, h4, h5, h6, cite, em, strong, code, samp, kbd, pre, blockquote, address, ol, dl, ul, dt, dd, li, dir, menu, table, tr, td, th

Comment policy

We will not edit your comments. However, we may delete your comments, or cause them to be hidden behind another link, if we feel they detract from the conversation. Commercial plugs are fine, if they are relevant to the conversation, and if you don't try to pretend to be a consumer. Annoying endorsements will be deleted if you're lucky, if you're not a whole bunch of people smarter and more articulate than you will ridicule you, and we will leave such ridicule in place.

Flutterby™ is a trademark claimed by

Dan Lyke
for the web publications at www.flutterby.com and www.flutterby.net.