Flutterby™! : Interprocess Standards

Next unread comment / Catchup all unread comments User Account Info | Logout | XML/Pilot/etc versions | Long version (with comments) | Weblog archives | Site Map | | Browse Topics

Interprocess Standards

2009-11-12 05:02:42.029536+00 by Dan Lyke 7 comments

An awesome semi-obvious idea that will take over the world and that I have no idea how to monetize.

Over on Twitter, Eric asked:

Watching Google Go video: http://bit.ly/3NUeUW Like what I see so far - but will there be library support?!?

I looked at Google's Go (as distinct from Francis McCabe's Go! programming language) and had a similar thought: I want to see what this language can do, I've been playing with Python and SDL recently, let's port that code to Go. Oh. Wait. No SDL bindings...

However, another realization: Processes are great ways to compartmentalize, and interprocess communication via pipes can be relatively cheap if it's implemented right. Why not make the GUI front-end one process in a language I'm familiar with that's fast and cheap, and write the back end that computes the (in this case) physics in Go? I could set up a pipe to pass data structures back and forth and... How should I serialize the data structures?

Back at Pixar I wrote a system for image output that had an API for image output. One side would say "the pixels are coming down the pipe as (signed|unsigned) (double|float|int32|int16|char) in (little|big)endian byte order" (the endianness was either defined, or native), the other would say "I want them in this format", I think quantization happened upstream anyway though there was probably some of that, and the library figured it out.

It could take multiple inputs, via network or pipe, the senders could even say "I'm sending this XY chunk of the image" and the receiver could say "buffer it and give it to me in scanline order." So you could be processing one portion of an image on a little-endian machine, another on a big-endian machine, and writing a driver for a format TGA was a matter of saying "give me scanlines in Red Green Blue Alpha order, throw out everything else", RLE compressing them and stuffing them to disk.

The flaw that I know of in it was that Tom Lokovic was, at the very time I was writing this, working on deep depth buffers, and my system only handled a fixed number of samples per pixel. But I think it got everything else mostly right (I'm sure Mark or the other Tom will drop in here to point out ways that it's been a beast, and given that the latter has used what I think is my code for a critique of coding style, I'm prepared for that).

The world needs a similar standard for interprocess calls. It's not XML, I've built a distributed system using it and that's way too expensive to parse. It might be JSON, but I rather think it's something binary. I also think it handles fairly complex data structures and has a notion of mirroring structures and persistence of objects at the other end.'

It is both a wire format and a set of libraries that provide an API that someone would recognize whether they were working in C, C++, C#, VisualBasic, Python, Ruby, Perl, Lua, or Google's Go.

And I think it'd be fun to write, but I don't have anyone to pay me to do it and I think it'll also need some names and/or applications big enough to push it so it becomes a standard. You listening, Google?

[ related topics: Interactive Drama Video Graphics Writing Pixar Python Monty Python Animation Web development Work, productivity and environment Law Perl Books Language Content Management broadband Software Engineering Open Source Boats ]

comments in descending chronological order (reverse):

#Comment Re: made: 2009-11-12 16:33:51.801788+00 by: Dan Lyke

BERT looks like JSON without the useful bits. Yes, Eric, it's probably DCOM, but COM has so many layers of obfuscation that I'm unable to dig past some of those truly hideous design decisions to find the goodness.

I'll look to ASN.1 and Protocol Buffers. Thanks!

#Comment Re: made: 2009-11-12 16:09:43.380421+00 by: markd

Yeah, protobufs were ringing in my ears on this one.

#Comment Re: made: 2009-11-12 15:12:51.607282+00 by: ebradway

Ah - I thought there was another language called Go!

Isn't what you describe essentially DCOM?

#Comment made: 2009-11-12 13:48:14.426873+00 by: spl

Ah yes, protocol buffers are another neat piece of technology. Nice one!

#Comment Re: made: 2009-11-12 13:47:01.921543+00 by: meuon

Bert looks clean and useful, I barely understood what Dan described, and I live in a world where the other end barely understands url encoded gets and posts or badly formatted XML (theirs). Sigh.

Does Google's Protocol Buffers work for what you want any?

#Comment BERT? made: 2009-11-12 10:44:09.983986+00 by: spl

BERT?

#Comment Re: made: 2009-11-12 10:42:46.404877+00 by: spc476

If you go a binary route, there's already a standard for that, and it's ASN.1 (shudder).