Flutterby™! : NNTP syndication

So, uhhh, hypothetically, if a tool were to turn up that magically turned weblogs into NNTP feeds (much like Flutterby), what should those NNTP feeds look like?

• All feeds in separate groups, or the same group? (Or separate groups crossposted into one big group, but that has implications on "Subject:" headers)
• What do the headers look like? Does the "Subject:" include a notion of the source weblog? How about the "From:"?
• Where should I put meta information. Right now I'm just using a separate class name in the HTML in the message, but should I put it in an "X-" header instead? Or as well?
• If you run a weblog, would you like having your content delivered this way?
• Is there a standard for retreiving and feeding back in comments?
• Anyone else out there got an NNTP server and want to experiment with feeds? I've got one other person I'm working with, and his server gets posts from mine fine, but we haven't figured out how to get his feeding back at me.

[ related topics: Weblogs Flutterby Meta Net Culture ]

comments in ascending chronological order (reverse):

#Comment Re: NNTP syndication made: 2003-08-07 20:34:34.421596+00 by: Larry Burton

1. I would think it would be something like weblog.flutterby, weblog.larryslog, etc.
2. With a weblog.specificweblog hierarchy, the subject is just the subject. From should be from the author. Shouldn't it?
3. If you continue with the HTML encoded version in the message I would put it both places.
4. Yes. Right now I'm generating a file with every post ferom my frontpage, each entry having an NNTP header as we discussed once before. As soon as I have time to get some plugins working properly I'll have that as a file with just my latest entry with an NNTP header. At that point it ought to be trivial to send it to you.
5. Yeah, the Comment API. It's even RESTful.
6. Do you know of an NNTP server I can install on a Windows box?

#Comment Re: NNTP syndication made: 2003-08-07 22:14:25.324595+00 by: Mark A. Hershberger [edit history]

Since I've been working on nnrss, I've had an idea or three about this sort of thing.

- weblog.flutterby would be good, but it'd be nice if people could self-categorize. If I want my weblog in your system, it'd be nice if I could pick the spot that it goes. Something like weblog.personal.my-mundane-life. That would keep it from being totally undigestable and make only very undigestable.

- Subject shouldn't differ from the title of the entry. From should contain the email of the poster (if available). Some X-Headers would be good for designating how to reply if there is another method.

- X-Headers are a good place for a lot of this information, especially if there are some widely understood headers already out there. Otherwise, just put it in the of the html in the body of the posting.

- Yes.

- CommentAPI & whatever Sam's group is working on for posting.

#Comment Re: NNTP syndication made: 2003-08-07 23:47:05.216189+00 by: Dan Lyke

So that sounds like I should use cross-posting and do a:

weblogs.everything

for people who want to let their scoring systems figure it out.

weblogs.name....

for individual entries by name for people who want to subscribe to a bunch of groups, we'd have "weblogs.name.flutterby" and "weblogs.name.larryslog" and such.

weblogs.category....

for weblogs which want to put themselves in some number of categories. Mark can have his "...my-mundane-life", we can have ".programming", ".sex" (and the usual hierarchy below that), etc.

weblogs.topic....

if we get some categorization going on, we can then do entries by topic. yes, this has the usual issues with topics, but if we let ourselves go broad rather than deep...

My guess is that much of the cross-posting will mainly be useful for people whose readers have good scoring systems.

Using a "From" address sounds fine, I think I'll need to do a little more intelligence with Flutterby's mail system to set up some /dev/null pipes for people who don't want to share their addresses.

And lots of "X-" stuff, again for the filtering.

#Comment Re: [Entry #6411] NNTP syndication made: 2003-08-08 07:16:03.185054+00 by: Shawn

On Thu, 7 Aug 2003 19:21:03 +0000 (UTC), Dan Lyke wrote:

> what should those NNTP feeds look like?

Separate groups. Among other things, this negates the "Subject:" issue in the second question.

>How about the "From:"?

I like the way it is now - name and e-mail address (with a "prefers anonymous" filler when no address is on file).

> * Where should I put meta information.

What meta information are you thinking about. I finally got a newsreader I'm happy with (so far) and I'm generally happy with what you're feeding me. I don't notice anything I want has gone missing.

> * Anyone else out there got an NNTP server and 
> want to experiment with feeds?

Too much other stuff on my plate at the moment (including playing with the Flutterby CMS code [http://www.flutterby.net]). Ask me again in about three months.

- Shawn

#Comment Re: NNTP syndication made: 2003-08-13 04:35:38.848625+00 by: Dan Lyke

So, Larry, per our private conversation I'm pretty close to having your blog syndicated with the method we discussed (specific class names), but I'm wondering if there's a way to do autodiscovery on the Comment API without retrieving every comment page individually.

#Comment Re: NNTP syndication made: 2003-08-13 14:29:09.125635+00 by: Larry Burton

Now that you've got my blog feeding into your newsfeeds I'm going to have to get on the stick with the changes in my comment system to accept comments through the Comment API. I'm currently using a hacked version of Six Apart's standalone Trackback script for trackbacks and comments on my blog. My idea is to write a PHP script that will act as a bridge between the Comment API and Trackback. Post the Comment data to my script and it will act on it appropriately, posting comments by pinging my trackback module with the comment.

I'll be reading the comment API spec pretty hard over the next few days. I'll let you know what it can do as far as Autodiscovery goes when and if I figure it out.

#Comment Re: NNTP syndication made: 2003-08-13 15:17:58.555197+00 by: Dan Lyke

Don't get ahead of yourself. I've got a script which teases apart your page and dumps it into NNTP, I still have to do a bunch of stuff regarding entry tracking and the like, and I'm not quite sure how to go about that. And I need to make sure I'm doing minimum network load on checking your page and figuring out what data I need to save for posterity.

I also want to integrate some of what I've learned back into my NNTP syndicator.

Aaaaaand... how are we going to go about feeding your comments into the NNTP stream? I'm not quite sure where this is going, there are various reasons I don't think that just moving the entire conversation over to NNTP is a good idea.

#Comment Re: [Entry #6411] Re: NNTP syndication made: 2003-08-13 16:46:03.50593+00 by: Larry Burton [edit history]

Dan Lyke wrote:

> Don't get ahead of yourself. I've got a script which teases apart your 
> page and dumps it into NNTP, I 
> still have to do a bunch of stuff regarding entry tracking and the like, 
> and I'm not quite sure how to go about that. And I need to make sure I'm 
> doing minimum network load on checking your page and figuring out what 
> data I need to save for posterity.

I'm not gettin ahead of myself. I want my comments to be accessable through the Comment API regardless of which direction your project takes you. If I get it implemented before you are ready then it will be ready for you when you get there. If you decide your project needs to go a different direction then I'll have a new project to work on. ;)

As far as network load goes, I check every fifteen minutes to see if I need to update my site with a new or changed entry. I republish my entire site every two hours regardless. I ping weblogs.com every time I have new or changed entries on my site... maybe. Weblogs.com doesn't *always* see a ping. I actually have new content only about three days out of the week unless I just happen to be in a posting mood. So if you checked weblogs.com for updates or just polled every couple of hours that shouldn't be too much of a strain.

Of course because of the fact that we are both sitting on the same class c network, I don't think it would hurt hitting me every five minutes but you might want to check with Meuon first. :-)

> Aaaaaand... how are we going to go about feeding your comments into the 
> NNTP stream? 

Well, since I don't get many comments on my site I hadn't worried to much about that until now. Do you want to pull or do you want me to push?

> I'm not quite sure where this is going,
> there are various reasons I don't think that just 
> moving the entire conversation over to NNTP is a good idea.
> 

-- Regards, Larry Burton

#Comment Re: [Entry #6411] Re: NNTP syndication made: 2003-08-13 21:06:03.376228+00 by: Dan Lyke

> I'm not gettin ahead of myself. I want my comments to be accessable
> through the Comment API regardless of which direction your project
> takes you.

Okay, so I'll try to implement the Comment API discovery mechanism in my RSS version, but we should probably do a "weblogentrycomments" class in the comments link. And perhaps a way to get the comments or check changes?

> I ping weblogs.com every time I have new or changed entries on my
> site... maybe. Weblogs.com doesn't *always* see a ping.

Weblogs.com seems increasingly flakey, it doesn't even seem to register changes in *.editthispage.com sites all the time.

> you checked weblogs.com for updates or just polled every couple of
> hours that shouldn't be too much of a strain.

I was going to use the LWP::UserAgent mirror function, which I think will do the right thing with conditional GETs and the appropriate HEADs.

> Of course because of the fact that we are both sitting on the same
> class c network, I don't think it would hurt hitting me every five
> minutes but you might want to check with Meuon first. :-)

I don't know about you, but I've only got a single 100baseT card in my server machine... [evil giggle].

> Well, since I don't get many comments on my site I hadn't worried to
> much about that until now. Do you want to pull or do you want me to
> push?

Push is obviously the right way, but part of what I'm trying to do is make it so that I can interact with my other favorite sites in NNTP. If you're using Perl, I just discovered MIME::Entity, which lets you do something like:

$mimemessage = MIME::Entity->build(Type => 'multipart/alternative',
                                    From => $identity,
                                    'Message-ID' => $messageid,
                                    Subject => $subject,
                                    Newsgroups => 'flutterby.larryslog');
$mimemessage->attach(Data => $textVersion);
$mimemessage->attach(Type => 'text/html',
                     Data => $htmlVersion);

Where $identity is something like "Dan Lyke <danlyke@flutterby.com>" and $messageid is along the lines of:

$messageid = '<larryslog_'.md5_hex($archivelink).'@mail.flutterby.com>';

So that I can later check against MD5s of the archive link to recreate the essage ID.

And then you just have a simple:

$nntp = Net::NNTP->new('mail.flutterby.com',Reader=>1);
$nntp->authinfo('yourflutterbyemailaddress','yourpassword');
$nntp->post($mimemessage->stringify());

To slap it into the newsgroup.

Once you get things debugged we'll figure out a real newsgroup hierarchy.

#Comment Re: NNTP syndication made: 2003-08-13 21:22:00.125149+00 by: Larry Burton

The trackback script I'm using form my comments is written in Perl. Let me see if I can find where to hack into it with what you are describing. It may be midweek *next* week before I get into it. I've got a few other irons in the fire right now.

Of course if you want me to send you that script and let *you* figure it out... ;-)

Nah, this looks like it should be simple enough and I need more Perl experience. I'll get this importing my comments into NNTP and then worry about getting comments from NNTP back into my system later.

#Comment Re: NNTP syndication made: 2003-08-13 21:26:58.024321+00 by: Dan Lyke

Oh yeah, you should put a "References" header in there that has the Message-ID as I derived it above where the $archivelink is the archive link of the original entry. That'll make threading work.

Speaking of trackback, that's next on my list.

#Comment Re: [Entry #6411] Re: NNTP syndication made: 2003-08-15 19:26:03.975907+00 by: Larry Burton

Dan Lyke wrote:

> Okay, so I'll try to implement the Comment API discovery mechanism in
> my RSS version, but we should probably do a "weblogentrycomments"
> class in the comments link. And perhaps a way to get the comments or
> check changes?

I just remembered that I'm producing an RSS feed for each entry that has comments. They can be found at http://www.dallasbay.net/tb_rss for whatever it's worth. The trackback ID I'm using is screwy, though. I need to fix that somehow.

-- Regards, Larry Burton