Flutterby™! : Syndication and clever hacks

Next unread comment / Catchup all unread comments User Account Info | Logout | XML/Pilot/etc versions | Long version (with comments) | Weblog archives | Site Map | | Browse Topics

Syndication and clever hacks

2004-01-05 04:12:05.146413+00 by Dan Lyke 12 comments

Scoble has been fanning the syndicator flames of late, and offers some reasons he thinks RSS is better than HTML (actually why he thinks syndicators are better than browsers, but we'll leave his syntax intact). Many of his reasons are based on using a horrid previous generation browser like IE[Wiki] that don't have tabbed browsing or "open every link in this bookmark folder", or that he reads sites with crappy web design ("don't do that, then"), but the one that struck a nerve is that syndicators only show you updated pages. I probably spend a few seconds per site seeing that it didn't update, and that's a few seconds we could probably save, and then there's a whole bunch of sites that are unfairly relegated to "Sporadic" or elevated to "Daily" that could just find their own place.

Well, a quick opera --help showed me that one can do opera -backgroundpage url to open a page in a new tab in the background. A few lines of Perl parsed my Opera bookmarks file, checked to see if the URL had a changed last-modified date or content length, and slapped up a new tab if it couldn't tell that things hadn't changed.

(Oh yeah, I should probably tie this into my last updated dates down there on the right somehow, 'cause the whole Weblogs.com really isn't working very well.)

This is totally not ready for prime-time, but if you're an Opera and Perl user (or other tabbed browser user willing to do a little coding) who'd like to help turn this into a real app, gimme a yelp.

[ related topics: Web development Dan's Life Content Management Weblogs Perl Open Source Software Engineering Graphic Design ]

comments in ascending chronological order (reverse):

#Comment Re: Syndication and clever hacks made: 2004-01-05 04:19:51.72208+00 by: John Anderson

I'd be prefectly willing to adapt it to mozilla (which will basically be changing your opera -backgroundpage url into mozilla -remote "openURL(url,new-tab)".

#Comment Re: Syndication and clever hacks made: 2004-01-06 03:42:54.908724+00 by: John Anderson [edit history]

Okay, I kludged on Dan's stuff a bit and now have a version that works for my local setup. Main problem I'm noticing is that not too many of the sites I read support either Last-Modified or Content-Length, so a bunch of stuff always gets loaded. That has me thinking that I need to make the input a bit more verbose, so I can tell the script "load this, but only if it looks like it has changed and/or it's been more than a week since you showed it to me" -- basically, the same logic I use when deciding what goes into the Daily Dose page on my site. Don't know when I'll get around to that, however; got a whole list of things I want to be working on...

#Comment Re: Syndication and clever hacks made: 2004-01-06 15:56:24.921911+00 by: Shawn [edit history]

syndicators only show you updated pages

This is exactly why I jumped on the RSS[Wiki] bandwagon. I don't have the cycles to spare waiting for pages to load just to see if something has been updated. The end result was that I just didn't visit web sites for news. Now, with RSS, I manage to keep a lot more informed than I used to be.

It's also the reason why I am a big fan of the News/What's New box on my own sites - a feature I also pushed over at Millerswork (adult content), although the implementation didn't quite turn out as I had intended.

John, would you be willing to share the code you've got so far, so maybe some of us can hack on it in bit too?

#Comment Re: Syndication and clever hacks made: 2004-01-07 00:56:37.192333+00 by: John Anderson

It's mostly Dan's code, actually -- I took his version and extended it to read my input format and use Firebird instead of Opera. I sent my mods back to him; if he wants to share the final product, that's fine with me. If I ever get around to extending it towards the direction I'd like, I'll try to post that here...

#Comment Re: Syndication and clever hacks made: 2004-01-09 04:08:41.010936+00 by: John Anderson

I finally realized today that you could have also been referring to the code I use to generate the DailyDose page. I've been intending to clean that up a bit and post it for some time now; I'll try to get to that this weekend and then drop a pointer here.

#Comment Re: Syndication and clever hacks made: 2004-01-10 23:28:11.199917+00 by: Shawn

No, I meant your extension to Dan's code. (Thanks, Dan :-) Don't expect any big returns from me at the moment on that front. I just thought it was an interesting project and worth poking at a bit if/when I can find some extra time. Unfortunately, I'm in the middle of a wipe-and-reinstall-Win2k circus at the moment. Nice how these things always happen the first week of school...

(This post coming to you live from Libranet. Speaking of which, I have lots of questions about how to do certain things in Debian [based distros]. Any advice on where to find good tutorials, documentation, etc.?)

#Comment Re: Syndication and clever hacks made: 2004-01-11 00:13:55.306065+00 by: Dan Lyke

Re: Debian... Ask here? And append "HOWTO" to all of your searches, there are lots of tutorials out there.

#Comment Re: Syndication and clever hacks made: 2004-01-12 15:38:31.31449+00 by: Shawn

Ask here?

I was concerned about hijacking Flutterby and turning it into a Shawn's Question Forum. I occasionally have questions about other things (along the lines of "has anyone seen/run across?") but always hesitate for the same reason. Not sure how appropriate such things would be here.

#Comment Re: Syndication and clever hacks made: 2004-01-12 18:16:53.581529+00 by: Dan Lyke

So maybe we need some filtering for the front page, or a secondary page somewhere for those more vested in the community? I used to have several levels of front page entry (specifically, "quickies", things I expected to have limited life, and "urls", things I expected to be long-term resources), maybe we need to split this up into topics (and it'd work better if I could get everyone to edit their topics after making a post).

#Comment Re: Syndication and clever hacks made: 2004-01-12 18:23:16.004885+00 by: Dan Lyke

Oh yeah: It occurs to me that this tool would be useful for sites that just have links to their latest update, like personal diaries. If they use a consistent dating scheme I could do a "if the HEAD for this day isn't 404, then open this window", and probably figure something intelligent for archives which don't have days, too.

All of this is leading to a GUI interface. Something I can copy-and-paste (or even drag-and-drop) URLs into, have it try to figure out the pattern of archives, offer control of other things. Is Gtk2-Perl acceptable to the two of you for that purpose?

#Comment Re: Syndication and clever hacks made: 2004-01-12 23:37:17.809174+00 by: John Anderson

You could follow in the steps of others and have an "Ask Flutterby" section...

Gtk2-Perl is fine by me; I've been needing a reason to force myself to learn a Perl GUI toolkit, and maybe this is it.

#Comment Re: Syndication and clever hacks made: 2004-01-13 19:31:38.878584+00 by: Shawn [edit history]

Dan; As a smaller, single utility I was curious and interested in poking at a code for producing a regular list of updated content. To be honest the larger, more encompassing project you seem to be talking about now doesn't catch my curiosity in the same way.

On top of that, my new year's work load is shaping up to be quite... um, solid (in a good way) is the best way to describe it. And to top it off, I'm back in the land of Serious Family Problems - although on an entirely different front, and in a way much more managable than last year. The upshot is that I currently don't expect to be able to do more than give the aforementioned code a quick glance-over. Trying to get involved in anything more is not advisable for me at the moment - that's a habitual trend I'm trying really hard to break myself of.

That said, my response to the GTK2-Perl question is pretty much exactly the same as John's.