A post about testing and then something about leiningen

I am writing this post in emacs tonight instead of the WordPress editor. I don’t know if it will make much difference, but I’ll keep it in markdown-mode for tonight.

I posted some questions about functional testing in rails. I am currently building what is most likely my most complicated rails application to date. I’ve been getting better at rails, but some of the things are still taking time to set in.

I’ve decided to really push the TDD aspect on this particular app. I am exploring using authlogic and declarative authorization to handle the security controls of the app. This is the kind of thing you want you want to be testing like crazy.

I’ve been having trouble with testing using loops to test access. One of the libraries uses a protected value that is only being set when a page is requested. I managed to log a user in, and that’s working quite well, but I’m having trouble logging that user out and a new user in within the same test case. I had this nice, elegant loop crafted that would allow me to establish a data structure that would allow me to control every aspect of the test. Alas, it would fail when it came time to change users. I put tons of debug code everywhere. I was hitting C-c ‘ r t <tab> <return> like crazy. I cleared everything that could even look like a bit of session. I couldn’t get Authorization.current_user to reflect the credentials of the current UserSession.

I do feel that I made up for it quite nicely. I managed to take about 3-4 lines of every test and pack it all into a single function that lets me specify the rules of the access check using a map. I’m quite happy with it

In clojure news. I’ve finally taken the plunge and have learned Leiningen. I was previously using Maven, so I’m not that hard of a sell. I like some of the ideas, but truthfully, I’m more interested in the polyglot maven project. I think it would be interesting if one or both of the projects could make it so they could try to read the other’s build files if available. So my first task was to take the clj-mql library, which I had just

checked out because I’m trying to learn more MQL, and convert the dependencies to maven. I hit a snag on something, and when I asked

about it in IRC, someone asked why I wasn’t just using lein.

So I threw out my pom.xml and made a project.clj for it.

Leiningen currently has a problem with it’s swank option which is causing src/ to not be on the path. You can always use the repl option

or the distasteful add-classpath option, but that will need to be fixed. Someone posted a horribly convoluted series of steps that I

attempted, but didn’t work for me.

On Writing. (a second blog post in the night)

Wow! The response has been impressive. I just posted a blog post, the first I’ve done in quite some time, and already that post is on the front page of duck1123.wordpress.com. I can’t believe. I can’t stand it. How dare that post take of the most coveted first post on that site that just so happens to be the one that takes it’s name from the online id I use. I must knock it down by publishing that one.

First I must sprinkle you with fairy dust and while you’re distracted, I go and grab a cigarette from my wife.

Ha Ha! She may only have two, but I have many. Not only that, but I found some Cherry Coke and the keyboard for my macbook.

I try to like the macbook. I will give it credit, it is not that bad, but for the past several years I have been pretty much exclusively an Ubuntu user. I like Ubuntu and I pretty much have it set up pretty well. I’ve run dual boot off and on, but recently it’s been I only rebooted on the weekends if I wanted to play sims or Toby wanted to play Portal and then I would immediately reboot back into linux because ironically I haven’t been able to get Windows 7 to share it’s NTFS drive with the mac using the windows share protocol. I have to log into linux to get that.

I’ve been becoming a big fan of XBMC lately. We don’t have a TV in our bedroom and I bring the macbook into the bedroom and hook it up to an old CRT monitor that we had extra and the iPod docking clock radio my dad bought me for christmas one year because he thought I had an iPod. I didn’t. I had a Creative Zen M:Video which I now can’t use anymore because Toby jammed the cord into it and bent the pins.

Anyway, XBMC (You know. I really wish the technology was (Pandora just asked me if I was still listening (my wife just messaged (can you tell I’m a lisp programmer) from the other room. ) I said yes) there to nest thoughts better in blog posts) No time for that tonight. I should write that in another post

I am the master of all (fictional) time.

I love freebase.com. It is a very aptly named site. Much like Wikipedia, where you can find yourself getting sucked from link to link, you could spend hours just clicking around. It becomes really dangerous when you consider how low the barrier to entry is to add new information. If you modify your account to keep you permanently in edit mode, then you see all these little blank spaces of where information is missing, and you just can’t help but fill it in. You don’t know what language that episode of Will and Grace that you are watching was in, well guess what? I do. Let me just click edit here, type out “English Language” pick the item off the list, enter and I’m done. Oh wait. This doesn’t list that this character was in this episode. Let me just fix that real quick, and… You can see how you can become lost.

My real focus, has of course, been in the fictional realm. I love the idea of being able to not only be able to look up any book, tv show, movie or comic book and not only see the actors and authors and artists associated with those works, but the character and places described by these works and the connections between them.

In my fleshing out of some of the fictional universes that are close to my heart, I decided that I wanted to be able to track who was killing who. I had to create two new types for the killers and the killed. Of course only fictional characters can kill other fictional characters, (act of the author notwithstanding)  so the killers and the killed are co-typed as fictional characters, but the killed characters are now dead. We need a type to track that information. So the deceased character is created, i need a way to track when they died. I can’t use real time. We know that Leto Atreides died in 10,191 AG., but when in AD years was the guild formed? In junior high, I calculated the events of Dune to be somewhere around 50,000 AD using a chronology in the non-canonical Dune Encyclopedia but I don’t really trust that for serious data entry.

So the Fictional Date/Time type was created. It sliced, it dices, it holds a slot for dates occurring in the Gregorian Calendar and a matched pair of slots for both the numerical date and the fictional calendar system that date is based on. Whether you want to know when lightning struck the clock tower or when Hayden Christensen managed to get himself inserted into Luke’s post-dramatic stress-induced acid trip you’re covered.

But wait, there’s more

If you act now, I’ll throw in a link to works of fiction portrayed in at no extra cost. No longer will you have to lose sleep trying to figure out exactly when the events of Care Bears II took place. This multi-purpose co-type works well in any situation

• Books
• TV Shows
• Plays
• Movies
• Comic Books
• Comic Strips
• Autobiographies of political figures

Well I am happy to announce that I was able to convince the powers that be at Freebase of the usefulness of the bottom layer in my quest to have a definitive list of every actor to kill someone in an episode of Monk. Fictional Date/Time, Fictional Calendar System, and Calendar System Directionality (forward and reverse) have been removed from my fictionaluniverse base and into the fictional_universe commons.

The deceased fictional character has many problems still owing both to a limitation of the levels of nesting that can occur with CVTs and the fact that because of DC comics, these damn superheroes just won’t stay dead. I’ll post more on that in another post.

Wave Client Protocol Whitepaper Brainstorm – Export 2009-10-24

This is a simple export of the conversation we’ve been having in the wave for client/server protocol development for the benefit of those in the community that do not have Wave access yet. I have applied a little bit of emacs-fu to make it a tad more presentable.

• [2009-10-14] Devyn (and wavegroupy@appspot.com, Angus Turner, …):Wave Client Protocol Whitepaper Brainstorm

  This wave is moderated to keep the responses down, do not be worried if your comments disappear. The result of each branch will summarized as part of the Whitepaper(IB)

  Related Waves:
  Interaction Flow: Wave Client Protocol Interaction flow
  Google Wave Conversation Model(Copied From Whitepaper): Google Wave Conversation Model
  Wave Definitions: Wave Definitions
  List Of Programming Languages Put Forward For Implementation: List Of Programmers
  Wave Data Model: Wave Data Model

  Requirements

  Functionality
  Handshaking
  Capabilities

  MUST have
  XML Schema?
  Operation Schemas (adding,OTing, etc)
  Recovery
  Persistent Changes / Temporary Storage
  Annotations (standard names as well? or just provisions)
  Push-based protocol. (must not rely on polling as the only means of fetching updates)
  Authentication agnostic (use OAuth in examples?)

  SHOULD have
  XMPP-based transport
  Compatible with BOSH / HTTP transports.
  Meta-Wavelets (Unread/Read Blips should be standardised)
  Bandwidth Throttling (blip by blip, character by character, word by word)
  [2009-10-14] * [2009-10-14] Andrew Hyatt:
  This probably isn’t necessary – just enable chunked messages.

• [2009-10-14] Daniel E. Renfer:Is this something built into the XMPP spec?
• [2009-10-14] Andrew Hyatt:Not that I know of, however I would be suprised if the XMPP spec would limit us to a character at a time. By making chunks variably sized, each side gets to decide.
• [2009-10-14] Daniel E. Renfer:How would it decide? Would this be something that is communicated, determined automatically by latency or platform capabilities?
• [2009-10-14] Andrew Hyatt:Probably any of those things (except for communication of diff size, which I don’t see happening). From the protocol’s point of view, it doesn’t matter.
• [2009-10-14] Daniel E. Renfer:There should be 3 states of update for a connection. Update Incremental, Update Commit, and Update Never.

  Update Incremental is what we’re now used to seeing. The rates are determined by load.
  Update Commit sends changes only “when done has been pressed” (for low capability clients)
  Update Never will not receive updates. The client must poll for changes manually. This is the default model for non-BOSH HTTP connections.

  • [2009-10-14] Daniel E. Renfer:There is also the possibility of a 4th state. Update Summary. In this state, the client would receive notification of changes to waves, but not the content.

    I’m not convinced of the need for this one yet, however.

  • [2009-10-14] Devyn:It could be useful, I guess. I’m sure someone could find a good use for it.

    Maybe for new message count-type things?

  • [2009-10-14] Daniel E. Renfer:That’s what I was thinking it would be used for. I should write out my idea for subscription views. Maybe others can tell me if the 4th state is wanted
    • [2009-10-17] Justin du coeur:+1 on this – definitely needed so that Summary views can be implemented. The existing Wave client demonstrates the need here: a user should be able to monitor a lot of Waves simultaneously. IMO it’s dead-critical for a good UI.
  • [2009-10-14] Devyn:Devyn: +1
  • [2009-10-14] Devyn:I’m going to hopefully write a little script for my custom UI… that will have a little count for Conky. Hopefully in Ruby.
  • [2009-10-14] Daniel E. Renfer:I would love my recent waves to show up in that empty spot I have at the bottom of my conky output.

    A conky script is a perfect example of the need to support dumb, Update Never, HTTP access. You would be able to define a view for exactly what you would like to have displayed and have those updates be retrieved as an Atom feed. There’s no need to have a full XMPP client for something like this.

    • [2009-10-14] joe.d.developer@googlewave.com:I agree, being able to get basic info from a curl / wget script is pretty handy
    • [2009-10-14] Devyn:Still, I don’t think Wave servers should have to start up an HTTP server.. I guess they can if they want to, but this isn’t the right place for it.
    • [2009-10-14] Daniel E. Renfer:http bindings would be an optional interface. The benefits of basing the model around Atompub would be that you only have to work out that model once, and it’ll work for these or future formats.
    • [2009-10-14] Devyn:Yeah… I agree, HTTP should be optional. That’s more of an application-specific API, not something that needs to be standardized.

      Good to note for the future, if anyone asks.

    • [2009-10-14] William Edney:Yea, exactly. If the protocol rides on top of XMPP, then you get the whole XMPP/BOSH thing ‘for free’ and the HTTP transport is happening at a lower-layer which you don’t have to worry about.
• [2009-10-14] Devyn:Ah, so like “live typing/etc.” and “after hitting done”, and “nothing” etc.
  RESTful design. (operations in terms of CRUD operations on resource)
  Distinct URI for every element of wave. (blips, wavelets, etc.)

  MAY have
  PGP signing
  Support for alternate serializations. (json, yaml, etc.)

  Actions

  Wave
  create
  add wavelet
  read
  all history [a summary]
  fetch changes (requires index number, as given by “all history”)

  Wavelet
  create
  add participant
  remove participant (no consensus on permissions, but at the present only bots can be removed)
  add blip
  read

• [2009-10-19] Michael Benz:we should also beable to create read only waves. for instance, at the moment, im creating a blog robot. and it would be nice, not have have people edit my blogs. it could be done by adding a participant (like public@a.google.com
• [2009-10-19] Daniel E. Renfer:Personally, I would like to see full ACL control built into the spec. Each wavelet is able to specify that all blips must be locked, unlocked, or no preference.

  Each blip has individual r/w controls with the ability to set inherit. The privledges of users in the wavelet are persisted to the blip. This would be the default for blips so that anyone with access to the wavelet, has access to the blip.

• [2009-10-19] Justin du coeur:In principle I agree, although I wonder if it’s overkill. I’m not coming up with many obvious use cases for quite that level of control. But yes, the general solution does mean we don’t have to worry about the edge cases…
• [2009-10-20] dalefrancis88@googlewave.com:Yes but i think this would be more a client side thing wouldnt it? what happens if a client is using a different protocol that doesnt have this finctionality?
• [2009-10-20] Justin du coeur:Hmm. Offhand, it seems to me that ACLs matter most to the server. I mean, all it really tells the client is whether they can try to edit/delete/whatever – the most relevant bit is whether the server accepts that change, and applies it to the official state of the blip.

  So if the client doesn’t have ACL support, and the server does, then the client might allow the user to apparently perform an operation that the server then rejects. Which implies that the server has some general mechanism for rejecting changes. (I assume that’s in the protocol somewhere, but haven’t gotten to that yet…)

  Blip
  create
  modify
  add contributor

• [2009-10-14] Daniel E. Renfer:Another question is if I create a blip, you change it, and then I revert or otherwise modify the content so that you no longer have any changes that contributed to this final state, are you still considered a contributor to that blip. Saying yes is easy, but not very accurate. Saying no would require some sort of “blame” operation where the contributions to a blip are clearly identified.
  • [2009-10-14]Monika Adamczyk (and Angus Turner):
    Given that waves tracks any change, I would lean to saying ‘yes’ because even if there was no actual content change, the blip was touched and perhaps the change was already pushed to the server before it was removed in the client. Sort of similar to opening a file adding content and then removing it.
    • [2009-10-14] Daniel E. Renfer:My comment didn’t concern what is sent and stored on the server, but only what data is sent when the client asks the server “give me the list of contributors whose contributions make up the current state of this blip” and then there would be a seperate operation for “give me all users that have ever made a edit in the history of this blip.”
    • [2009-10-14] Monika Adamczyk:I am not sure I understand difference and purpose of these two operations.
    • [2009-10-14] Daniel E. Renfer:suppose you have a blip (generally an IB) that has been edited many times by many people. You could have content that was added by one contributor early on that has been completely replaced by later editors. The first operation would list only the contributors that have added content still visible in the current state of the blip, while the other would list all of the people that have ever edited that blip for it’s entire lifespan.

      Consider commands such as `git blame` that show who has written what lines of a program. (except that wave isn’t line based)

    • [2009-10-14] Monika Adamczyk:If I want to see the history of a blip, then I simply replay the wave and the issues of blame is moot. The history will tell me who and which order made changes.
    • [2009-10-14] Daniel E. Renfer:The only real problem I see with that is it would not be possible to determine who did what to a blip without loading the full history of that blip. If we all agree that that is an acceptable loss, then this sort of advanced display could be handled entirely client-side and you can forget I said anything.
    • [2009-10-14] David Hubbard:I’d rather see it a part of the client-server protocol, myself, but if not, it could be an extension, and thus a good test of the protocol’s extendability.
      • [2009-10-14] Daniel E. Renfer:A spec for searching and indexing waves is a natural extension of what we’re doing here. We’re just not ready for it yet.
    • [2009-10-14] David Hubbard:As an extension, I’d hope it ends up being as good as if it were really in the protocol.

      That would simplify the initial protocol, for what it’s worth.

    • [2009-10-14] Angus Turner:I think that would be the way to go, create a protocol that can be extended easily, and thus have a simple base protocol
    • [2009-10-14] Daniel E. Renfer:We should reference the OpenSearch protocol as a basis for integrated wave searching. (adding link to reference section)
    • [2009-10-14] Angus Turner:ok, i’ll take a read

      Note that this implementation of client doesn’t have ‘Cancel’ button.
      The only time a change might be not tracked if client modifies blip in ‘Draft’ mode, although it is not clear to me what happens when someone wants to back out of ‘Draft’ based change (in this case having ‘Cancel’ button seems necessary.

  • [2009-10-14] Daniel E. Renfer:I would assume that as long as no changes have been made to the content before the “draft-mode” is enabled, then the server has no idea that a change is being worked on and would have no way to track it. It’s as if you wrote out a reply while offline, and then discarded that reply before syncing.
  • [2009-10-14] Monika Adamczyk:That would also be my assumption how this should be implemented but how do you discard the reply offline if you have no ‘Cancel’ option. I assume the client detects every keystroke in the blip as ‘blip content change’ event because I have seen robots to be notified about this type of event on every keystroke.
  • [2009-10-14] Daniel E. Renfer:A client must record all changes in a queue of unsynched commits. It is up to the client to decide how or when those commits are pushed out. Google wave pushes those changes out as soon as possible. That is why we are able to see the stream of keystrokes typed. (assuming the user’s client/webpage has a reliable connection to the wave server.)

    An advanced client would be able to hold off on sending out those changes and allow the user to review, modify, and delete changes from their queue before allowing the client to send them off. Discarding an edit would simply be removing all edits related to that edit from the queue.

  • [2009-10-14] Monika Adamczyk:Isn’t this what the Draft mode is for? Even this client intends to support it at a later time.
    I mean, I would imagine you could have a client that always waits for user to approve changes and push them, but that’s not different from any other IM client. The novelty and beauty of wave is that things are meant to happen in realtime.
  • [2009-10-18] Justin du coeur:That’s part of it, but I wouldn’t oversell the character-by-character aspect. There’s a lot going on here, and that’s just one component. Even without that, you still have:

    Threading
    Semi-realtime (that is, conversations that can progress quickly in realtime, pause, and then are easy to pick up as other people come online – character-by-character isn’t the only aspect of “realtime” here)
    Topical split (that is, conversations are described by their purpose, not just the people)

    And so on.

    Personally, I consider the jury still out on whether character-by-character is a good idea for initial replies – I’d like to experiment with seeing edits happen in realtime, but replies get sent out whole. (Although this would require some kind of “Joe is writing…” protocol to indicate that a reply is pending.) Even without that aspect, there is a massive difference between the Wave concepts and what you generally see in IM…

  • [2009-10-18] Daniel E. Renfer:XEP-0085 should take care of that if needed.
  • [2009-10-18] Justin du coeur:Yes, that seems reasonable, so long as we’re using the <thread> mechanism to identify particular wavelets. (I’m still catching up with the protocol definition so far, so I haven’t yet seen whether that’s the case, but it makes sense.)
  • [2009-10-18] Daniel E. Renfer:Truthfully, I don’t think the <thread> mechanism of XMPP will be as useful to us. IIRC that allows replies to be grouped to a thread of conversation with the same client. In the wave case, we’re not connecting to another user in the same way as we do in XMPP. We’re connecting to our personal wave server which is managing our connection to the wavesphere on our behalf.

    This is one of the reasons I am pushing for using Atompub as the basis as it will allow us to use the atom threading extension to track which blip another blip is in reply to..

  • [2009-10-18] Justin du coeur:Hmm. I’m not quite making the “personal wave server” assumption – IMO, we’re likely to wind up with a mixed ecosystem of personal and commercial servers. But that probably doesn’t have much effect on the architecture.

    As for atom threading, I’m open to that (need to read into it), but note that XEP-0085 is written in terms of traditional XMPP threads; at this point, it’s not quite obvious to me how it works if you are identifying threads in other ways. May become clearer after I read into the atom stuff…

  • [2009-10-18] Daniel E. Renfer:By personal, I mean “per-domain”
  • [2009-10-18] Justin du coeur:Ah, okay. We’ll eventually need to grapple with that sort of terminology.
    • [2009-10-18] Daniel E. Renfer:I still believe my point remains valid. When I communicate with you, my (hypothetical) wave client is not sending xmpp messages to you, but sending it to the wave component of my xmpp server, which is analyzing my changes, applying them to any other changes it needs to do, and submits it you this waves originating server.

      Those changes are then analyzed, sent out to your domain’s xmpp server, and then sent on to you.

      While it would be possible to build the threading mechanism off of XMPP threading. I feel it would be very brittle and wouldn’t hold up to people communicating with a wave server by means other than XMPP. It’s best to make whatever we need to identify threading part of the content of the message, not part of the envelope.

    • [2009-10-18] Justin du coeur:Oh, absolutely – we’re really not disagreeing in any significant way here. I’d actually been wondering about how we were going to identify wavelets and blips, because it clearly needs to be some kind of robust distributed identifier.

      I’m still wrapping my head around the atom thing, since I think of the two specs as fairly unrelated (I associate atom with RSS). So I’ll need to take a look at an example or two of how you expect to integrate the ideas. But I’m entirely content to believe that there’s a reasonable way to use them together.

    • [2009-10-18] Justin du coeur:Okay – I’ve skimmed the atom spec, and taken a fresh glance at the Protocol Interaction flow wave. I’m not seeing any concrete examples yet, so a bit of question-and-answer, so I can get straight what you’re thinking here.

      Is the notion that we treat each blip like an <entry> in the atom spec? With a unique identifier under the wavelet, and use the <thr:in-reply-to> syntax to identify the relationships between blips?

      Basically, I’m assuming that you’re not talking about using atom per se here, so much as lifting the relevant bits of XML syntax where it seems to be useful to us. Is that correct?

    • [2009-10-18] Daniel E. Renfer:That’s basically what I was thinking. Each blip is linked to it’s parent blip via the in-reply-to element.

      I think the relevant bits are pretty much the entire atom spec though. Entries do not need to be wrapped in a feed element and many of the standard optional fields in an Atom entry wouldn’t make much sense in a Wave context and would be simply ignored or omitted. (they should still be preserved when present to allow for further extension)

    • [2009-10-18] Justin du coeur:Makes sense. Are you envisioning that this stuff is transmitted as metadata? I’d been assuming that this kind of relationship information would be sent as part of the blip, but the atom way of doing business seems to be more about separating the content from the metadata about it.
    • [2009-10-18] Daniel E. Renfer:Maybe I’m misunderstanding when you say metadata. The linking from blip-to-blip would be part of the atom representation of the blip, but would not be considered the content of the atom entry.

      I need to re-read the atom specs, but I think it may be possible to only need to send that data if you are creating or moving relationships between blips. It would still show up for full representations of a blip at a given point in time, but again would probably only need to be sent when it changes for incremental mode.

    • [2009-10-18] Justin du coeur:The way to think about my question is like this. Atom is, I gather, heavily about metadata. The atom description of a feed entry isn’t the entry itself – that comes off the Web. Rather, it’s a pointer to the entry, with a lot of information about it. That pointer can contain the entire entry, but typically doesn’t: more often, it’ll have a summary and a bunch of info about the entry.

      So I’m trying to map this to blips in my mind, and it raises the question of where the atom info and the blip info live in the XMPP stanzas. Most specifically, is the atom info inside the blip XML, or parallel to it? The atom way of doing business would suggest the latter, which hadn’t been the way I’d originally assumed, but might have a lot going for it.

    • [2009-10-18] Daniel E. Renfer:I feel that a lot of your impressions of atom are based on how it’s actually used and not on good atompub implementations. Take a look at any of the Google GData api’s for an example of how atompub can be used.

      I picture like this: You have an XMPP stanza, inside that is an atom document that describes all of the characteristics of the blip, and inside that you have whatever xml or plain text content you still need to describe the actual content (or the changesets) of that blip.

    • [2009-10-18] Justin du coeur:Okay, sounds reasonable. I clearly need to read into atom more properly – I’ve mainly got the layman’s impression of, “Kind of like RSS but better”, and from the sound of things there’s a lot more to it than that.
    • [2009-10-18] Daniel E. Renfer:The atompub protocol is basically a textbook example of how to do REST right. If you understand REST, it’s very easy.
    • [2009-10-18] Justin du coeur:Okay, I’ll read into that. I understand REST moderately well – that is, I know the theory, but haven’t had any good reasons to implement a serious RESTful system yet. (Closest I’ve come to that is building some very simple Rails apps.)
    • [2009-10-18] Justin du coeur:Is RFC 5023 an appropriate starting place for reading into atompub, or is there something else you’d recommend?
    • [2009-10-18] Daniel E. Renfer:That’s the definitive guide. If there is a better tutorial to gettingstarted, I don’t know it. It’s been a while since I actively tracked it’s development.

      If you’re into Java (or JVM languages) you might want to check out the Abdera library.

    • [2009-10-18] Justin du coeur:Okay, cool. I’m going to read the wikipedia page for a quick overview, then dive in. And I’ll look into Abdera – I’m a general languages geek, and pretty comfortable with all the major ones. (More Scala than Java these days, but that’s still JVM.)
    • [2009-10-18] Daniel E. Renfer:I’m a major Clojure geek lately, so I know how it is to love the JVM but hate Java.
    • [2009-10-18] Justin du coeur:Heh. Yeah, after spending most of a year trying to program without closures, I was ready to claw my eyes out. Hence my switch to Scala. (Clojure is interesting and cool, but Scala’s a little closer to how I think: I’m pretty OO to my bones, and I like Scala’s object-functional attitude.)
  • [2009-10-18] Daniel E. Renfer:I tend to think of per-domain issues as personal issues as I own my own domain and manage the services of that domain just for my own use. Obviously the Google, Facebook, and Yahoo servers are by no means “personal”
  • [2009-10-18] Justin du coeur:Makes sense. I’m coming at it from a somewhat different direction, since my interest here sprang from a commercial system I’ve been building as a garage project for the past couple of years. The inspiration there was mainly LJ, and the discussions over the past couple of years about federating LJ inform my mindset a lot here.

    So I tend to expect a range of services, from geeks like us who have our own servers (I’ve got a half-dozen domains myself, so I understand where you’re coming from), to giants like Facebook, to boutique servers in the middle that are likely to provide services for distinct communities. (That middle category is, to me, one of the most exciting.)

  • [2009-10-18] Daniel E. Renfer:I’m sure we’ll be able to expect to see many companies comming forth and offering to host wave servers for small buisnesses.
  • [2009-10-18] Justin du coeur:Yeah, definitely. Indeed, it occurred to me a few days ago that, if my current job were to go south, there’s a fine business plan lurking in that sort of thing. (Not just hosting, but enterprise installations as well.)
  • [2009-10-18] Daniel E. Renfer:Keep in mind that you’ll be competing with Google Apps and Microsoft Live Wave
  • [2009-10-18] Justin du coeur:I know enough companies that are hosted-phobic that there’s still clearly a business there. (Indeed, I was chatting with a sysadmin friend of mine about it the other day: her company specifically rejected Google as their mail server, because they were simply too queasy about the security.) It wouldn’t beat Google, but I suspect a lot of little services companies will wind up co-existing with it, providing in-house enterprise installation of Wave and related services.
  • [2009-10-18] Daniel E. Renfer:I’m looking forward to the day that Wave federation is just a “sudo apt-get install wave-server wave-client-php” away.
  • [2009-10-18] Justin du coeur:Yaas. I could believe that we could get there in 12 months; I’m sure of it within 36.
  • [2009-10-21] Zsolt Dollenstein:
    if (debian)
    if (unstable)
    months <= 36
    else
    months > 60
    KTHXBAI

    There was a use case in the ACL thread about using wave to communicate with a group of subordinates. (in this case, teacher / students.) If the contributor list isthe list of any user that has ever attempted to make modifications to that blip, (even if that change was canceled or reverted) then that students icon would always be displayed next to the teacher’s icon, and you would never again be able to determine easily that all the content came from a single contributor.

    read
    delete

    Events

    Wave
    new Wavelet

    Wavelet
    contributor added
    contributor removed
    special?
    new blip

    Blip
    live update
    commit update
    header update (summary of changes, no content sent)
    deleted

    Extensions

    Search and Index Waves

    References

    Wave
    Expand On Federation Protocol Spec (http://www.waveprotocol.org/draft-protocol-specs/draft-protocol-spec) using a XMPP extension
    Wave Data Model: Wave Data Model
    Wave Definitions: Wave Definitions
    Conversation Model Spec Here: http://www.waveprotocol.org/draft-protocol-specs/wave-conversation-model (Google Wave Conversation Model )
    FedOne Client and Server Here: http://code.google.com/p/wave-protocol
    Whitepapers (including a Google C/S Protocol): www.waveprotocol.org/whitepapers
    Access Controls: http://www.waveprotocol.org/whitepapers/access-control
    Wave Client Protocol Interaction flow

    XMPP
    Put Good XMPP Protocol Links here: Good XMPP Protocol Links

• [2009-10-14] dalefrancis88@googlewave.com:I think we should kill of this wave and just put the links in here
• [2009-10-14] Angus Turner:mmm maybe, except there is no way to delete waves
• [2009-10-14] dalefrancis88@googlewave.com:not specifically deleting the wave just stop using it, saves people jumping around, the too many clicks theory
• [2009-10-15] Angus Turner:as W3C says: “depricate”
• [2009-10-15]dalefrancis88@googlewave.com (and Angus Turner):
  yeah but to what point, we already have other sections that are lists of links, maybe if it was a specific XMPP discussion that contained links it would be good
• [2009-10-18] Daniel E. Renfer:I was hoping that as the actual content of a whitepaper gets further developed, the list of links to good references could get moved to a different wave and the only links that would remain would be the specs we actually need to reference in the spec.

  Atom
  Atompub over XMPP: http://tools.ietf.org/html/draft-saintandre-atompub-notify-07
  OpenSearch: http://www.opensearch.org/

  Languages
  Implementation Languages: Python and/or Java (put your preferences here: List Of Programming [reference implementation] Languages )

• [2009-10-14] David Hubbard:Currently, the wave preview uses the initial blip (IB, the root of the conversation) as the header. Is this the header – so an update would be like any other blip update?
• [2009-10-14] Daniel E. Renfer:I didn’t write these points, but I’m assuming they refer to the 3 types of an update to a blip discussed elsewhere in this wave.

  Although, there really wouldn’t need to be a difference between the live and commit update. It’s simply a normal update. The only one that would be any different is the header or summary update, as that one would only contain minimal information.

• [2009-10-14] Angus Turner:could they be refering to a metadata blip (as in the headers in text files etc)
• [2009-10-14] Daniel E. Renfer:possible, but the “no content” note that got wrapped due to this thread indicates it was refering to a summary update.
• [2009-10-14] Angus Turner:like what displays in the inbox in the current client?
• [2009-10-14] Daniel E. Renfer:Right, you would want a summary update when you only want to be notified when, and the counts of changes in waves, but don’t care to know the actual content of those changes. (for bandwidth considerations)
• [2009-10-14] Angus Turner:like an email client that downloads the message content but leaves you to click download images and attachments, or on a mobile client where it downloads it as needed
• [2009-10-14] Daniel E. Renfer:I’m still not convinced if it is needed or not because in order to do things like show the live preview in my inbox, the client still needs to parse the full delta feed for that view.

  It would be best, however, for showing unread counts next to my saved searches (views).

• [2009-10-14] Angus Turner:or a growl notification when you get a new blip
• [2009-10-14] Daniel E. Renfer:I figure that a summary update would give you the type of action on the blip, and the blips id, but not much else. Most blips don’t have titles that you could send, but that could be also sent for new waves in a view
• [2009-10-14]Angus Turner (and me):
  so its decided it stays
• [2009-10-14] David Hubbard:Okay, maybe we can change how this is phrased to “Header update (summary of changes, no content sent)” ? Okay.
• [2009-10-14] Daniel E. Renfer:go ahead
• [2009-10-14] Angus Turner:how come its moved down here :\ i’m confused
• [2009-10-14] Daniel E. Renfer:This was an inline reply that lost it’s anchor. Now that I see that the conv wasn’t lost. I’ll delete my copy and paste job.
• [2009-10-14] Angus Turner:ahh ok, i still think the copy/paste job is good, its in the right place and we can delete the blips down here
  (maybe this is the way to prevent us hitting the 500 blip limit + finalizing discussions)
• [2009-10-13] Daniel E. Renfer:Every Wave User ID should be a XMPP jid and should share things such as presence and contact lists with it’s XMPP roster.
  • [2009-10-13] Daniel E. Renfer:Assuming that this is the case, we need to consider the user experience if a user logs into an IM application (ie. Pidgin) using their WaveID? (note: what is the official term for the id of a wave user)
  • [2009-10-13] Daniel E. Renfer:Basically, what we have to think about is that your standard XMPP chat application is going to try to display any message stanzas to the user. If we are sending a bunch of binary diff data, (I’m not familiar enough with OT to know what it actually looks like) then it will quickly annoy users that want to dual purpose thier JID.

    There was some discussion in one of the other threads about using iq messages for much of this, so that would aleiviate much of this. Does anybody who understands what was discussed in the other thread better than I care to respond?

  • [2009-10-14] Devyn:Hmm… I think Wave should attempt to determine the client… maybe have some kind of special metadata or something for a Wave client? I don’t know… I’m not that familiar with the internals of XMPP.
  • [2009-10-14] William Edney:XMPP allows the server (and other clients) to ‘discover’ things about a client (and other servers) via a mechanism called ‘service discovery’.
  • [2009-10-14] Daniel E. Renfer:If my idea for views is adopted, then the XMPP server wouldn’t begin to send wave updates to the client till the client announces itself to the server as being able to process wave updates and sets the update status of one of it’s views to a state other than “none”
• [2009-10-13] dalefrancis88@googlewave.com:So in which section in the IB do you think you would put your ideas?
• [2009-10-13] Daniel E. Renfer:I wasn’t sure. That’s why I chose to make this a blip. I figured we would get some discussion on this point before adding it to the IB.
• [2009-10-13] Tad Glines:I agree that the JID should be the same as the participant ID. As for the case where a non-wave client connects to a wave enabled XMPP server, the easiest way to deal with this a two stage process. In the first stage the client does service discovery on the server to locate the WAve JID (e.g. wave.example.com) and then makes a request to the wave component. In the second stage, the wave component does service discovery on the client using the JID in the initial request from the client (e.g. joe_user@example.com/wave_resource1234). This does two things; 1) prevent a non-wave client from ever getting wavey stuff, 2) allow the server to tailor its responses based on what it discovers about the client.
• [2009-10-13] Daniel E. Renfer:You know, there were some minor points there that I was just missing. I didn’t think about the fact that a wave server would almost always be done as a component.

  So would the Wave server show up like most transports do?

• [2009-10-13] James Purser:A wave server won’t necessarily be done as a component of a XMPP server. It may infact build that functionality into it. Also remember that wave servers may in fact be extensions of other communication packages (Email servers and so on).
• [2009-10-18] Justin du coeur:Just so. I’m vaguely toying with the notion of building a Scala-based server (the needs of the protocol we’re talking about suit Scala extremely well), and while that might be done as first writing a generic XMPP skeleton and then adding a component, it would actually be easier to start with something Wave-centric and then refactor out the general framework if and when somebody really cares…
• [2009-10-13] Tad Glines:The basic exchange I mentioned would be the same even if the wave server was also the XMPP server. The key is that a wave server would not just start spamming every XMPP client it knows about. It would wait until the client starts a wave conversation, then do client capability discovery (or accept whatever the client says about itself via some other method).
• [2009-10-13] Angus Turner:is the conversation model part of the C/S Protocol?
  • [2009-10-13] Daniel E. Renfer:I would say that, while not directly part of the protocol, the conversation model is a very important thing to reference. We need to have API hooks to access any of the relevant bits of that model.
• [2009-10-13] Rémy Sanchez:what do you call the conversation model ?
• [2009-10-13] Angus Turner:http://www.waveprotocol.org/draft-protocol-specs/wave-conversation-model
• [2009-10-13] Rémy Sanchez:oh that one ok…
  /me should stop asking stupid questions and read the whitepapers first
• [2009-10-13] Angus Turner:i’m working my way slowly through them…….
  it seems 5 minutes reading, 5 minutes break is good
• [2009-10-13] Devyn:Great work guys!
• [2009-10-13] William Edney:Yes – this is excellent. When I get some time, I’ll do some review. Thanks guys.
• [2009-10-13] Daniel E. Renfer:Does anyone know of a hosted version of the FedOne javadoc? It would be helpful to have that for reference when developing this spec.

  I generated a copy and could put it up somewhere, but I don’t really have a stable heb host at this point, so if someone has space to offer, I could send you the files.

  • [2009-10-17] Zsolt Dollenstein:There you go: http://zsol.web.elte.hu/wavespec/wavespec.html
    Have fun
  • [2009-10-18] Daniel E. Renfer:This is a good reference to have, but I was actually looking for a documentation of the java source code to the FedOne server. It’ll be handy to be able to reference the implementations of existing servers to know what functionality we are not yet supporting in this spec.
  • [2009-10-19] Zsolt Dollenstein:I can get that up as well when I get home today. stay tuned.
• [2009-10-14] Daniel E. Renfer:Views

  When a client connects to a server, it sends it’s requested update state for each of it’s saved views or any new ad-hoc ones that it wishes to track. A view is a fully specified and parameterized saved search. When changes happen on a server, the server compares those events against the users’ views. The server will then push out updates for any changes that match a user’s views depending on the update status.

  • [2009-10-14] Devyn:Sounds good
  • [2009-10-19] Michael Kleber:Is this really a reasonable thing to require of every server? Is there some cap on how much saved-search work or complexity the client can force the server to do on its behalf?
  • [2009-10-19] Daniel E. Renfer:Well, my intention was to reduce the work required to be done by the server by only requesting the waves that match a query. this way a server doesn’t have to send updates for the thousands of waves that match the spam view and not the inbox view. The server would only have to compare events against a clients active subscriptions.

    It would be perfectly reasonable to for a server to decide that it’s going to limit the number of active subscriptions a user is allowed to have at any one time.

    If you can think of a better way to control what updates the user is interested in, I’m glad to hear it.

  • [2009-10-19] Michael Kleber:The naive alternative is that you only see updates of the search you’re looking at – in the current UI, that means the one being shown in your “search results” pane on the left. When the user switches to a saved search, the client queries the server for results, then either subscribes for some limited period of time or polls periodically for updates.
  • [2009-10-19] Daniel E. Renfer:That’s exactly how I envisioned the views system working. The client would announce a subscription for any “mailbox windows” open. When the user closes one view. (inbox) and opens another view (By Me) the client changes the subscription on the inbox view to none (or summary if you want unread counts displayed) and sets the new view to a more active state. (incremental, but you could probably get away with commit for a mailbox view)

    When actual waves are opened for viewing, the client sets or unsets any relevant subscriptions at the wave-only level so the server is only sending information for the conversations the user is actually interested in.

  • [2009-10-19] Michael Kleber:Ah, great! Glad to see I was misunderstanding you almost as much as you were me below

    So what if a client wants to have a list of “saved searches”, like this UI does? Probably few of them will be active views at any moment, so most of them aren’t covered by what you wrote under Views above, right? Do you plan on a place to store these many inactive searches?

    (Obviously this could just be saved in a per-user private wave, in the same way I talked about for read-blip metadata below.)

  • [2009-10-19] Daniel E. Renfer:There are 2 types of views. Saved and Ad-hoc. Saved views would handle the case of the system defined sections you see in the navigation section as well as the saved searches. Switching to one of those saved searches would require simply changing the subscription state of that view.

    Ad-hoc views would be for the searches you type into the search bar. They would function exactly the same as any other view, except they are removed once the subscription is canceled.

    A server can choose to proactively filter updates into inactive views if it wishes, or it can do the search only when the view is activated. (implementation detail)

    • [2009-10-19] Justin du coeur:Assumption check: I’ve been figuring that, if the server is extending the system with additional semantics – specifically, specialized views – that these would be treated as “saved views” with predefined semantics.

      I’ve got a specific agenda here: I’m expecting to extend the system with integration into social networks, which would probably means “views” that correspond to externally defined communities in those networks. So I want to make sure I understand how that would fit into the design…

    • [2009-10-21] Zsolt Dollenstein:If what I think is anywhere near your idea, then yes, these things defined by the server (additionally to what Google’s current implementation defines) should be considered views. Whether they are saved or ad-hoc doesn’t really matter IMO, that should be a client-only notion.
  • [2009-10-19] Michael Kleber:Got it. So the possibility of proactive filtering does provide a reason for “saved searches” to be handled differently than arbitrary per-user metadata that a client might want the server to remember for it.
  • [2009-10-21] Zsolt Dollenstein:Don’t you guys think this will overcomplicate the protocol? I think the best solution is to let the client open multiple views at once. That should solve it.
  • [2009-10-21] Daniel E. Renfer:The client is free to open multiple views still. The client could very well to decide to open a subscription to every view they have access to. Subscribing to a new view wouldn’t cancel the subscription to any other views. If you want to stop receiving updates you would either have to send a update-none request, or send an unavailable presence. (other presence states, such as DND might have an effect on the flow of information as well)

    The servers will have to take care, however, that if an update appears in multiple views, that only a single notification is sent.

  • [2009-10-21] Justin du coeur:Also, we should keep layering in mind. I suspect that we’re quickly going to find that thinking in terms of “the protocol” is the wrong way to go. My guess is that we’re going to wind up with an entire suite of XEPs, not all of which will be mandatory.

    So I’m not too worried about overcomplicating the protocol – I’m more concerned about separating concerns carefully, so that bits can be separated from each other…

    Internally, these views could be considered as pubsub nodes. Every user has their own default view, (your inbox) and will be able to perform operations on their set of views. (add, remove, change update status, etc.)

  • [2009-10-18] Justin du coeur:Sounds good, but we’ll need some kind of server-based concept of a saved view, essentially as user metadata, so that thin clients can operate. Might be out of scope for this particular spec, but we ought to say so explicitly if so. (I suspect we’ll wind up with a list of “out of scope” topics that will eventually wind up generating their own specs.)
  • [2009-10-19] Michael Kleber:Let me just point out that there’s no need for this to get any special treatment from the server: the alternate approach is one where clients store this kind of per-user metadata in a private wave. If many clients end up agreeing on a format for that per-user metadata, we end up with a de facto “client-client protocol” – it makes a lot of sense for, e.g., the “read blips” metadata to go that route, for example.
    • [2009-10-19] Justin du coeur:Sure, that’s a possibility. Indeed, CommYou was just starting to go down that road: I was in the middle of designing a generalized metadata system a few weeks ago, expecting a fair amount of stuff to migrate into it.

      That does imply that we need a convention for how to look up a well-defined “kind” of private wave, though. Would name work? I dunno. The CommYou design was leaning towards what amounted to GUID identifiers for metadata types (but open, so that anybody could add a new type), to address questions like this. I haven’t noticed anything obviously like this in here yet, although I’m still reading in.

      (ETA: And I see that Daniel made the same point below – good, we all seem to be thinking in similar directions here. (As I desperately attempt to not use the word “wavelength” in conversation…))

  • [2009-10-19] Daniel E. Renfer:So you’re suggesting that read state should be stored in the client itself? Are you sure about that? That would mean that if I read a bunch of waves in Client A, when I switch to Client B they will all be unread again. That doesn’t seem like a very nice situation.

    Even if you had some sort of C/C protocol to sync read information, you would require both clients to be running and able to communicate with each other. My client at home would have no chance syncing with the client at my work. (behind a firewall and shut down for the evening.)

    It seems better that metadata for a user’s read state be stored on that user’s wave server. (to reduce the calls to originating servers when checking read state) Also, this mechanism for per-user metadata could be extended to things such as bookmarking and private tagging of blips as well as private notes about a wavelet/blip

  • [2009-10-19] Michael Kleber:No, sorry, I seem to have been completely unclear. I was the original person asking for metadata to be stored on the server, for which there wasn’t a mechanism in Elliott’s first straw-man proposal. The question is, should that state get some special consideration? – meaning both (1) a place in the client-server protocol and (2) standardization across clients.

    I’m just pointing out that you can decide “no” on (1) – that is, the C-S protocol can be silent on read-blip-state – but still get all the benefits as long as you get (2). The state is stored on my local wave server: but it is stored in a special wavelet, with private (=to user only) visibility. The server treats it like any other data, but the client knows to fetch that wavelet and use the data in it to mark blips as read/unread, and update it as appropriate.

    If many clients all used the same format of private wavelet to hold this data, then what we get effectively is a client-client protocol. But it’s all server-mediated; the clients are never communicating directly with one another, and thin clients might hold no such state at all. Multiple clients can share the state of the metadata blip, and update it using the full OT machinery.

  • [2009-10-19] Daniel E. Renfer:Yes. I misunderstood you. I agree with what you described, but I feel it’s probably worth specing out how to deal with private metadata. (which should just be stored in waves to make the processing easier.)
• [2009-10-14] Daniel E. Renfer:Atompub over XMPP?

  I’ve brought this issue up in other waves, and it didn’t generate the discussion I had hoped. How do you all feel about using Atompub and minimal Atom documents in the protocol.

  • [2009-10-14] Devyn:I’m fine with it… as long as most of the popular languages have support for Atom. Most do… but not all. I know Ruby, Python and Java do. Most likely languages like C as well. Haskell probably, and Lisp-variants likely. So, should be fine.

    (helpful tip: spewing out thoughts directly without further formatting is not a good idea, as I just did xD)
    Atom takes care of many of our issues. It gives us a well thought out means of tracking modification dates, unique identifiers, threaded documents, authors, extensibility, and RESTful architecture.

  • [2009-10-14] joe.d.developer@googlewave.com:Atom seems like a good match for a number of the things we need, much as you have outlined. Perhaps it would also make some of the ‘simple’ publishing mentioned earlier a non-issue to implement.
• [2009-10-14] David Hubbard:There is an interesting comment, in the fedone code:
  src/org/waveprotocol/wave/examples/fedone/waveserver/RemoteWaveletContainerImpl.java:129

  // TODO: only one request history should be pending at any one time?
  // We should derive a new one whenever the active one is finished,
  // based on the current state of pendingDeltas.
  federationProvider.requestHistory(waveletName, domain, expectedVersion, appliedAt, -1, new HistoryResponseListener() {

  This is the most useful when a federated host is getting back in sync. But it has a direct corollary in client code.

  It would be possible to define semantics for multiple “request history” messages to overlap, or it would be possible to re-define how history is tracked, instead of using the current version numbers. Using a great big lock to limit the number of possible history requests in flight would probably hurt scalability.

• [2009-10-14] Daniel E. Renfer:Would this be an issue with push-based XMPP? The server would track what updates need to be sent to a client based on the last state the client communicated and then would push out new events as they occur.

  This should only be an issue for HTTP-based pull requests, and even then it’s a client issue as the server will just send whatever it’s asked for. It’s up to the client to not get confused by overlapping messages.

• [2009-10-14] David Hubbard:Ok, yeah that works for the basic cases for a client. I think federated servers definitely need to all behave correctly for this case, but it’s the same as for the client, they just need to not get confused.

  The remote host that hosts the wave will push updates and the requestHistory() method is the equivalent of HTTP-based pull requests. fedone not defining it doesn’t mean we can’t come up with a sensible way of behaving.

  Like – if the federated host (or a client) is missing pieces of history, request them (pull). Otherwise, wait for new events (push).

  It would only be challenging for a host to keep up if a new event arrived out of order and the host had already sent a pull request. Events probably will always arrive in monotonic order of version number (so, version 3, then 4, then 5…) This question is relevant for clients, too – but clients are easier to deal with, like just shutting it down and logging in again.

• [2009-10-14] Daniel E. Renfer:I need to search through the whitepapers again, but I think a method already exists for determining if messages are missing with federation. (may just be the ack of <iq/>) That method could probably be applied to this case as well.

  regardless, the drawback of using a http pull backend (non-BOSH) is that the responsibility is on the client to make sure they have an up to date view of the system.

• [2009-10-14] David Hubbard:If the client is logging in from a computer (phone, whatever) that hasn’t been connected to the wave before (the user bought a new computer), they will not be in the same state as the server is expecting… Like TCP, ultimately the edge of the network has to have the smarts… IMHO
• [2009-10-14] David Hubbard:fedone determines that a pull request is necessary by looking at the version number. If the local repository has all the previous version numbers already, the update is applied. Otherwise, it is deferred and a pull request fetches the missing updates.

  This is only used (currently) when the remote host initially adds a participant on the local host. After that, the local host can keep track of the wave, and doesn’t need to pull again. Similarly, a client would only rarely make pull requests, and those would typically occur during the “initializing” stages.

  I am not aware of anything about this in the whitepapers.

  • [2009-10-14] Devyn:There’s a lot of intense discussion going on here… keep it up

    It helps, it really does… even if someone is completely wrong, sometimes interesting ideas come out of them…

    Oh, and I really wish there was a way to leave a wavelet… you can join, but not remove yourself… it should also be smart enough to get rid of a wavelet that no one is attached to.

    • [2009-10-14] David Hubbard:So leaving a wavelet is a way of expressing “I’m not going to participate anymore.”

      And up above, Daniel is suggesting a way of finding out the “git blame” – well, to use other words – the current participants in a wave based on actual content in blips, and the edit history.

      • [2009-10-14] Devyn:Yeah, I use Git a lot I’m a Rubyist. That would be neat.
  • [2009-10-14] Daniel E. Renfer:Google Wave should be more like git

    Of the two possibilities, I think the one that lets you express your intentions is more valuable in the short term.

• [2009-10-14] James Purser:OKay I’m going to stick this here in the hopes that it proves to be useful. I’m working on a Federation protocol walk through.

  http://wavingtheshiny.collaborynth.com.au/books/fedone-book/federation

• [2009-10-21] Daniel E. Renfer:Here’s a question for the group. If every changeset is given a unique xsd:dateTime timestamp by the originating server of a wave, and every edit to a blip is given a simple integer revision id, will that be enough to track and transmit the state of a wave for synchronization purposes. It seems like it should be enough to me, but I was wondering if anyone had any thoughts as to why that wouldn’t work.

  So if I need to fetch only the updates to a wave since I last signed in, I need only to transmit the timestamp of the most recent change that I have along with my subscription request.

  Also, keep in mind that these timestamps must be the time that the originating server actually committed the change to the wave, not any notions of time that either my client, or my federated server may have.

• [2009-10-21] Zsolt Dollenstein:Don’t the waveprotocol.org papers talk about this issue?
• [2009-10-21] Justin du coeur:I haven’t read deeply enough into the protocol papers yet to know if they answer this, but I do have a concern about duck’s question. Basically, the issue I’m worried about is whether timestamps suffice, or if the server needs to track every “read” operation.

  This has been lurking at the back of my mind pretty much since I started on Wave, because it’s a conspicuous difference between Wave and my CommYou prototype. CommYou worked exactly as duck suggests: things track the last time I read this thread, and think in terms of the difference between then as now.

  But it looks like Wave is actually tracking blip-by-blip. I haven’t tried the experiments to verify this yet, but it appears that it knows exactly which blips I’ve read and which ones I haven’t, even if I’m reading in an ad-hoc order. So when I come back and do “next unread”, it needs to know which ones I’ve actually read.

  The protocol entirely aside, the storage implications of this are kind of terrifying: it implies a (user x blip) number of records to manage. There are some optimizations that could be performed (at the cost of code complexity), but more than anything this has given me pause about implementing a practical, fully-functional reference server.

  So here’s my related question: does anybody know for sure whether this last-read state is being tracked blip-by-blip, or simply as a per-wavelet timestamp? It has dramatic implications for the UI, the protocol, and the server implementation…

  • [2009-10-22] Michael Kleber:The have-I-read-this-blip status is definitely tracked for each blip. In the current implementation, this per-blip read bit is stored in a private wavelet, much as we talked about for other metadata above.

    Why is the storage cost for this so awful? Your usenet news reader did the same thing decades ago. Sure, it’s stored on the server instead of in each user’s client; welcome to the cloud…

  • [2009-10-22] Justin du coeur:Sure, but this is still a lot of data, even for the cloud, especially when you consider how fast things are likely to move once the Wave infrastructure matures. I mean, my guess is that you’re reading, what, a blip every 10-20 seconds while you’re going through this? Wave makes it very easy for conversations to become extremely fast-paced, especially once they become interactive.

    Basically, the point is that, while Google can handle that level of data, I must admit that it makes me a little nervous for the open-source projects. We need to put some serious work into optimizing that table (what was called the ReadState table in CommYou), because it’s likely to swamp everything else in the system if you’re not careful.

    (Granted, there are some interesting unanswered questions of usage and convention here. It’s obviously less of an issue if the average Wave has three participants. But in a lot of the usage models I’ve been looking at, 75-person Waves wouldn’t be uncommon, and that can really stack up pretty fast…)

  • [2009-10-22] Michael Kleber:Sorry to keep prolonging this, but I’m still not quite seeing it. The blips in a wave get some canonical ordering, and then the per-user “what have I read?” state takes up one bit per blip per reader. So the read-blips state for, say, this very blip? It will only be larger than the blip itself when it has ~2500 readers.
    • [2009-10-23] Justin du coeur:It’s easy to say, but the coding involved is more than a little hairy. Even assuming the canonical ordering is easy to calculate and track (plausible in the Wave protocol, but generally a bit hairy to get the DBMS programming efficient), packing the bits into the readstate record is another bit of somewhat hairy programming, especially to do it efficiently. (If you’re doing this as a conventional RDBMS system, it almost demands a stored procedure in the DB, to keep the traffic to reasonable levels, and that’s a PITA when dealing with OS projects that need to be DB-neutral.) And note that the average Wavelet isn’t necessarily all that long, so the packing is likely to be inefficient. (Since the record is going to be reader x wavelet.)

      I’m not saying it’s impossible – just that, like I say above, we have to pay extremely close attention to the optimization. Frankly, it looks like one of the harder aspects of implementing the system, so we need to go into this with our eyes open.

      (Granted, a lot of this is probably not quite so awful in the Actors-oriented architecture that I have in mind for the hypothetical Scala implementation, which would manage the Wavelet as an in-memory Actor that serves as a write-through cache. But that’s easier in Scala than in many other languages, and is a fairly unusual architecture still.)

    • [2009-10-23] Justin du coeur:Okay, so putting together the conversations I’ve had with Michael today (here and f2f), let me summarize my current thinking about the readstate question.

      Assume:
      Wavelets are big, complex and evolving. Until today, I had thought that “wavelet” == “thread”; Michael corrected this misimpression. Basically, a wavelet is a complex tree.
      ReadState – the relationship between a reader and a Wavelet – is stored on the reader’s “home” server, not the authoritative server for the Wavelet.

      First off, I don’t believe in the bitmap-by-blip approach, for two reasons. It’s a bit tricky to begin with, because the “canonical ordering” isn’t obvious. We’re talking about a complex tree structure that is getting in-line insertions frequently, so the order isn’t trivial.

      More importantly, even if we assume that the server imposes some arbitrary canonical ordering (say, by creation time of the blip), it still doesn’t work, because blips are live objects. Just because I have read this blip now doesn’t mean that it stays read. So we have a remarkably messy problem of marking blips unread when they get edited. That’s hard enough when everything lives on the same server; it’s nightmarish in a distributed architecture. (Probably possible, but I don’t like the smell of it.)

      So I think that, if we are using a packed bitmap, that implies that we are using a bitmap of the history instead – reader * wavelet * transforms. That does have a simple canonical ordering, and never reverses – once you have read a particular history state, the server will never cause that transform to become unread. (You might do so manually, but that’s not a problem.) But it implies that the space constraint is probably hundreds or thousands of bits per reader * wavelet. And the code is still pretty complex: the server probably needs to deal with a lot of calculation of what it means for the client to say which transform is affected when the client says, “I’ve read this blip”.

      I also suspect it’s not the best optimization, at least from an efficiency POV. In practice, you probably get better compression by thinking in terms of the transform stamp that this reader has read through, plus an exception list. (That is, “I’ve read through frame 273, but not transforms 270 and 271.”) I’d bet that that turns out to be considerably less storage than the flat bitmap, but the code is even hairier and harder to get right.

      So overall, my main takeaway is still that this is a tricky problem, and one we need to tackle relatively early in developing a serious reference server, because I’d bet that it’ll take effort to get right and will touch a bunch of problems…

    • [2009-10-23] Michael Kleber:Great point about blips becoming un-read — and more generally, the view in the UI may well be a function of what’s changed since you last read a blip, as shown in highlight and strikeout on this client.

      However you store it, what you ultimately need is a map from blips to last-read-transform (not a bitmap over blip history, since what you read is the current state of a blip always), right? But since reading everything is a common use case, I’d guess store it as “All blips read through transformation N; also {b1->N1, b2->N2,…}”

      Probably at this point we should find out what the present client actually does

    • 2009-10-23 12:19 Justin du coeur:Correct – I was thinking about that common use case in the above suggestion. I’d guess that most often you will be caught up to a certain history marker in the Wave; the exception list is needed because you can read random-access, and can sometimes read the wave linearly down threadwise, which isn’t the same as history order, so you can leave things in a pretty messy state relative to the history.
  • [2009-10-23] Zsolt Dollenstein:Well the problem is that your assumption of one bit per read blip doesn’t really hold, because we also need to identify the blip itself. The obvious solution to this would be a blipid -> read bit mapping, which is significantly larger than one bit per blip.
    Remember, read state is stored in a private wavelet, not in the blip itself.
  • [2009-10-23] Daniel E. Renfer:Also keep in mind that if the read state is stored by the wave server the user connects to and not the originating server, then that 2500 reader number (which in truth is still quite large for even the busier waves) would be broken up by the number of wave servers that have users that are reading that wave. It’s still potentially a lot of information, but I’m sure there are ways of optimizing it. You wouldn’t necessarily need to store the full wave id if you could use a smaller id to point to the full id. You could reduce the size of the id to say, a long int. (in practice though, you would probably want to allow for more room.)
  • [2009-10-23] Justin du coeur:Actually, that’s an interesting point. Are we clear about which side stores the readstate? I don’t find it 100% obvious: it depends on how we define the relationship between the wave and its readers. I suspect that the approach Daniel mentions (readstate on the user’s “home” server”) is right, but it’s worth thinking about whether there are tradeoffs…
  • [2009-10-23] Michael Kleber:I’m definitely with Daniel on this: which blips I’ve read is private information. First, storing it in multiple places, or even in one place far from me, is wasteful. And moreover, as long as I trust the wave server I’m connected to, I shouldn’t need to trust the originating wave server to protect my privacy as well.
• [2009-10-21] Brian May:Do you really trust the clock on every computer? What happens if a clock is wrong?
  • [2009-10-21] Devyn:I agree with that, I think it should be based on UTC server time.
  • [2009-10-22] Justin du coeur:Probably so (it simply makes things easier and more robust than if everybody has to translate time zones around), but I think that goes past Brian’s point. Even if everybody’s using UTC, we need to make sure we don’t have major problems if, eg, someone’s server is running 20 minutes slow.

    Brian, where are you seeing dangers here? Are there areas in the protocol that you think might break down if a server clock is off?

  • [2009-10-22] Devyn:I doubt it would “break down,” however maybe it would be a good idea to recommend the servers to synchronize with pool.ntp.org?
  • [2009-10-22] Justin du coeur:Oh, absolutely – everybody’s going to be happier if the clocks are synched. I just want to make sure that things are resilient if they aren’t – in a distributed environment like this, somebody’s occasional going to have a ridiculous clock setting. (Especially if, as Daniel’s suggested, some folks are running personal Wave servers – more points of possible failure, with fewer eyes on them.)
  • [2009-10-22] Devyn:It shouldn’t break it, but it may or may not be confusing to the user.
  • [2009-10-23] Daniel E. Renfer:This was why I was suggesting using the time settings of the originating server. Clock synchronization doesn’t come into play as long as that server is consistent. The client would be using the timestamps from the server, even if they didn’t make sense for it’s own time settings. (last update was in the future)

    I’ve been holding off on contributing further on this issue until I have time to go over the server protocol again. There are some things in that spec I still don’t fully understand, and I think they may give clues as to how we should be progressing. Even so, the C/S protocol need not bear any relation to the S/S protocol, but it pays to reuse as much of the components as possible. We should still consider that the clients will most likely not be nearly as advanced as the servers will be, and we should try to reduce the amount of complexity for the client.

  • [2009-10-23] mailtolanger@googlewave.com:Just as Zsolt Dollenstein said why not use the allready available version number and wavelet history hash to bring wavelets in order? Why do we even need a timestamp?
  • [2009-10-23] Daniel E. Renfer:The reason I initially suggested looking to the Atom protocol is because Atompub already comes with well-defined rules for modifying and synchronizing collections of authored XML-serializable documents. (documents with a unique uri, one or many authors, a set initial publish date, and a mutable last modified date.) I felt (and still do) that support for the dumb clients out there will be key to the adoption of Wave.

    It’s still unclear to me as to how exactly these hashes are produced, and if the history hash is suitable information to be transmitting to a client, or not. (as I said, I need to look over the whitepapers again.)

    If we are using hashes to control synchronization, then some (but not all) of the reasons for looking at Atom go away.

  • [2009-10-23] mailtolanger@googlewave.com:As far as I understand it the hash is produced by hashing the value from version 0 up to the current version much like git hashes the changeset of each commit. So simply checking if the hash is the same should give you enough confidence that the two states are in sync.

    So first you check if the version numbers match and only if they match you actually do the hash and see if they match, if they do you know you are up to par with the server.

    This has the advantage of being independent of any time information and the versioning gives a natural ordering for the different operations while still being independent on the order they are actually being executed.

• [2009-10-22] Zsolt Dollenstein:Okay, quoting from the Client-Server paper from waveprotocol.org :
  A client and server can verify that they are referring to the same wavelet state by exchanging a version number and a “wavelet history hash”. The latter is a rolling hash over the sequence of operations between version zero and the provided version number.
  So, I pass back the question by asking: why do we need the timestamp?
  Tags:
  Files
  Wave Client Protocol Whitepaper Brainstorm

Author: Daniel E. Renfer
<duck@kronkltd.net>

Date: 2009-10-24 15:15:56 EDT

HTML generated by org-mode 6.30c in emacs 23

Test Post from Hero

It seems inevitable but whenever I test out a new blog client I have to post some stupid message that i’m doing a test.

Well here it is.

Test message from my android phone

Architecture of Mycyclopedia, Pt. 1

I forgot I had this blog. I no longer have a website. I once took a small amount of pride in my website, but my heart has always belonged to Mycyclopedia.

It’s coming along well now. I want to release it, because I know I’ve come up with some cool tricks to do with compojure, but I’ll have to wipe my history first. It’s been a long road to where I’m at. I played around for a time using Jena and a real RDF parser, but I’ve found it has made my job much easier taking a simplified approach and just faking it.

One of the beauties of RDF is it allows you to say anything about anyone. I needed to take this one step further and track information about who said what, and what are the privacy levels of what they said, and when did they say it, and was them saying this prompted by another user saying something that they saw.

I realized that I could assign my own uri to the class of Entries that appear in the Subject position of any statements. I can always refer to another uri by giving it a sameAs relation, but that, like all other statements, is subject to the trustworthiness of the person that made that claim.

So if I can create my own uri’s, then I don’t have to muck about with using the full uri internally, and can refer to entries and statements by their id. When I was modeling this using a pure RDF engine, I had to store all of the metadata as my actual statements, and had to construct model views of the information using reification.

Now that I am using a simple MySQL database, everything is much simpler. The statement table is the most complicated.

I no longer feel like writing, so I will stop this now.

Heartless: Song Titles as Answers Meme

Put your iPod or other music player on shuffle.
For each question, press the next button to get your answer.
YOU MUST WRITE THAT SONG NAME DOWN NO MATTER HOW SILLY IT SOUNDS!
Tag your friends who might enjoy doing this.

IF SOMEONE SAYS “IS THIS OKAY” YOU SAY?
Throes of Rejection – Pantera

WHAT WOULD BEST DESCRIBE YOUR PERSONALITY?
Pushing Me Away – Linkin Park

WHAT DO YOU LIKE IN A GIRL?
Born of a Broken Man – Rage Against the Machine

WHAT IS YOUR LIFE’S PURPOSE?
Conspiracy of One – The Offspring

WHAT IS YOUR MOTTO?
 Overrated (Everything Is) – Less Than Jake

WHAT DO YOUR FRIENDS THINK OF YOU?
Gasoline – Seether

WHAT DO YOU THINK ABOUT VERY OFTEN?
Heaven Sent – Hinder

WHAT IS 2+2?
A Whole New World – Brad Kane and Lea Salonga

WHAT DO YOU THINK OF YOUR BEST FRIEND?
A Condom? – Dane Cook

WHAT DO YOU THINK OF THE PERSON YOU LIKE?
The Ghosts of Me and You – Less Than Jake

WHAT IS YOUR LIFE STORY?
Symphony of Destruction – Megadeth

WHAT DO YOU WANT TO BE WHEN YOU GROW UP?
Lucas With the Lid Off – Lucas

WHAT DO YOU THINK WHEN YOU SEE THE PERSON YOU LIKE?
Capricorn – 30 Seconds to Mars

WHAT DO YOUR PARENTS THINK OF YOU?
Fell on Black Days – Soundgarden

WHAT WILL YOU DANCE TO AT YOUR WEDDING?
Second Chance – Shinedown

WHAT WILL THEY PLAY AT YOUR FUNERAL?
Cheer Up, Boys (Your Make Up is Running) – Foo Fighters

WHAT IS YOUR HOBBY/INTEREST?
Remember the Time – Michael Jackson

WHAT DO YOU THINK OF YOUR FRIENDS?
Suicide Note, Part 1 – Pantera

WHAT’S THE WORST THING THAT COULD HAPPEN?
Miracle – Foo Fighters

HOW WILL YOU DIE?
P.S. Shock the World – Less Than Jake

WHAT IS THE ONE THING YOU REGRET?
Ich Will – Rammstein

WHAT MAKES YOU LAUGH?
The Story – 30 Seconds to Mars

WHAT MAKES YOU CRY?
Vendetta – Slipknot

WILL YOU EVER GET MARRIED?
James – Blue October

WHAT SCARES YOU THE MOST?
Chemical Marriage – Mr. Bungle

DOES ANYONE LIKE YOU?
Year of tha Boomerang – Rage Against the Machine

IF YOU COULD GO BACK IN TIME, WHAT WOULD YOU CHANGE
Epic – Faith No More

WHAT HURTS RIGHT NOW?
Long Way Down – Goo Goo Dolls

WHAT WILL YOU POST THIS AS?
Heartless – Hinder

ScribeFire has a new toolbar

I just went back to my blog for the first time in a long while and discovered that the newest update of the ScribeFire Firefox plugin contains a toolbar for quickly adding links and youtube videos.

I’ll have to play with it a bit more and see how well it works out.

Identi.ca Got So Much Better

So I logged into my identi.ca account this morning only to find out that they pushed out a new version.

In addition to the snazzy new theme and interface, we also finally got groups. Groups are similar to what Jaiku had going, and what I was emulating by using the track feature.

When I first looked into it, there were only a handful of groups created. Checking back a few hours later shows that that number has exploded. Perhaps the barrier for creating a group is too low, who knows. I kinda fear that the vast majority of these groups are going to languish.

My only problem is, this has just pumped up the amount of messages i am now receiving from various micro-blogging sources. I subscribe to my identi.ca feed, my twitter feed, my friendfeed feed, and my laconi.ca track feed.

1 message from a “friend” could potentially produce up to 4 messages to be sent to me. I really wish all of the micro-blogging platforms would do a better job of linking all the copies and only sending me the first instance. (This would probably require a special XMPP client to do this)

Oh well, as long as I can continue reading all my messages in emacs, I’ll find a way to keep up.

Edit: Writing this post reminded me of jaiku, so I turned that feed back on as well.

need a new host

What is the best blog hosting service that’ll let me point my domain at it for free. My domain is without a home since the place I was using freaked out on me.

For the longest time, I was using this other guy’s company. A friend of a friend who moved to New Mexico knew him. He charged me for the first 6 months, and then never hit me up for money after that. It was a pretty basic service, and since I hadn’t been paying him for a while, I felt guilty asking for anything.

For instance, it was still using PHP 4. I had been wanting to play with the PHP 5 stuff, but I didn’t want to ask him to install it. Also, the database creation messed up. Because of that, I couldn’t re-install my laconica instance on the server.

Then I started getting 500 errors one day. I just took back all my stuff, and I’ve been without a domain ever sence.