A life more eclectic

Note: While some of my motivation for writing this comes from conversations that I had while at WWDC, I believe that everything that I will talk about is either solely my opinion or covered on Apple's public pages on MacOS X 10.5 Server. Thus I do not think I am violating NDA. If someone feels that I am, I encourage you to tell me, and I will remove things where appropriate.

During the WWDC 2006 Keynote it was revealed that Apple's Spotlight search technology is being extended to allow for server-based searches. This is great news for those of us who manage workgroups that have both user and group shares on the server. Currently one of the problems with Spotlight and server shares is that if you are going to index the share each individual client has to index the share, and then keep that index up-to-date. When you get more than a few clients constantly scanning everything they can see this can overload your server.

So having the server do all of the indexing and then allowing a protocol for the clients to ask the server to respond to queries when they are made is a great step forward. But here comes the part that is prompting me to feedback: the indexing is a part of MacOS X Server and does not seem to be on the list of things that Apple is making open source. That means that if your network shares are not served up by MacOS X Server they will not be available for this service.

My request is not that Apple opensource its Spotlight Server but rather that Apple make the protocol that is uses to communicate to the client openly available so that other vendors can incorporate this technology into their server products and provide MacOS X clients the same great service regardless of what is on the back-end. I realize that this would reduce the appeal of MacOS X Servers slightly, but to be honest in a lot of big locations MacOS X Server does not scale well enough to handle the very large file serving load that is required. And Apple does not really have anything on the very low end for small workgroups.

And by opening this protocol up Apple is doing exactly what Microsoft is being required to do in Europe: allow interoperability. They are not really giving out a competitive advantage because everyone out there can figure out what this would require, the really hard part is making the indexing systems fast and seamless, and I think that is somewhere that the engineers at Apple can compete successfully with anyone.

My request does entail a few things:

It requires that the Spotlight protocol be out-of-band with the file serving. That does mean that it would require separate authentication, and does allow for mistakes in this authentication that would allow client views to get out of sync. Having the communication out-of-band does have a few advantages: You can change protocols on the file serving without changing the protocol in any way. So AFP does not change, and you can quickly get this ready to work with SMB, AFS, NFS, HTTP, WebDAV, etc.. without making any modifications to those protocols.

Having this out-of-band means that you could also have the Spotlight connection established to your network shares open even if you don't have those shares mounted. This would be very nice as you could search everything available for a file without mounting everything, and allows for simple searches of http content on an intranet. If the results were URL oriented then you could even just click on a search result to trigger the network share mount.

A final point about out-of-band communications: it also allows for third-party implementations. Our file servers are Windows2003 computers (not my choice, but they work well). I don't imagine that Microsoft is going to lift a finger to implement Spotlight compatibility in their code. But if the searching protocol was out-of-band a third party could create an indexer that runs on Windows and offers that information to clients that connect to it.

The base protocol should be open standards based if possible (let someone else do the heavy lifting). I am sure that people better versed in this area than me can make good suggestions about what the protocol should be, but to me the requirements would be: bi-directional - once a channel is established both sides should be able to initiate messages (server-based change notification), encrypted, and be able to pass through FireWalls and NAT in a straightforward manner (single, predictable TCP port).

It is fine if on top of that something Apple-defined is established.

I hope that other people will echo this feedback to Apple so that we can make sure that Spotlight Server is implemented in a a way that we in the Enterprise market can really take advantage of.

Karl Kuehn