March 2009 Archives

Another chapter outline for you to comment upon. I'm writing a book for O'Reilly and I'd love your help and suggestions. I don't write code for a living, so I need your input to strengthen this chapter. See earlier post for more information on the book. Let me know what is missing and what examples you'd like to suggest for the book. Comment below or send me an email me at gavinbell dot com or zzgavin on twitter. How social applications differ

• Identity is bound in, other people’s identity and interests become the focus of your site
• Linking, tagging and content reuse are important
• Asymmetric follow makes for tricky scaling issues
• Privacy complicates implementation
• Polymorphic nature of relationships between people, content and metadata, even functionality
• Small is good, let your community shape what you build, rather than launching a big app
• Rapidly evolving protocols and standards

Agile methodologies

• Preference for agile development approaches,
• flexible development a month at a time is better than rigid multi-month plans, ie release early and often.

Deployment and version control

• Retaining flexibility
• Importance of source control and the shift to distributed models like git or hg
• Dev, test, staging and production machines, continuous integration and TDD
• Conditional releases - one site different versions depending on who you are, tags and flags
• Alpha users (set list, or tagged groups)
• Tagged features available to certain tagged users (dopplr model and others)
• Testing with high loads and real data is hard, apache bench etc
• Rolling restarts for no downtime eg haproxy and mongrels
• Puppet and capistrano, managing collections of machines and code deploys.

Implementing the design of social applications

• Design is how it works
• Use cases for developers, understanding the whole picture vs implementing a feature
• The app has a point of view, understanding what fits and what fails as a feature
• How code review helps
• Linking the designed for activities to implementation
• REST approaches compared to RPC / CGI approaches,
• better urls, relying on the verbs in HTTP properly, not everything is a GET
• Feeds and APIs from your app

User interfaces and prototyping

• Failing gracefully is a strong capability for your site
• What makes a website tick (good error recovery)
• Accessibility is important, unobtrusive javascript is a great approach. (Not a tick list)
• Using the console for code level prototyping of ideas
• Templates and CSS vs application code and database schemas

L10N and friends

• UTF-8 and not ascii for data storage everywhere.
• Understanding internationalization and when to do it

Security

• SQL injections, cross site scripting, escape everything, only unescape content you know you really want to display.
• Let them use html and filter approach is doomed, whitelisting is best approach, CSS has similar vulnerabilities
• Cross site request forgery
• Google web accelerator means every link can get processed
• Ensure that form interactions really do come from your own site, use a token to prove this, unique per user and tied to session, this token is now extremely valuable
• Invisible iframes and click jacking
• Flash and browser plugins complicate the whole situation and web browser javascript and quirks mode means they are black boxes

Scaling and messaging architectures

• Forget WS-* unless you really need it for integration
• http://www.37signals.com/svn/posts/1509-mr-moore-gets-to-punt-on-sharding
• http://www.new.facebook.com/notes.php?id=9445547199
• http://highscalability.com/amazon-architecture
• Not about optimisation, architecture more important
• Load balancers, proxies and caching
• Realtime vs non-realtime
• XMPP etc and see http://www.loiclemeur.com/english/2008/12/xmpp-pubsub-aka.html
• NOT just XMPP, the importance of caching
• The qualitative argument for asynchronous development.
• From the SXSW Scaling Boot camp 
• "Design for parallel steps and use queuing as much as possible
• http://www.slideshare.net/rabble/beyond-rest-building-data-services-with-xmpp-pubsub

URLS and IA

• URLS should be as far as possible
• Short, meaningful, canonical, permanent and hackable, plus opaque if needed eg amazon asins
• Pagination and url structures, what was on &page=3 is a recurring problem for aggregation pages, providing date based aggregation alongside simple pagination can help. Imagine Twitter with yyyy/mm/dd aggregation pages too.

Search

• Search is critical to your site, but often left to the end.
• Understand what needs to be searchable and under what context
• Searchable items can grow, eg photos by place on Flickr
• Search can be provided by a third party eg summize and twitter
• Build your search based on the content you store, it is not enough to return simple text matches.
• Finding a photo means returning the picture and metadata, similarly for an event.

IDENTITY and management of user data

• REGISTRATION patterns
• Standard email address, password and screen name approach
• Deleted accounts and how to handle them
• Need for separate admin accounts alongside normal user accounts
• Staff will have admin account and user accounts with different capabilities

OPENID and OAUTH - the open web standards

• Why do they exist
• How to implement them and what impact they will have
• Relative importance of OAuth, balance between identity and access to data, selective permissions
• Portable contacts POCO and what it solves
• OpenSocial, plus the various connect products
• OAuth address book access

Federation

• Distributed systems linked via common protocols and data formats
• Not yet mainstream, but an important direction
• Ma.gnolia M2
• Identi.ca (openmicroblogging)
• http://www.jaiku.com/blog/2009/01/15/were-going-open-source/
• http://www.zengestrom.com/blog/2008/12/foreign-friends-from-a-servicecentric-to-an-objectcentric-social-web.html
• Activity Streams and DISO work

Reputation systems

• What to build and how.
• Data to base reputation upon
• How to aggregation and modify raw data
• Avoiding being gamed
• Bryce Glass and Yahoo pattern library.
• Managing and understanding your community

Need for admin tools

• When to build them
• What to put inside them
• How to maintain them loosely tied to TDD (no admin, no feature)
• Balance with just enough admin to get job done

Statistics

• Will your community want stats
• How to build a personal stats tool
• Granularity, fine is expensive
• Simple overviews eg favouited content etc
• Stats for your own use
• Collect and timestamp everything
• Avoid monthly reporting focus, allow for real time reporting
• Think too monthly and you’ll just get monthly stats
• Good metrics are vital

Collective intelligence

• Recommendation systems
• Machine learning
• Classifiers and filtering tools
• Starting simple and figuring out what data to start with
• Person, location and content are all valid places to build upon

Making your code green

• Average costs per person
• Optimisation of HTML and CSS etc
• Server side optomisation, caching etc
• Flickr ops data graphs and notes
• YSlow and energy efficiency
• Speed = less energy used in general.

The cover for my book, you can pre-order it from Amazon UK and Amazon US.
I'm fondly calling it the spiders book.

Working on the API chapter and feeling like I'm being watched...

I'm writing a book for O'Reilly and I'd love you to comment on this outline, see earlier post for more information. Let me know what is missing and what examples you'd like to suggest for the book. Comment below or send me an email me at gavinbell dot com or zzgavin on twitter. Launching and evolving

• The front page needs to communicate in seconds what your app is about
• 20 words and a picture, a compelling use case.
• Offer a demo, but words are better
• Aim is to get people to signup, but gradual engagement leading to signup is also desirable
Launching in "beta"
• The private alpha (non-public) early and often is a good approach to launching,
• start small and iterate it until you are ready for strangers
• Launch when you have something useful, not every feature on the wishlist
• Small applications are easier to understand and for other people to talk about, it does X
• Invite tickets risks / benefits
• Allow limited numbers of people into see your new app
• Can create an echo chamber and thus skew feedback
• Arriving with social context
• Dopplr
• Twitter
• Contact import APIs and their importance in bootstrapping context
• Tools and services for launching and support
• Getsatisfaction
• Twitter for comments and visibility
• Twitter inside getsatisfaction
• A blog
• Mailing lists
• Nurturing the first few hundred people
• Link to me (digg, reddit, del.icio.us)
• Face to face meetups
• Speaking at conferences and barcamps
• Nounification and verbification twittering, flickring etc
• importance of making the community self supporting or sustaining, they should surprise you
• Feature request vs bug management
• Adding functionality
• Watching for demand from the community
• Keeping up with the competition (or not)
• Listening
• Feature led development (featuritis)
• Making useful products (experience led)
• Staying focused and coherent
• supporting emerging activities
• determining what is next
• BIG changes vs features vs maintenance, eg Flickr localization
• Marketing approaches examples
• Word of mouth
• Being responsive on twitter etc
• Eat your own dog food, use your own product from day one
• Dopplr personal report using Obama
• Flickr blog - connect with your community
I'll of course credit any contributions I use. many thanks, Gavin

I'm writing a book for O'Reilly and I'd love you to comment on this outline, see earlier post for more information. Let me know what is missing and what examples you'd like to suggest for the book. Comment below or send me an email me at gavinbell dot com or zzgavin on twitter.

• On the internet vs part of the internet
• Amazon and google vs most publishers
• New York Times and The Guardian
• Seeing the wider internet
• Integrating with the wider internet
• Building an API
• Why make an API?
• People and objects, the stuff of social applications
• Collections, tags, places and time as containers for objects via API calls
• Services not sites, programmatic access to data, not page generation
• Interoperability and Snowflake APIs (your data is not that unique)
• API traffic volume to twitter / flickr
• Supporting activities, not implementing features
• Using your own public API to make your stuff
• Microformats and RSS (read only apis)
• Where appropriate, bulk usage better via api, so that data only can be sent, not images and css along for the ride
• HTML badges
• Your data on someone else’s site, simple and easy to use
• Writable APIs
• Authentication required
• Security and OAuth
• Latency
• Examples Flickr / Twitter / MovableType and blog apis
• Sticking with standards
• Identica and other apps mimicing the twitter API
• RSS, Atom and AtomPub
• API scaling issues
• XMPP
• queues
• Real time vs near realtime
• NY TIMES, Flickr panda bears
• Privacy and security around APIs, commercial aspects
• Developer community management (wikis, mailing lists etc)

• Integrating existing or competing projects (whatever the route)
• Multiapp integration
• Registration SSO - OpenID and OAuth
• User management
• Data migration headaches and how to avoid them
• Having one of everything, not duplicates
• Email notifications - managing your output from multiple applications
• Internal integration (message queuing)
• On the cheap polling RSS
• Database calls, fast and brittle
• Queues and message passing
• Dealing with multiple claims of authority
• Family of identifiers better than one?
• Search
• Single search databases (big solutions like MarkLogic)
• Cooperative approaches (polling multiple databases and integrating)
• Multiple content sources with a single search interface is a hard problem, common for many publishers
• Community and published content, blog vs publishing workflow tools, who wrote this?
• Integration with other people’s APIs
• Functionality for free from others FTW
• How to manage external dependancies on others APIs
• Understanding provider and consumer relationships for your application

I'll of course credit any contributions I use. many thanks, Gavin

I've mentioned before that I'm writing a book for O'Reilly, entitled Building Social Web Applications. It should be published this August. I've written quite a lot of it based on a mixture of interviews and conversations with people at conferences and in London. I'm now writing the more technical chapters and I need your help. I'm an interaction designer and product manager with a good head for technical detail, but being honest I don't write code daily. So it would be great if I can get your input into these three chapters.

• Writing the application
• Extending the application, APIs and integration
• Launching and evolving

The earlier chapters cover the questions of why are you building a social application and look at how people behave and interact with one another online. These three chapters cover how to implement and extend the application. The intent is not to give a lot of technical detail and reams of code, but to explore how social applications differ. They are not the same as building a content publishing system or an online shop. I've created some outlines to give a framework for what I think should be in the chapters, but please let me know what is missing.

I'm not interested in recommending a particular language or framework, nor one particular approach to scaling. I'm also interested in your opinions on centralisation vs distribution and federation. Examples that you can share and suggestions of content to include are most welcome. Dopplr, Flickr, Twitter and sites like LinkedIn or Facebook are the kinds of things I've been featuring, but I've also been looking at adding social features to newspapers, magazine and product companies.

I'm going to put the API and integration chapter up first, the other two, tomorrow. I want to start writing the chapters on the 23rd of March, so about ten days or so from now, so please let me know what you think. I'm zzgavin on twitter and I'll be talking about the book there too.

Ever since we got burgled in 2005 I've intended to replace the video camera that got stolen. We bought a Canon 20D SLR with the insurance money instead. Today I was looking at older pages on backpack and found the feature list I thought was essential about a year or so ago.

16:9 native
manual focus
firewire i/o
manual white balance / exposure
head phone
microphone
stills capture whilst recording
image stabalisation (preferably optical)
c. 500g
all features on the camera, no dock

Lucy got me a Kodak Zi6 recently and whilst it is 16:9, it has little else on the spec sheet. What it does have is pocketability and near instant on. A lesson to me in getting over involved in the features and not the activity. Of course, I'd want a slightly wider angle lens and better low light, but then it wouldn't have been about 100 pounds. I'll probably get a video camera at some point, but for long photo use, especially of my sons the Zi6 is perfect.