You can’t just have a sysadmin on call

A good article on the importance of Dev Ops: Server Density Blog

At IGN & vudu, a dev team member for each stack was on call for a week at a time. He was free to enlist a substitute for short periods such as a commute, dining out, or a medical appointment. A notebook PC was supplied.

Let's be realistic: It is unlikely that the on-call engr will be familiar with the code which breaks. The company should pay cell phone bills to guilt engrs to keep their phones charged & nearby. For a newly deployed app which has the highest chance of failure, the original author must be on call. If there's an upcoming busy period, all authors of code under stress must be on call.

htmlburger & 99designs

Hands for Science is a non-profit helping teachers do hands-on science activities more easily. I programmed the web site www.handsforscience.org, visual design was outsourced to 99designs.com, and web coding was outsourced to htmlburger.com.

As expected, 99designs.com was a cost-effective way to get several concepts. Many of the designers don't understand CSS so they suggest unrealizable designs such as a paginated float & colored auto-numbering. Don't expect one designer to excel at everything. The logo and page templates were done in separate contests. Be sure to point out that the logo must look pleasing in the global nav. One template is for the description of the hands-on science activity. Unfortunately, the designers did not seem to excel at printed page design.

It took some doing to adapt the htmlburger.com coding of the 99designs.com PSDs. The client needs to give htmlburger specific instructions or the code will be too specific to specific pages instead of being reusable style/html frags/templates. Insist that all directory, file, & style names be prefixed w/ hb- or the like. This makes it easy to distinguish their stuff. They improved my jquery slideshow. They even wrote JS to shrink a long caption's font-size.

On the marketing pages, I wrote custom tags (rails view helpers) which alternate sides for the graphics.

HandsForScience CMS Uses mongodb

HandsForScience is a non-profit which helps teachers do hands-on, science activities more easily. I'm using mongodb for the HandsForScience CMS: http://www.handsforscience.org/activities

The stack is rails 3, mongoid, cancan, & jquery hosted via Heroku, mongolab, S3 (for attachments).

Any CMS will have User, Article, Link, & Attachment classes--in our case, Activity takes the place of Article. Our schema adds:

• Standard: A curriculum standard which the activity cross-references. So far we include the California K-5 standards.
• Material: An activity includes materials. The list is crowdsourced.
• Term: An activity includes vocabulary terms. The vocab is crowdsourced.

Instead of a Grade class, an activity's grades are an array-valued field. Instead of a Subject class, an activity's subject has a string-valued field validated as an enum: life, earth, physical sciences.

The most interesting tradeoff is the Activity/Material relationship. There must be enough material for each activity. The quantities are classroom, small group, student. Instead of creating a join class ActivityMaterialQty, there are three many-many relationships.

Another tradeoff is object creator fields are string-valued instead of many-many-User relationships. I should probably switch to BSON::ObjectId.

It would be nice if mongodb supported case-insensitive sorting. I'm working around this by sorting in memory. I should switch to denormalized, lowercase fields.

Separation of Software Development & Production

When I was managing software engineering/development and data center operations teams at excite, IGN, Fox Interactive, and vudu, a large percentage of my staff questioned my methods. I required the dev team to manage its app in production--including the app servers. I required the ops team to manage the httpd/OS/network stack. I had a separate DBA team to manage the data stores. The dev team wanted to code--not deploy/operate. The ops team didn't want devs to access their servers and wanted the trust to run it all. I wanted specialized teams who I could hold accountable.

I dreamt of the day I'd have enough resources to split each team into dev/ops subgroups. I tried various approximations: ops role for a week; devoting fixed half-days to dev; outsourcing after-hours ops (DBA; netops).

google's Site Operations has the resources to address this dilemma. Here's an interesting video by one of goog's Site Operations Engineers.

The CEO & CTO must interact

Startups typically have a CEO and a CTO who trust each other. Dangerously, in the early days the CTO role is sometimes played by a consulting firm. It is important for the CEO to check on the "CTO": 

1. stack (aka tool set): Is the stack evolving? Which big sites are powered by it? Can you pay for support? Would TBHs want to work w/ it? This can be a big barrier to hiring.
2. architecture: Is it scalable? What if the site is overwhelmed? How is an attack deflected? How are analytics handled? What happens if cloud costs rise and you need to in-source hosting?
3. usage: How are errors handled? How are backups done? Where's the source code? How are upgrades handled?

It is equally crucial that the CTO check on the CEO:

1. Is the cap table too generous or stingy?
2. Is the sales forecast on track? Missing business milestones is a red flag.
3. Is the expense forecast on track? Excessive legal fees are a common burner.
4. Never run out of money.

we picked the winner of TechCrunch Disrupt -- sort of

We invested in getaround before they won TechCrunch Disrupt. Let's hope this translates into a huge success.

Our investment in Berkeley Bionics made the front cover of UC Berkeley's Forefront magazine. It is nice to be helping people in a small way.

angel investing: 3 structures for cash distributions

There is no way all apps will achieve the critical mass required to exit. There are three investment outcomes: 1) failure 2) acquisition or IPO 3) SMB. In the SMB outcome, investors don't participate--their investment is lost. To avoid this binary outcome, I've encountered the following structures.

1. royalty-based financing: The investor receives a percentage of revenue--just like the sales force does.
2. investor put option: If the company reaches a level of success, the investor has the right to force a buyback (hence the term put option) a fraction of his holdings at a multiple of his original investment. The investor retains the balance to participate in future upside.
3. dividends: Employee cash compensation is capped. To pay bonuses, a dividend is declared.

From the point of view of an investor, royalties are most attractive. From the point of view of management seeking an exit, the put option is most attractive because the company's hit is constant. From the point of view of management seeking a lifestyle business, dividends are most attractive.

Angels seeking to optimize portfolio returns should abandon the traditional "swing for the fences" VC mentality in favor of structures such as the above.

Entrepreneurs seeking to build successful companies instead of "getting rich" should employ structure 3. If a lucrative exit presents itself, that's always an option.