- How many of you had the problem where you had a great idea, something that can help the business, only to have IT say, “well, maybe you can have it in 2016, when the planning calendar opens up?” (Lots of hands go up.)
- Where Did the High Performers Come From?
- Non-commissioned officers.
- Chemical engineers
- Auditors
- What do they have in common?
- Rigor and discipline!
- When we wrote Visible Ops, we saw a downward spiral
- Fragile applications are prone to failure
- A long time required to figure out what went wrong
- Detection comes from a salesperson who said “why are the banner ads being shown upside down”?
- Too much firefighting and unplanned
- Planned work cannot get done
- Frustrated customers leave
- Market share goes down.
- Business misses Wall Street commitments
- The key aha: This isn’t an IT ops problem, this is a business risk.
- Mission: Figure out how to break the IT core chronic conflict
- Every IT organization is pressured
- Tribes needed:
- Ops
- Dev
- Op Security
- Design
- Velocity talk in 2009 at Flickr: 10 deploys per day
- Dev and ops working together
- Ops who think like devs
- Devs who think like ops
- In 2011, Amazon doing a maximum of 1,079 deployments per HOUR.
- That’s 11 seconds per deployment.
- If your company can deploy at most once a month, how can you compete against someone who can deploy daily or hourly?
- DevOps is a real movement
- I would never do another startup without employing devops principles
- It’s happening in enterprises, government, and non-profits.
- The Three Ways
- The First Way
- Systems Thinking: Left to Right
- Never Pass defects to downstream work centers
- Never allow local optimization to create global degradation
- Eradicate blockages in the flow
- Outcomes
- Faster cycle times
- The Second Way
- Amplify feedback loops (right to left)
- Expose visual data everyone can see how their decisions affect the entire system.
- Outcomes
- The Third Way
- Culture of Continual Experimentation and Learning
- Foster a culture that rewards
- Experimentation (taking risks) and learning from failure
- Jared Spool story of Intuit, where the CEO, in a monthly ceremony, gives a lifepreserver to the person who took the biggest risk, and they share their knowledge of what they learned.
- Repetition is the prerequisite to mastery
- Outcomes
- 15 minutes
- Prescriptive
- Meeting the DevOps Leadership Team
- Typically led by Dev, QA, IT Ops, and Product Management/Design
- Agile Sprints
- typically one week to one month
- at the end of each sprint, team should have potentially deliverable product
- But where this breaks down, is that typically dev uses up all the time in the project, leaving none operations or testing
- Help Dev and Ops Build Code and Environments
- Dev and Ops work together in Sprint 0 and 1 to create code and environments
- Create environment that Dev deployed into
- Security must integrate security testing into continuous testing through automation. If it takes 2 to 3 weeks to perform a security check, it won’t fit into the agile process, and it will be marginalized.
- Keep Shrinking Batch Sizes
- Waterfall projects often have cycle time of one year
- Sprints have cycle time of 1 or 2 weeks
- When IT Operations work is sufficiently fast and capable (e.g. it’s a < 1 hr process) we may decide to decouple from sprint boundaries.
- Now we don’t have to wait two weeks for a feature to go out.
- And the deployments get real small: we push out a single feature.
- This is lower risk than pushing out hundreds of features together.
- IT Operations Increases Process Rigor
- Letters To Stakeholders
- Development:
- Be aware of the downstream effects of your actions
- Unplanned work comes at the expense of planned work (features)
- When we take shortcuts at the front of the line, it has an amplified effect downstream.
- Technical debt retards feature throughput
- Environment matters as much as code
- QA
- Ensure test plans cover not only code functionality, but also:
- suitability of the environment the code runs in
- The end-to-end deployment process
- Help find variance
- functionality, performance, configuration
- duration, wait time, errors
- Operations
- Expect and tolerate failure (use Chaos Monkey)
- See: “5 Lessons We’ve Learned Using AWS”
- “The best way to avoid failure is to fail constantly”
- Harden the production environment
- Have scheduled drills to “crash the data center”
- Create your “chaos monkeys”
- Product Management:
- Marty Cagan: Led product mgmt organization at eBay
- He inherited the organization at eBay when they were suffering from chronic outages.
- “you must take 20% of dev cycles to paying down technical debt.”
- Designers
- Help IT Operations codify their work and requirements into great and ever increasing library of user stories
- Realize that IT processes are likely the largest impediment preventing your great ideas from making it to market
- By working on the processes of how code gets into production, you can remove the impediments, and get more
Category Archives: sxswi
SXSW Tips for 2012
Here’s my annual list of tips, updated for 2012.
That’s one full conference room. Get to your session early to get a seat. Popular sessions fill up quickly, and once they do, you aren’t allowed in. One more reason to plan your schedule in advance. |
- Power equipment: Get yourself a travel power strip, and/or auxiliary battery for your laptop. Being able to take notes, follow the twitter stream, or research sessions for 10 hours a day is a stretch for almost any laptop or phone. I own and love the Monster Outlets to Go travel power strip, with built-in USB port. It packs down small, and let’s you walk up and use any outlet, even one already occupied.
- Plan out your schedule. There are thousands of sessions you can attend – usually from 40 to 60 during any given timeslot. Which are the ones that are most interesting and applicable to you? Although you should choose primarily based on interest and applicability, all other things being equal it is usually a good bet that speakers in larger rooms are better speakers than speakers in smaller rooms. So get familiar with the map of SXSW, and figure out which rooms are which. Also, it’s not a given that you can make it to any given talk from any other given talk in the allotted time, and still get there with a place to sit. So look at the overall map of all the hotels, and figure out what you can make.
- Choose your backup talks: For a given time slot, you might have a favorite talk you want to attend. Maybe it will be awful, or maybe it’ll be full and you can’t get in, or maybe it will be cancelled. With SXSW Interactive spread out across many blocks and different buildings and different floors, it’s not possible to get from any given room to another quickly. So once you know your preferred talk for a given timeslot, pick out a backup talk that is nearby.
- Wean yourself off coffee: Depending on just how hardcore you are, you may want to consider weaning yourself off coffee, or at least reducing your dependence on it. That way, when you get to Austin, you can restart your caffeine habit, and enjoy the full stimulating effects of it. (I wean myself down to one cup per day ahead of time, then plan to enjoy 3 or more cups per day there.)
- Holy Basil: Like to drink? I’m really fond of Holy Basil, which is a natural hangover remedy. I’ve found it to be highly effective. (I’m not a doctor, this has not been evaluated by the FDA, etc, etc. Be smart.)
- Business Cards: It’s surprising for such an online environment, but business cards are still pretty popular. If networking is important to you, bring some. Make them simple. Name, email, phone, twitter handle.
At the Start of Each Day
These are the registration lines. Plan to give yourself at least an hour to get your badge on the first day. |
- Food/Coffee: Get your coffee on the way to the conference center, not at the actual conference center. Lines for coffee are 10-20 minutes long. Also, starting in 2010 they started having sessions go through the lunch hour. I think that sucks, but I hate to miss anything, so I go to them all. Since it’s hard to get food quickly, you may want to bring another snack bars that you’ve got food in your backpack to cover you through to dinner. I’m partial to KIND Nut Delight bars, which are relatively low on sugar and high on protein, and the closest bar I can find that is 4 Hour Body (4HB) compatible.
- Start charged: Start the day with charged laptop/phone/etc.
- Clothes: Bring a light jacket in case you don’t make it back to your hotel room. It’ll cool down at night. Conversely, it will be warm enough at some point during the day for short sleeves.
- Reschedule: Learn anything interesting yet? Find some new track that seems interesting? Reevaluate your list of planned talks, and see if you want to make adjustments.
Don’t sit in the back. Go ahead, find a seat up front! Make friends with the person sitting next to you. |
- Be in the moment: Don’t go to a session and then check out and read email, surf the web, or do work. SXSW is precious. Make the most of your time by being totally immersed in what is going on.
- Recharge: Look for outlets in hallways, restaurants, outside, anywhere, and use them when you find them.
- Conserve power: If you are taking notes on your computer or blogging the sessions, you may want to turn off wifi on your laptop to save power (and to keep your focus on the session, so you don’t start random web surfing.) I usually use my smartphone to follow twitter and email so I’m still connected.
- Follow the #SXSWi tag on twitter: You want to follow #SXSWi so that if another session is excellent and your session kind of sucks, then you can make the switch quickly. (or conversely find out if a room is already packed and can’t fit any more.)
- Follow the twitter tag for whatever session you are in: There will be a back channel of conversation about the session you are in that is almost as valuable as the primary speakers. SXSW is full of experts, both presenting and in the crowd, and you want to tap into all of that wisdom. This doesn’t violate tip #1, because you are not being distracted by something different, but rather tapping into more of what you are already there for.
- Talk to the people around you: SXSWi is a social place. The people around you are likely to be very experienced, smart, interesting people. Start up conversations, make dinner plans with strangers, and keep going until 2am. The wisdom of the crowd is not just an abstract thing at SXSW – it is manifest in the people all around you. Talk to them.
- Attend my session: Wall-E or Terminator: Predicting the future of Artificial Intelligence, with author Daniel H. Wilson and scientist Chris Robson – Tuesday at 9:30am.
Flamethrower Music Video
This was the video shown at SXSW on the topic of curating and preserving internet content several years ago. I lost track of the video, but wanted to repost here so people could find it:
Please Vote for My SXSW Interactive Panel
Please vote for my panel on Artificial Intelligence at SXSW: Terminator or Wall-E: Predicting the Rise of AI
The questions we’ll be tackling: When and how will human level artificial intelligence emerge? Will they kill us all like Terminator, or protect the planet like Wall-E? Daniel Wilson, author of Robopocalypse (being filmed by Steven Spielberg) will be on the panel, and the brilliant Chris Robson.
Your votes help decide which of the over 3,000 proposals talks get accepted. Last year you helped my talk on innovation get accepted. Will you please vote again this year?
Thanks,
William Hertling
Summary of SXSW Interactive 2011
Here’s a summary of key insights from SXSW Interactive 2011.
15 Slides, 3 Writers
At 15 Slides, 3 Writers, we learned about three different writers’ techniques on the same fifteen twelve topics. A few key insights:
- All of the writers favored very simple text editors, such as BBEdit. They didn’t want the distractions of a more complex editor.
- There was a common theme around writing first thing in the morning.
- If they get stuck, they change context, either by: going away and coming back to it later, going back and working on grammer and structure issues in the earlier part of the writing, or changing colors on the screen to create a new visual context.
- They proof in a different context: either they print it out, or they publish to the web and read it there, where it just looks different as compared to their editor.
Tim Ferriss spoke about the 4 Hour Body, about which I have written plenty, including this nifty cheat sheet. A few insights:
- The 4 Hour Body was the result of more than 10 years of data gathering, and 3 years of intensive experimentation.
- The Extremes Inform the Mean, not Vice-Versa: Example given was a design firm working on garden shears: didn’t want to know about the mean user. They wanted to know about the extremes: the parapalegic who was gardening. the elderly user. If they could design for the extreme, the middle will be taken care of.
- Make it a game (Drucker and five sessions). People who do something 5 times will keep doing it.
Rig the first 5 sessions so you’ll keep doing: go to gym for 15 minutes, not 3 hours. - The best protocol doesn’t matter if you abandon it.
- Q: From your new book, what could you apply back to the 4 Hour Work Week?
A: Richard Branson was asked what thing to do to be more productive, and he said “work out”
There is a lot to provide that exercise has a big positive impact on learning, cognitive ability, and productivity. Use exercise/recreation to bracket your day: before and after. - Some fantastic doctors out there. But… C=MD. As long as you pass your tests, you get an MD. There is always someone who is the worst in their class. The average healthcare visit is 11 minutes. Many times a doctor doesn’t or can’t spend the time to work with you.
Hey! This was my talk. Here are a few sources for more information:
- Visual notes from the live session
- Powerpoint is Still King
- Building a Base of Support
- Finding the Time to Innovate
- The Good, The Bad, The Forgotten: Lessons Learned Presenting at SXSW
Chris Poole, founder of 4chan, gave a keynote address. Key insights:
- In comparison to Mark Zuckerberg, who believes authenticity can only come through identity, Poole believes people need the ability to be anonymous to be authentic. Trying to explain to this someone, I came up with the following analogy: We do and say certain things at work, and do and say certain things with our friends, and we don’t always want the two to cross. So it does make sense that different identities with different groups allows us to more fully behave the way we want to behave in that group, as opposed to constantly filtering our behavior.
- People focus on scaling as an architecture problem. The real problem is not scaling, it’s building a community worth scaling. 4chan was not an overnight success. it’s been a slow steady build over 8 years. there was no hockey stick. The core community forms over time. With Poole’s new site, Canvas, there is concern about the culture growing: if they let 10,000 people into the site tomorrow, it would destroy the fragile culture that is developing.
- Some way for the quiet many to participate makes a big difference in the life of the community: In canvas, like all other communities, a small percentage of users create all the content. They wanted to create a new middle ground – so they made virtual stickers which could be dragged and put onto other posts. Now you didn’t need to be an artist to create visual content. 100,000 stickers placed in a few weeks.
- Historically self-publishing has been looked down upon by traditional press. But in the comic industry, it’s the complete opposite. You have to be self-published, to prove you are committed, you have an audience. That will start to happen in the traditional publishing.
- All three authors on the panel extensively used social media, especially Twitter, but also Facebook, to represent themselves professionally, stay abreast of editors and keep track of editor/author relationships, and of course key in touch with readers.
- The marketing/publishing people are obsessed with how many hits my blog gets, what are the search terms, how many followers and friends do I have. [I saw this echoed at other writer’s conferences as well.]
- I took my self-published book, which was $15 for the printed book, and did a $2.99 book on Kindle. It was slow for a while, but sales have taken off, and they’ve even driven up the sales of the printed copy.
- My main motivation is to have as many people read my stuff as possible. I don’t care how it happens, I just want it to happen.
- You always have to do the marketing yourself, even if it’s a traditional publisher. You have to market the book, and you always have. Now it’s just easier to do. Once upon a time you had to get in the car and drive to every bookstore in the country. Now you can get a national or international following through online tools.
This talk by Thomas Myer was a lot of fun. The premise was that if you can develop a product you can sell and make $100 per month, then you can repeat that process ten times, and start to build up an income stream. Full session notes. A few highlights:
- Wrote eight books. Not a lot of money, but checks still do trickle in. No need to provide tech support, which is nice.
- Wrote backup software, which was a plugin for a platform of some type. Turned out to be tricky and require a lot of tech support calls. But then it turned out that much of the same code could be leveraged horizontally, so that he could release backup software for a different platform. Gave him more product reach.
- Wrote a few iPhone apps. Very simple apps to solve very specific problems: e.g. writers encounter writer’s block or need ideas, and the app is an idea generator. The UI design is extremely simplistic, but that doesn’t matter: solving the customer’s need is what matters.
- Examples of assorted information and teaching businesses: produce a video on how to do X, then sell access to the video online.
- Donations doesn’t seem to be viable. Needs to be an actual payment for a product.
Gamestorming is a technique to facilitate an ideation meeting that is a cross between too much chaos and too much structure. It uses game techniques to make it fun while still oriented towards a goal. Full session notes. Probably the best thing to do is to buy the book Gamestorming, which I plan to do.
Here are excerpts from two presenters at Cloud Camp.
My long time buddy Gene Kim, founder and former CTO of Tripwire, spoke about devops.
Key insights:
- Now more than ever, we need great IT operations. Organizations are held up by the ability to get features released, get things through operations. It’s not just an operations problem, it’s a development problem and a business problem.
- They benchmarked 1,300 organizations – to link controls and performance. High performance organizations exist and they are 4-5 times more productive than ordinary organizations.
- High performers find and fix security break fast
- Unplanned work comes at the expense of planned work
- Traditional organizations have a Vicious Downward Spiral:
- Too many fragile applications (prone to failure) -> Too much time spent firefighting and in unplanned work -> Planned project work cannot complete
- Downtime causes frustrated customers to leave, while features fall further and further behind -> Market share goes down
- More urgent, date-driven projects put into the queue -> Even more fragile code put into production
- More releases have turbulent installs -> Release cycles lengthen to amortize cost of deployments -> Bigger deployment are more likely to fail, and more difficult to diagnose
- Zone #1 for improvements:
- Decrease cycle time of releases
- Create determinism in the release process
- Move packaging responsibility to development
- Release early and often
- Decrease release cycle time (example: Reduced deployment time from 6 hours to 45 minutes)
- Never fix forward, instead “roll back”, escalating any deviation from plan to Dev
- Verify for all handoffs (e.g. correctness, accuracy, timeliness, etc.)
Earnest Mueller from National Instruments also spoke on devops. I found his very explicit details of how their web development/operations team works to be fascinating. A few highlights:
- The Process:
- Agile
- All systems work used the “developer” tools and systems [Revision control: Perforce / Bug Tracking: HP / Specs and reviews: Atlassian Confluence Wiki / Task tracking and burndown: JIRA/Greenhopper]
- All members collaborate on all aspects of the product. This was the key to making it work – using all the same tools. We could prioritize better, because it was all in one system.
- There can be a fear that the systems tasks would always get pushed out. This seemed to be mistaken impression. When presented alongside requirements in the same requirements tool, decision makers seemed to understand the need for systems work.
- System Automation
- We built our own: PIE, the “Programmable Infrastructure Environment”
- Looked at Chef/Puppet, and others. What we needed wasn’t quite any of those.
- XML System model defines systems, services, code installs, runtime interaction, variable substitutions
- PIE autobuilds the system from the model: provisioning, software installs, monitoring integration
- Zookeeper as a runtime registry for systems info and eventing
- Allows us to start/stop/control/install/autoscale on bunches of dynamic environments
- We have our dev environment, test environment, production environment – multiplied across our many products. We could deploy a new environment in a couple of hours.
- Challenges
- Overcoming the thought that it was impossible: Can i do infrastructure tasks using agile? with sprints? By actually trying it, it turns out to be possible. Write our own software provisioning sounds hard, maybe we can’t do it. but when you try it, you can.
- Building trust between dev and ops: Working together in one team and using the same tools really helps. You get transparency. You can’t build the sense of trust when you don’t know what each other are working on.
- Explaining core web development/performance/availability/management needs to desktop developers. Needed to write “why you should log” paper to explain core concepts
Presenting at SXSW: The Good, The Bad, and the Forgotten
Gene Kim and I organized a session at SXSW Interactive this past weekend. Our topic was Be Heard: How to Drive Innovation in Big Companies.
Gene already shared some of the work we did to prepare for the presentation.
I wanted to share a few more thoughts while they were fresh in my mind. This is my own opinion – if you were there, I’d love to get your perspective. Here are our lessons learned from presenting at SXSW.
The Good
1. We got right to the point, and built energy quickly.
We started with a vision: to get the session started quickly, establish credibility, tell a story to build energy and give an example, and then get going. We didn’t want to get bogged down.
Keeping that vision firmly in mind, we refined the content relentlessly. We had started in early February, shortly after we learned that our proposal was accepted. I had written a half dozen personal stories, and we each wrote bios.
By the time we both had gotten to SXSW, I had 15 minutes of material I was planned to cover. However, I could feel the energy lagging just rehearsing the content, and I knew 15 minutes was much too long to make the audience wait for a session that was intended to be audience-driven.
We spent about eight hours together in person at SXSW refining the talk until we had the minimum amount of material that established our vision for how the session would go.
2. We didn’t waste time building motivation.
There were 48 programming slots for 12:30pm on Saturday. That meant that anyone who came to our session came because they were interested in our title and our session descriptive. So we didn’t need to convince anyone who was there was it was important. By choosing our session out of the 48, it was a sure sign they thought it was important.
Therefore, we focused on making sure people left with actionable tips and techniques. We’ll leave the motivational stuff to the keynote speakers.
3. We involved the audience both in soliciting the problem statement and in offering the solutions.
Knowing that it was a core conversation format, the expectation is to involve the audience. And one of the unique characteristics I’ve noticed about SXSW is just how smart everyone is: You could pick any person at random and have a very interesting conversation with them. Everyone is an expert in their area.
Therefore, we expected people to know more than us, much more. So not only did we want the audience to offer solutions, but we also wanted to make sure we got the inventory of problem types from them as well. This also gets to the actionable techniques: they are only actionable if they match a problem you are having.
4. We time boxed the problem solicitation portion, the brainstorming for each problem area, and each problem.
We knew that one downside that can occasionally occur with core conversations is lagging energy, or a slow pace. If someone comes up and hogs the mic, or if you spend too much time in an area that large swaths of the audience isn’t interested in, energy and attention lags.
So we time boxed everything. (As it turns out, I don’t think we ever had to cut off an individual speaker. Yay. I wasn’t looking forward to that.)
1. Too little sleep.
There was no projector to put up our names or contact information, nor a flip chart to write them on. Ideally SXSW would provide this, but given that they did not, we should have spent more time telling people our names, twitter handles, and websites.
We would like to connect with people and have the opportunity for follow up.
2. Continuing the conversation
Gamestorming Session notes from SXSWi
- imagine it’s your job to facilitate a meeting. the stakes are to solve global warming. you only have a day. you have the people in the room. what do you do?
- options
- the boredom of business as usual
- agenda 9am 10am 11am
- slides being projected
- boring as hell
- people are checking their email
- the smart people get an important phone call and have to leave
- chaos of creativity
- post it notes
- everyone talks over each other
- it’s wild and crazy
- they feel kinda good, they generated a million ideas
- but at the end of the day you have a million post it notes, and that’s it.
- nothingness <——> chaos
- how would you like to be the meeting jedi?
- everything starts with energy
- it’s true of people, cars, and meetings
- camp fire vs forest fire
- one is out of control, and one is in control
- one will kill you, and one will keep you warm
- require a spark + fuel to keep going
- a campfire has a structure you design in.
- business at usual = no fire, chaos meeting = forest fire
- energy will follow the path of least resistance
- traffic will flow through streets
- water through channels
- ball down hill
- you can structure where the energy goes in a meeting
- What kind of result do you want?
- new ideas, agreement, tough decisions, problem-solving, unraveling complexity
- Monopoly
- board world
- pieces: players
- objects: properties, houses
- making flow: rules, flow around board
- breaking flow: dice, chance
- Gamestorming
- Players
- objects: sticky notes, index cards
- whiteboard = world
- making flow: rules, flow
- breaking flow: shuffling
- Tools on the Table
- Firestarting
- Sketching
- Improvisation
- Meaningful
- Firestarting
- you should never go into a meeting with no idea of how to ignite conversation
- strategically have a provocative question that gets people engaged and dictates where the conversation will go
- fill in the blank: I want my appliances to start telling me ________________
- Sketching
- is visual language
- is just as important or more important than text based language
- having a meeting with just text is like having the meeting with one hand tied behind your back
- you don’t have to be divinci
- Improvisation
- it creates possibilities
- if you are in a meeting, and are only comfortable with not being surprised, you’ll only get predictable stuff
- being comfortable with the possibility of something going terrible wrong
- Meaningful space
- we think an idea is a lightning bolt or isolated incident
- it’s a bunch of neurons firing in sync at the same time in a new way
- you need a space which is going to stimulate neurons to fire
- an ordinary space is not going to generate new neuron patterns
- always try something new, keep it fresh
- MEME MACHINE game
- treat the room as a giant computer
- lots of parallel processing
- AWESOMENESS app
- Phase 1:
- pick your favorite software app and your favorite physical product
- think of a new business idea using attributes of both
- on a card, create a “pitch slide” to sell your idea
- Phase 2:
- Move awesome ideas to the front
- move lame ideas to the back
- Move serious ideas to the left
- move funny ideas to the right
- ideas migrate: you have to sell people on the direction you think it should go.
- it’s a series of pair-wise trades
- Bringing it back to work
- 3 personalities you are likely to meet
- the ghost
- we never follow through
- we get lots of good ideas, and we never do anything with it.
- good ideas die on the vine
- solution
- write things down. create artifacts.
- the ghost thrives in the dark.
- a challenge with distributed teams is that we have no tangible artifacts.
- paper is great technology.
- put the paper on the wall.
- the bad apple
- not while i’m here
- the person who shows up late, question the process, shits on someone’s idea, and then leaves early
- challenge the notion of this prototype
- “creative bad apple”: i’m too creative for this
- “busy bad apple”: i am too busy for this, i have to attend 3 or 4 meetings at the same time.
- solution
- if you have to engage with them, give them something to do. give them a job, a role in the meeting.
- you are going to be the scribe
- plenty of good techniques (p.61 in the book) to insure people don’t dominate the meeting
- make it voluntary: if they don’t want to be there, give them the option to leave.
- the kid
- i wanna play. play fair!
- you’re going to have people who just want to share their point of view.
- solutions
- kids are showing up to this playground, and your job is to ensure they can play well together.
- Try this:
- 4-5 people
- small room, no desks. compress the space – physically intimate
- provocative question
- open it and close it – 90 minutes
- The book:
- Want to try it out with a friendly community?
- VizThink has local communities all over the world, and they are doing gamestorming.
- Questions
- Q: Some cultures are conservative. What can you do?
- You respect their culture, and get them to take a step.
- It may be success just to get them to use post-it notes.
- Stretch their comfort zone just a bit.
- Run a short 20 minute game, and they will have a positive experience. Then you can do on bigger scale.
- most people are grateful just not to be bored.
- Q: When kids that have different learning styles, when you provide them an opportunity to learn it in a way that meets their style, they learn best. if you ask them to do things they are not comfortable.
- It’s a choice between boredom/disengaged and engagement/uncomfortable, it’s better to have engagement.
- gamestorming incorporates all forms of learning styles.
- Most meetings are generally highly skewed by verbal/writing modalities. so the people most threatened by gamestorming are likely to be those really good at verbal/writing. The loss in participation from those people are more than made up by the increase in participation from the people who are normally left out.
Freelancers: You’re Five Products Away From Freedom
- How many of you have products? (many)
- How many of you are making the money you want to make from those products (none)
- PSA: http://sxsw4japan.org trying to raise at least $50k, already at $40k.
- What does freelancing look like?
- my neighbors thinks maybe i am drug dealer.
- the hardest thing I do all day is walk outside in my bathroom.
- $100/month -> $100/week -> $100/day.
- I will probably never be 100% free from clients.
- My first litmus test: can I make $100 per month? If I can do 10 of those, then I’ve got some money that can make a difference. It’s not enough to stop working, but it is enough to turn down the worst clients.
- From there, can I get to $100/week. Because ten of those is $4K/month, which is a substantial contribution.
- And the end goal is $100/day. Then you could stop working.
- What would happen to my business if I went away for two weeks?
- As a freelancer, my business would not survive. I am my business.
- 5 Guys in Kilts
- When you go into the product business, you do things you might not otherwise do. You get a tech support call, and you have to field it.
- You make money all the time. You wake up, and you have five orders. You have money.
- Get Off My Lawn
- http://myerman.com
- http://tripledogs.com
- Written eight books
- Six month lead time
- Do lead to little checks here and there
- “Lead Generation on the Web”: Took two weeks to write. Still get checks from it.
- ExpressionEngine
- Didn’t have a backup utillity
- So wrote one.
- But lots of tech calls about weird problems
- Took about a week to write.
- Went to $100/month almost immediately.
- You must have tech support. Because customers get really pissed if they have to wait even 24 hours.
- Makes money, but not that profitable because of all the support calls.
- Mojo Addons
- Backdoor: Allows the admin to never be locked out by the client.
- Backup: Allows to backup.
- Built horizontally: If you write a backup utility for one platform, then port it to many platforms, and leverage the investment you have.
- You have to write a readme and other support document, because it beats having to answer 50 support calls.
- Report: So, You Think Your Children Are Safe on Facebook?
- $17 ebook on Lulu.
- Not going through any publisher.
- StoryStarter
- iPhone app to randomly generate a protagonist, antagonist, setting.
- Started selling
- Then wrote StoryPrompter, then BlogPrompts.
- Each took less than 3 days to write and ship.
- Got 500 suggestions on what to do next, including things like “include a genre”.
- The real people in the marketplace are the only ones that matter.
- 26-year old Amanda Hockling
- making $2 million per year from Amazon kindle sales
- Publishers don’t give you an advertising budget or any help pushing your book
- 99 cents -> $2.99 -> 99 cents -> $2.99
- You sell at 99 cents, and start to move up
- Then when you are at top of best seller list, you sell at $2.99 to make money.
- When you move back down the bestseller list, you go back to 99 cents.
- Making 10% of 7 dollars from Amazon for a book that sells for $40.
- For a book that sells at $2.99, and you keep 70%, you are keeping more.
- For $97 you can learn how to become a freelance social media manager.
- Let’s Get Social – The Most In Demand Job in the World.
- 43 minute page.
- Lots of content to help sell it, but at $97, it is worth it.
- And then sells access to her time/network, limited to $10/month.
- Basecamp
- WooThemes: WordPress themes for all types of web publishers http://woothemes.com
- design web themes, and then sell them.
- Genesis – web site design. $97. Awesome new web look in an hour.
- “Build Your Own Wicked WordPress Themes”
- Create a teaching business
- Teaching Sells
- Put together workshop materials / interactive learning environment.
- $100 product. You become a member, get access to their goodies.
- If you are an expert in a topic, you can put together the content. It could be about tech, gardening, knitting.
- Clickbank
- affiliate network of products
- you make the product, it goes through the clickbank platform
- other people in the network sell your product and take 50%.
- more than $1B in sales for clients
- Etsy
- crochet pattern: party pet costume
- it is so much easier to sell via etsy than to try to put the internet pages up yourself.
- the time saved can be used to make more products.
- Questions
- Q: Do you have one LLC or many?
- We became an S-Corp a long time ago, and haven’t really thought about it since.
- Don’t attempt any of this as a sole proprietor because there aren’t enough legal protections.
- If there is something not congruent with triple dog media, we might create another imprint.
- Q: What’s this about basecamp?
- 37signals got tired of working with clients. they made basecamp, sold 50,000 seats at $5, and suddenly they no longer had to have clients.
- lesson: productive your process.
- lesson: take something ugly and complex and simplify and beautify it.
- Q: What about piracy?
- O’Reilly says: it’s just the cost of doing business.
- But O’Reilly makes it easy to get the live edition, get updated information. make it easy.
- You can get the o’rielly books via bittorrent, but o’reilly sells a ton of books.
- Q: How do you choose a product?
- it’s got to be viable to generate $100/month
- tiny little group for whom you can totally solve a problem: the number one complaint of authors is that they need ideas.
- Q: What do you think of sites that ask for donations, or kickstart where you ask to raise money?
- I’ve never found that it works to ask for donations.
- Q: You showed several different options for learning products. How do you decide where to distribute? all of them? focus on one?
- I’ve been trying all the different ones, and seeing what works.
- We can launch a book on pay.com, and start getting metrics immediately. “oh, it sells at 2am in japan.”
- The 80/20 rule takes place almost immediately once you start getting data back.
- If you have to find out where the fish are.
- Q: What if you are trying to do video? None of them seem to work very well.
- Yes, you are right.
- You need optimization to convert page impressions to sales. because we can see many cases of 5,000 impressions, and 0 sales.
- video could help with that by showing off the product
- Q: You are apologetic about your UI. But you focus explicitly on your market. You are just a doer. Any advice about what to do if you are more hesistant because we are perfectionists?
- No one will know what you are putting out there. You are not Apple. You don’t have a global brand. Just put it out there. Then you are a doer.
SXSW CloudCamp / DevOps Session Notes 2011
- About
- Managed DNS and email delivery provider
- names to numbers: twitter.com -> 168.143….
- 4 million clients
- 200,000 zones
- 100,000 domains
- 17 global data centers
- Meant for scalability, automation, redundancy
- Why CloudCamp
- rapidly scale
- pay as you go
- API automation
- operational efficiency
- Dynect Platform
- Managed DNS
- Global anycast network
- active failover
- load balancing
- traffic management
- cdn manager
- ui and api access
- SLA – five 9s
- Load balance between your infrastructure and the cloud
- split across amazon, rackspace, your servers
- How it works: when a client comes in from Australia, don’t send them to an U.S. server, route them to the server that is closest via the network.
- Was at Yahoo
- During hurricane katrina Yahoo requests went up to 30,000 requests per second, and stayed there for 6 days. 10x the traffic expected.
- intermittent failovers from various places.
- wheeled in 50 extra machines
- took a day just to setup, even though the machines were already sitting there
- then serving 40,000 requests per second.
- Now with rackspace, you could scale up within minutes: no need to build new machines
- One of the goals is that you have no single point of failure.
- You can’t have one network service provider, one dns provider, one database. You can never have one of anything.
- For Cloud, you can’t rely on a single database server, single file service
- Rackspace is part of the open-stack platform
- open source for cloud computing
- Our goal is to get Rackspace running on the open source platform by the end of the year.
- You can run the openstack.org platform yourself.
- You want infrastructure that makes it easy to scale, easy to deploy, easy to add services.
- Work with 20,000 companies
- Send over a billion transactional emails per month
- your app sends email
- problems:
- deliverability: inbox vs spam folder
- analytics: did the customer receive, open it, click on it
- platform: enhance email from open architecture
- backend problems
- isp rate limits
- blacklists
- dk/dkim
- spf records
- content inspection: ratio of images to text
- “email isn’t fun”
- customers: gowalla, plancast, foursquare
- Integration
- APIs: STMP, Web API, SMTP relay
- receive emails w/ parse api
- pull data w/ event api
- manage sub users
- dedicated ip addresses
- whitelabeling
- Morris Worm 1988 – took down 10% of internet
- Wrote Tripwire at Purdue in response
- Tremendous passion for studying high performers
- started as gene’s list of people with good kung fu
- people who had the best security, best mean time between failure
- codified these practices for Visible Ops Handbook
- Not only in security, but also did work full time doing operations
- and worked as developer
- show me a developer who doesn’t cause problems for ops
- Now more than ever, we need great IT operations
- Organizations are held up by the ability to get features released, get things through operations.
- But it’s not just an operations problem, it’s a development problem and a business problem.
- Put livechat link on their website
- Using 6 lines of code and Tropo service, was able to monitor group chat to send IM
- A carpenter doesn’t weld his own hammer or cast his own screwdriver. He uses an off the shelf tool.
- Why would a programmer write their own email functionality or notify functionality?
- See tropo demo that sends map position information through IM to a particular web browser
- Historical culture of dev and ops fighting
- The cloud is forcing us to have to get along
- You Better Care: It’s about your business
- Devops is Velocity
- The velocity of innovation
- How do we compete today?
- By Scale: Scale of users, scale of data, scale of compute power. Businesses can compete on scale.
- By Velocity of Innovation: How fast can you react to and execute on new market forces or opportunities.
- Doing 30 deploys a day.
- It’s about getting ideas to customers really, really fast.
- How fast can you go from “Ah ha!” idea moment to the “Ka-ching!” cash moment?
- It’s the applicable lifecycle that seperates the two: and dev and ops and test all contribute to the delay.
- Once we move to software as a service, everything we thought we knew about competitive advantage has to be rethought. – Tim O’Reilly: Operations: The New Secret Sauce
- Operations is no longer just the girl friday you bring in once a week to do a deploy.
- It’s critical to the business, to the ability to innovate, deliver value to customer.
- If you have crap operations, you might as well shut down your business.
- New Face of a Rock Star: John Allspaw – VP Technical Operations at Etsy.
- One of the first things you have to do is find one of these guys.
- So What’s Your Culture Dog?
- “We will encourage you to develop the three great virtues of a programmer: laziness, impatience, and hubris.” – LarryWall.
- We became lazy for the wrong reasons: We should have been lazy so we develop automation, so we don’t have to repeat tasks over and over.
- Leadership?
- The good guys are going to leave your company if you aren’t showing leadership
- Wall of Confusion: Dev vs Ops
- Break Down Walls
- Force a breakdown in the wall:
- Take an Ops guy, and put them in Dev
- Take a Dev guy and put them in Ops
- Respect each other
- Enemies are outside the wall: Work together to beat the bad guys outside the wall, don’t fight each other.
- It’s really clear for a small startup
- It’s harder for a big company
- Fearless Culture
- Failure is the New Black: We need to embrace failure. You shouldn’t get fired when you break something.
- At Amazon, they’d have a game day where they’d try to take down a data center. It’s not tested until we have a failure in production.
- Sense of Urgency
- Gotta build a sense of urgency – to be purpose driven.
- “We gotta do this, or we’re going to go out of business.”
- The Etsy guys, they have fun going to work.
- Partners
- We are partners in getting things done.
- When it becomes a question of survival, you had to become partners or die.
- The Smell Test
- From Chris Reid
- If you are in an organization, and you don’t know what that guy does, there’s been a failure.
- Shaman in your organization
- There are the guys who know why that flag is passed on that command line.
- They are the communicators between the people: they are enablers. Everyone comes to find out what is going on.
- Passion:
- When you go to work, are you a guy who presses keys on a keyboard, or are you the great programmer that ever lived?
- Universal pattern that shows up when your organization needs Devops
- Three sets of patterns you can do inside dev and ops
- Benchmarked 1,300 organizations – to link controls and performance
- High performance organizations exist and they are 4-5 times more productive than ordinary organizations
- High performers find and fix security break fast
- Unplanned work comes at the expense of planned work
- Vicious Downward Spiral
- Ops Sees:
- Way too many fragile applications, prone to failure
- It takes too long to find out which bit got flipped
- The problem is detected by a salesperson or customer
- Too much time required to restore service
- Too much time spent firefighting and unplanned
- Planned project work cannot complete
- Frustrated customers leave
- Market share goes down
- Business misses Wall St commitments
- Business makes even larger promises to Wall St
- Dev Sees
- More urgent, date-driven projects put into the queue
- Even more fragile code put into production
- More releases have turbelent installs
- Release cycles lengthen to amortize cost of deployments
- Failling bigger deployments are even more difficult to diagnose
- Most senior constrained IT ops resources have less time to fix underlying process problems
- Every increasing backlog of infrastructure projects that could fix the underlying problems
- Operations inside the Dev/Ops Super-Tribe
- Increase flow from Dev to Production
- Increase throughput
- Decrease WIP
- Goal to create system of operations that allows…
- Zone #1: Decrease cycle time of releases
- Create determinism in the release process
- Move packaging responsibility to development
- Release early and often
- Decrease release cycle time
- Reduce deployments time from 6 hours to 45 minutes
- Refactor deployment process that had 1300 steps spanning 4 weeks
- Never fix forward, instead “roll back”, escalating any deviation from plan to Dev
- Verify for all handoffs (e.g. correctness, accuracy, timeliness, etc.)
- Zone #2: Increase production rigor
- Define what work is and where work can come from
- Protect the integrity of the work queue: e.g. infrastructure/process improvements
- To preserve and increase throughput, elevate preventive projects and maintenance tasks
- Document all work, changes and outcomes so that it is repeatable
- Contact Gene for slides/resources.
- Visible Ops
- Visible Security Ops
- Lean IT
- Web Operations
- NI
- 30 years old, 5000+ employees, mostly engineers
- Robots and stuff: scientific data acquisitions
- Before
- Traditional siloed IT department
- programmers split by business unit
- infrastructure split by technology
- Large complex Web site with dedicated operations team
- 100 programmers
- 6 ops guy (doing support, release, systems engineering, security, performance management)
- Low agility: 6 weeks to get a server
- Uptime problems with complexity and silos
- Grand vision: “Don’t spend a lot of money please”
- The Tipping Point
- NI decided it was time to make some SaaS products
- Some existing product to web integration points, but uncordinated and poorly maintained
- R&D realized they didn’t have web knowledge, started up a new time
- Blessing and Curse
- Everything was new, so we simultaneously developed:
- Team, Process, Systems, Code, Providers, System Automation
- (existing processes oriented around annual software products, not frequent web releases)
- The Team
- We built up our team to fit our role of internal ISV
- Application architect
- System architect
- Operations lead
- 2 developers
- 1 automation developer
- 2 follow-the-sun operations staff
- Work with other product developer teams
- The Process
- Agile
- All systems work used the “developer” tools and systems
- Revision control: Perforce
- Bug Tracking: HP
- Specs and reviews: Atlassian Confluence Wiki
- Task tracking and burndown: JIRA/Greenhopper
- All members collaborate on all aspects of the product
- This was the key to making it work – using all the same tools. We could prioritize better, because it was all in one system.
- There can be a fear that the systems tasks would always get pushed out
- Seemed to be mistaken impression
- But when presented alongside requirements, decision makers seemed to understand the need for systems work.
- The Systems
- Cloud!
- Decided on Amazon EC2
- Needed control and agility we wouldn’t be able to get internally: dynamic requirements, fast scaling
- Needed Linux and Windows both for software
- Currently taking on Microsoft Azure as well
- System Automation
- We built our own: PIE, the “Programmable Infrastructure Environment”
- Looked at Chef/Puppet, and others. What we needed wasn’t quite any of those.
- XML System model defines systems, services, code installs, runtime interaction, variable substitutions
- PIE autobuilds the system from the model: provisioning, software installs, monitoring integration
- Zookeeper as a runtime registry for systems info and eventing
- Allows us to start/stop/control/install/autoscale on bunches of dynamic environments
- We have our dev environment, test environment, production environment – multiplied across our many products. We could deploy a new environment in a couple of hours.
- Code
- All REST-based web services
- Cloud and PIE code mostly in Java, product code mostly in C3/.NET
- The developer must create and deliver the PIE XML code that will build, deploy, monitor their own code. They must deliver the XML along with their code. The developer is the only person who knows what their system downs.
- Providers
- CloudKick – monitoring
- PagerDuty – paging
- DNSmadeEasy
- Postmark – email
- Results
- Win!
- Continuous pipeline of products delivered quickly
- LabView Web UI Builder (http://ni.com/uibuilder) in release
- FPGA Compile Cloud in beta
- One big one in the pipeline and others knocking on our door
- Using cloud, automation, and collaboration through devops, we’ve been able to deliver the apps quickly and continuously.
- Vastly less time that it used to take with the traditional web organization.
- Challenges
- overcoming the thought that it was impossible
- can i do infrastructure tasks using agile? with sprints? by actually trying it, it turns out to be possible.
- write our own software provisioning sounds hard, maybe we can’t do it. but when you try it, you can.
- building trust between dev and ops
- working together in one team and using the same tools really helps. you get transparency. you can’t build the sense of trust when you don’t know what each other are working on.
- various customer dev teams, some globally distributed
- explaining core web performance/availability/management needs to desktop developers.
- Needed to write “why you should log” paper to explain core concepts
- maintaining vision through rapid change
- figuring out how to apply dev concepts to systems: what does it mean to have unit tests for systems?
- Where to Go Next?
- improve testing -> monitoring
- monitoring = lightweight, repeated integration test in production
- culture change is the single most important thing. culture is driven by the demands place on people: if the demand is “don’t spend a lot of a money”, then you get a culture that results from that. if you have them own a product, they operate at a higher level.
- You want people to build Rugged software because they desire the benefits of it, not just because they are scared of auditors.
- Am I Secure?
- latest and greatest vulnerabilities
- Justification of
- What do you think of security people?
- paranoid, jaded,
- It’s an us vs. them mentality:
- dev vs ops, ops vs security, dev vs security
- security professionals often degrade developers
- there is interest across the isle, but often ruined by negative language
- As bad as the ratio between ops and developers is (many dev, few ops), it’s even worse for security: 1 security for 1,000 dev
- “How Complex Systems Fail”: google it, great paper
- Rugged Software Manifesto
- Rugged characteristics:
- Availability
- Longevity
- Scalable, Portable
- Maintainable and Defensible
- Rugged offers affirming values, rather than the Fear/Uncertainty/Doubt of Security
- You can sell Rugged as a Feature
- Using Rugged product labels
- Simple understand of rugged in various characteristics
- Custom lines of code by category
- Libraries used and their ruggedness
- Rugged is Implicit: Customers expect that their money won’t be stolen, their password won’t be intercepted, etc.
- To achieve:
- People
- Sit near the developers: DevOpsSec
- Track Security flaws or bugs in the same bug tracking system
- Security guys that are so outnumbered have to make broad statements like “don’t use PHP”, because they don’t have the time/bandwidth to have a full conversation.
- Recommended
- Visible Ops Security
- Web Operations
- Beautiful Security
Session notes from The SINGULARITY: Humanity’s Huge Techno Challenge at SXSWi
- Doug Lenat
- Using inferences to combine knowledge
- Example:
- Forces pushing us toward Singularity
- Competitive cutting edge apps
- Demand for personal AI assistants
- Demand for real question answering
- Demand for smarter AI in games
- Mass vetting of errorful learned knowledge
- for the common good: wikipedia
- for credit (citation credit)
- for credit (gamification) <- knowledge economy
- Forces pushing us against Singularity
- Large enterprises can stay on top in other ways
- Bread and Circuits: most of us fnd a BTL game
- Pick your favorite technology ender: energy crisis, neo-luddite backlack, ai suicide
- Pick your favorite humanity ender: machines vs people, machines use up all energy/matter/sunlight
- Michael Vassar
- Different concepts that are meant by singularity:
- progress: now has a sort of retro feel, so people are looking for a new word. new institutions, dot com related or similar offshoots are pushing progress in new ways.
- superhuman intelligent systems of any sort: as popularized by vernor vinge.
- the future is unpredictable past a certain point
- the future is predictable up to a certain point by extrapolating trends forward and see how those trends would affect the world.
- vernor vinge has an amazing track record of predictions from CGI to others… but that his method doesn’t work to extrapolate past 2030. So if he has a method that works up until then, and doesn’t work after that, and he has been shown to be right before, then it’s pretty good chance he’s right about the singularity.
- now through 2030 is the entire window of opportunity we have to affect history
- the intelligence explosion: artificial intelligence plus the exponential feedback loops resulting from artificial intelligence.
- The history of human intelligence progress has been characterized by the amount of deliberation.
- Hundreds of years ago, there was very little structured, deliberate thinking.
- A little deliberation goes a long way: even today, even only a small amount of all total thought is focused on deliberation.
- By comparison, computer thought is likely to be far more deliberate.
- Therefore, we are likely to see a very rapid acceleration of intelligence.
- the way to mitigate risk would be to slow down. but if you look at the institutions we have today, such as the free market, collective human will, governments, we have never been able to effect deliberate slow down.
- if traditional information processing continues to improve along with moore’s law, at a certain point everyone improves along with moore’s law.
- Natasha Vita-More
- When Vernor Vinge wrote about the singularity, he wrote about it from science-fiction perspective, but in a basis of mathematics.
- When Transhumanism took on the singularity, they looked at what it meant to extend humanity
- When Ray Kurweil took on singularity, he broadened the scope, and brought it to the mainstream.
- It may not happen, but it probably will.
- It may not happen in one big wall, but in surges.
- The transhumanist movement thinks it will come in surges, rather than one big wall.
- 4 things
- Quality of Life
- Technology Human Enhancement
- Bio-Synthetic Ecology
- Biopolitics: Concerns and consequences of merging more and more with technology.
- Quality of Life
- The Singularity is presumed to be an event that happens to us rather than an opportunity to boost human cognitive ability.
- There is a theory that these things will be available to those that are rich and have the means, creating a further divide.
- But when you look at cell phones, you see that these are widely distributed across all economic and social strata.
- How can we use super-intelligent technologies to solve our pressing problems: access to clean water, food, shelter, education, medical supplies, solutions to genetic problems.
- The Singularity needs smart design to solve problems.
- there is a big gap between the raw intelligence and the intelligence needed to solve human problems.
- Human Enhancement Technology
- Therapeutic enhancements
- We are a biological beings. We can’t just attach technology to us. We need to get busy to understand our brains.
- Selective Enhancement
- Wearing technology, using social identities.
- We have multiple personas and platforms.