DevOps Leadership Series

Derek Weeks

Subscribe to Derek Weeks: eMailAlertsEmail Alerts
Get Derek Weeks: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Article

Applying DevOps to Deliver Quality at Speed

Applying DevOps to Deliver Quality at Speed

Fannie Mae operates in the already heavily regulated financial services/mortgage markets. Add to that it is a government sponsored entity, which means it is laden with many of the policies and regulations of federal government agencies. Plus, they have 6,000 employees and annual revenue of $22 billion. So, they could be a poster child for, "You can't implement DevOps there."

Barry Snyder, DevOps Manager, Developer Platforms at Fannie Mae, begs to differ, which is what he did in his 2016 All Day DevOps talk, How Fannie Mae Applied DevOps to Deliver Quality at Speed. He looks at:

  • Agile and DevOps transformation
  • Doing more with less with greater quality
  • Lessons learned

Fannie Mae started implementing DevOps in 2014, and Barry was in the middle of it from the start. He gave his talk at the end of 2016, and, as he said, they still aren't fully adopted (are you ever), but he gave a status and a comparison between where they were and where they are now.

2012

2016

10 projects using modified-Agile. Essentially waterfall with terminology/concepts from Agile

155 using Scrum/SAFe and growing

9 to 18 months to release to production

Projects releasing every month

9 governance gates/inspect 100% before release to production

1 working group, 1 CAB, inspect 10% after release to production

2-4 months to provision dev and test servers

Minutes to provision dev and test servers

300-400 builds per month; 900-1200 deployments per month

3-4K builds per month; 12-15K deployments per month

Testing took 50% the time it took to construct the solution

80% savings in testing time and executed within the sprint

No means to measure and evaluate the productivity from release to release

Productivity increasing by an average of 30%-40% with costs going down by 30%

barry1.png

According Barry, these results are being sustained - they are not just a blip. To get there, they had to get an appetite for risk. Overcoming that hurdle required them to realize when you do 100% risk mitigation, you are going to spend more time avoiding risk than it would take to mitigate something.

They are also executing Application Lifecycle Management (ALM), a step up from CI/CD. They look at all of the tools and processes along the lifecycyle, peer reviews, automated building, testing, etc. They are implementing automated processes with a variety of tools to achieve this.

As an example, at Fannie Mae, it previously took 5-6 weeks to request a library, now it is immediate because it is automated through Sonatype's Nexus.

barry2.png

As we hear from all practitioners, it is a journey, not a sprint (pun intended). Fannie Mae started at the end of 2014, and they are still in progress. They went from 9-18 months per deployment to a month on a consistent basis. They are close to always deploying during sprints, and are working towards having an idea, writing code, and releasing it into production.

Barry presented lessons learned:

Leverage what they had while shifting left:

    • Release Management Infrastructure (evolved into our CI/CD)
    • VM provisioning infrastructure (evolved into our RapidIT)
    • Test Automation
    • Code Quality
  • Recognize it is a culture shift in how work is performed:
    • Agile Adoption "You do this for the people, not do the people.""We started on the IT side, then the business side saw the value add."
    • Automate Oversight/Transparency - You can't do this manually. With 120 apps, there were 8,000 libraries not accounted for.
    • Portfolio Analysis (Risk Reduction, Quality, ROI, Cost Reduction)
    • Application Quality is a Supply Chain
    • Trust developers and get out of the way
  • Measure Results
    • Shifts in quality
    • Adoption rate
    • Accelerated delivery
    • Benchmark productivity and associated productivity cost

You can watch Barry's entire talk online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year's event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It's all free and online on October 24th.

More Stories By Derek Weeks

In 2015, Derek Weeks led the largest and most comprehensive analysis of software supply chain practices to date across 160,000 development organizations. He is a huge advocate of applying proven supply chain management principles into DevOps practices to improve efficiencies, reduce costs, and sustain long-lasting competitive advantages.

As a 20+ year veteran of the software industry, he has advised leading businesses on IT performance improvement practices covering continuous delivery, business process management, systems and network operations, service management, capacity planning and storage management. As the VP and DevOps Advocate for Sonatype, he is passionate about changing the way people think about software supply chains and improving public safety through improved software integrity. Follow him here @weekstweets, find me here www.linkedin.com/in/derekeweeks, and read me here http://blog.sonatype.com/author/weeks/.