Chef + Docker: Automating Container Workflows

More than likely most of you are familiar with Linux containers but let’s briefly review what the buzz is about.

Application containers are an operating system feature that allows you to run your app in an isolated environment without the need for a separate kernel. They’re kind of like a mini-VM without all the overhead.

Containers give you a great way to start and stop applications and control the resources they use. They’re also easy to deploy, especially with tools such as Docker.

Chef is the first (and only, as far as we know) IT automation platform to show how to incorporate containers into an industrial strength workflow. If you attended #ChefConf last week, you saw Mandi Walls deliver a delightful demonstration of how Chef can automate the creation, management, and monitoring of Docker containers.

Check out Mandi’s demo, below. You’ll see how:

  • Chef creates a container image.
  • Chef provisions and configures a Docker host environment.
  • Chef launches container instances.
  • Chef configures, monitors, and manages the new containers while they run.
  • Chef lets you analyze what you’ve done using a new feature called actions.

The result is a process that’s versionable, testable, and repeatable. Using Chef with Docker helps you build a container workflow that’s ready for production and automated build environments.

To be clear, we’re still in the early stages, but we’re working hard on moving these capabilities forward and are pumped about what we’ll deliver in the future.

Chef Development Kit

Ohai Chefs,

The first version of Chef Development Kit (a.k.a. Chef DK) is here.

What is Chef DK?

Chef DK is a package that contains all the development tools you will need when coding Chef. It combines the best of the breed tools developed by Chef community with Chef Client.

Here is what you can do with Chef DK:

  • Get your cookbook dependencies under control and have a sane way of composing the cookbooks you need with the hot new Berkshelf 3.0
  • Take advantage of built-in testing with the de-facto lint tool for cookbooks FoodCritic, cookbook unit testing framework ChefSpec & the leading integration testing framework for coded infrastructure Test Kitchen
  • Easily setup and upgrade the Chef Client on your workstation
  • Get introduced to the brand new Chef workflow tool called chef

The community developed tools that you have known and have been using for a while are now housed in an official Chef product.

What’s new?

New workflow tool: chef

One of the strongest feats of Chef is its flexibility. It grew organically adapting to the many different workflows our users needed to run the technology stack behind their businesses. As a side effect of this organic growth, the new comers to Chef has faced with different ways of doing a thing with individual pros and cons. This decreased the approachability of Chef.

With Chef DK this paradigm is being broken. Chef DK includes a brand new tool called chef. The design goal of this tool is to “Streamline Chef Workflow for all”. Chef DK will offer new users a streamlined workflow, while maintaining the powerful flexibility for advanced Chef users.

We have a long way to get to this goal. But some of the things we can see when we get there may be:

  • Configurable generators that support the commonly used cookbook patterns
  • Deep integration between Berkshelf, Knife and Chef Client
  • Builtin development and test environment provisioning in the cloud

With version 0.0.1, we’re taking a small step in this direction with two small features:

  • chef gem: Easily install gems or knife-plugins into your Chef DK setup.
  • chef generate: Minimalistic cookbook generators powered by Chef.

PS: A couple words on the naming… We’ve always find it surprising that when you install Chef, you do not get any tools named “chef”. For a while we’ve considered calling one of the tools we have developed “chef” but never found the perfect match. Given that this tool is going to be the primary tool that our users will interact with in the future; if a tool is ever going to be called “chef” it would be this tool. Hence the name “chef” :)

Continuous Delivery

We are fans of Continuous Delivery. We believe it’s a significant competitive advantage. Also what’s the point of waiting for 2-3 months to deliver a bug fix or a new feature?

We have been working closely with our Awesome Release Engineering (ARE) team to build new continuous integration clusters which will be able to deliver well tested builds of Chef DK to you daily. This means you will be able to see your contributions in Chef DK the day after they’re merged.

More details to follow after ChefConf about this topic.

Built-in Performance

We are also fans of speed. Who wants to wait for a couple of minutes for their cookbook dependencies to get solved, or some 10 seconds after running knife -v?

When developing Chef DK, we’re taking performance as a primary consideration. Having a responsive tool at your workstations is the first step in having a delightful user experience. Here are couple of things in Chef DK that makes it more performant:

gecode in Berkshelf

The first major collaboration effort between Berkshelf Team and Chef has been on the dependency resolution. Berkshelf 3.0 ships with gecode as its dependency solver. You will hear a lot about this during and after ChefConf. But in a nutshell, this effort ensures that dependencies are resolved faster and in agreement with the Chef Server (which also uses gecode as dependency resolver).

How fast? Here are some numbers (in seconds) comparing gecode with pure ruby implementation while resolving Chef Inc. platform cookbook dependencies:

Screen Shot 2014-04-14 at 10.09.35 PM

appbundler

appbundler locks down an application’s dependencies to the versions selected by bundler using a Gemfile.lock. This way applications run faster because rubygems doesn’t resolve the dependency constraints at the runtime. This also protects the application from incompatible dependencies. All of the binaries included in Chef DK are using appbundler. You can read more about it here.

Platform Support

As being a workstation tool, Chef DK will be supported in all the common workstation platforms. The first version of Chef DK is built and tested on:

  • Mac OS X 10.9
  • Ubuntu 12.04
  • Ubuntu 13.10
  • RHEL 6

Yes we’re missing Windows. And yes it’s the next thing in our to-do list. Support for Windows 7 and windows 8.1 is on its way.

The Future?

The ambitious goal to “Streamline Chef Workflow for all” is really making us bite our nails. Thankfully we have a strong community that will help us get to this goal.

Our next big step towards this goal will be to prototype the streamlined Chef Workflow, discuss, work & iterate on it with you. We will have public online discussions in the near future. But this week, all of us are in San Francisco @ ChefConf2014. Feel free to grab anyone with a Staff T-shirt and say “I want to talk about this streamlined Chef Workflow” and let’s talk.

Finally, as usual Chef DK is Open Source. Feel free to open a Github issue or send a PR. Yes not a typo, I meant to say Github issue. We’re working on our contribution and issue tracking processes to make it easier to contribute and send us feedback. Stay tuned for more updates on this but feel free to use Github issues on chef-dk project.

Awesome Chefs

Thanks for hanging on and getting to the end of this long post. A lot of good work went into Chef DK. Even though it’s not possible to list all the names here, here is a courageous attempt to do so (Apologies in advance for any name that’s forgotten).

Grab Chef DK here and as usual feel free to reach out if you have any questions or seeing any issues.

– Chef Client Team: Dan DeLeo, Lamont Granquist, Claire McQuin & Serdar Sutay

Leading Enterprises Embrace Web-Scale IT Automation with Chef

Chef Continues Rapid Growth, Strengthens Partner Ecosystem, and Broadens Community Footprint to Help Companies Thrive in Today’s Digital Economy

SEATTLE – April 14, 2014 – Chef, the leader in web-scale IT automation, today announced that this week’s #ChefConf 2014 will unveil a software and services ecosystem that helps businesses move faster and meet customer demands. The company today also unveiled new partnerships and product enhancements that make it easier for enterprises of all sizes to automate through code and standardize on the Chef platform.

#ChefConf attendance reflects Chef’s blue chip customer portfolio. Keynotes include presentations from GE Capital, Nordstrom, Target, and Yahoo. Nearly 70 percent of Chef’s total sales are from Fortune 1000 companies.

Today’s economy is a digital one. Enterprises must transform their operations to deliver services at speed in order to delight customers and achieve critical competitive advantage. Automation enables developers and IT operations teams to collaborate effectively and deploy software and services more rapidly and at scale. Chef is at the center of this movement. Chef helps enterprises achieve web-scale IT.

The foundation of Chef – the Chef Community – will come together at #ChefConf with the common goal of promoting IT automation in the enterprise.

#ChefConf 2014 News Highlights:

Exponential sales growth: Chef’s 2013 total sales grew 188 percent year-over-year, demonstrating robust demand for Chef’s IT automation platform and its expertise in DevOps skills and adoption patterns.

Massive open source footprint: The Chef Community includes tens of thousands of registered users, with millions of Chef downloads, and thousands of contributors, which demonstrates Chef’s wide adoption and influence across many industries. Chef is building a new generation of IT skills and tools that are turbocharging businesses.

Technology ecosystem unites at #ChefConf 2014: Chef’s ecosystem is expanding as companies including Amazon Web Services, Docker, Google, HP, IBM, Juniper Networks, Microsoft, Rackspace, VMware, and others collaborate to empower enterprises to accelerate software delivery and enable enterprises to win in today’s customer-driven economy.

Chef platform expands: Chef continues to extend its open source and commercial automation platforms. The new features accelerate software delivery and simplify infrastructure management for organizations of all types. Featured at this week’s #ChefConf are these platform enhancements:

  • Ease of Use:
    • Download and Management: Getting started with Enterprise Chef has never been easier. With the latest version of Chef’s commercial platform, a simple download installs and starts the Chef server, making it quicker than ever to access and make the most of Chef.
    • Training: Chef’s always growing #learnchef library features even more extensive resources for getting productive with Chef in no time. Chef also offers a comprehensive suite of online training videos that guide novices and experts alike through all the operational skills needed to stir up delight with Chef.
    • Chef DK: Chef DK – Developer Kit – consolidates open source components into an easily installed package that provides a best practice tool chain for building Chef workflows. Featuring Berkshelf, Chef Spec, Food Critic, and Test Kitchen, the first version of Chef DK leverages the best of open source technology to make users productive with Chef in the least amount of time possible.
  • Patterns of Enterprise Success:
    • Chef actions: Chef actions is a new feature of Enterprise Chef that provides users with visibility into all activity on the Chef server. Chef actions delivers notifications on who is changing what on the Chef server and allows administrators to track cookbook usage, roles, environments, and changes to infrastructure, all through an easy-to-use dashboard. Combined with Chef reporting, Chef actions delivers the most comprehensive insight into IT infrastructure of any automation platform available today.
    • Chef metal: Chef metal delivers policy-based provisioning that allows you to automate entire clusters of machines with the same approach you use to configure a single node. Chef metal automates the provisioning of infrastructure at any scale.
  • Community Engagement:
    • Supermarket: Chef has open sourced its community site to provide the entire Chef Community with the means to build the best open source community resource in the industry. By opening all of Supermarket to the Community, any organization can leverage this collection of code and best practices in order to create its own community resource.
  • Partner Platform Support:
    • Networking Automation: Chef recently released full integration with Juniper Junos OS for streamlining the configuration of networking infrastructure. In addition, Cisco’s 3000 and 9000 series switches feature full Chef integration for automating both network port and server configuration from the same platform. Chef also integrates with Arista and Plexxi’s networking platforms.
    • For full information on Chef’s support for partner platforms, including AWS, Google, IBM, Microsoft, Rackspace, and VMware, see this separate news release.

Supporting Quotes:

“Many of the world’s leading companies and technology providers are betting on Chef as the de facto standard for web-scale automation in today’s IT-powered economy. Our accelerating sales and community growth are built on a strong foundation of customer and partner support, enabling us to deliver speed and scale for forward-leaning enterprises and web innovators alike.”

-        Barry Crist, chief executive officer, Chef

“The Chef community is at the heart of Chef, driving our success and continued innovation. We’re standing united with our Community on a common goal – delightful customer experiences – and whether you’re dev, ops, or something else all together, we hope you’ll join us in making IT better for everyone.”

-        Nathen Harvey, Director of Community, Chef

Additional Resources:

New Research Reveals Notable Economic Influence of Developers in Business and Society

Survey from Chef Shows Majority of Software Developers Feel Empowered to Experiment and Will Be a Revolutionary Influence in the Next Five Years

SEATTLE – April 14, 2014 – Chef, the leader in web-scale IT automation, today released results from a sweeping new survey  that indicates the emergence of a power class of developers. The findings show that developers are starting to be recognized as a highly influential population both in business and in society. An overwhelming majority of developers – 93 percent – feel empowered to experiment in their companies, and 94 percent believe they will be a revolutionary influence in the private sector, government and non-profits during the next five years.

News Highlights:

Chef surveyed 1,000 software developers in the U.S. to determine trends in their business, societal, financial, and political behavior. Key findings from the survey include:

  • Staying Power: Developers are invested and committed to their companies and see longevity in their careers. There is a common perception that those involved in software development are constantly changing jobs. Despite the wealth of opportunity available to them, Chef’s survey found that developers are career-monogamists, and not constantly searching for the next big thing.
    • The average software developer plans to stay at his or her current company for nine years.
    • Seventy percent of developers say they plan to stay at their current company for five years or more and 25 percent plan to stay for more than 10 years. Further, nearly half of them (49 percent) plan to be at their current company for seven or more years.
    • Eighty-two percent of software developers report they are more satisfied with their jobs than their peers who are not developers.
  • Economic Power: Developers are a stable class and are the engine powering our economy today and in the future. Despite the ups and downs of technology companies, the developer population remains stable, maturing and growing in size, influence, and financial power.
    • More than two-thirds (69 percent) of developers describe their profession as “recession-proof.” Developers see growth opportunities across industries including healthcare, manufacturing, and education.
    • Ninety-four percent expect to be a revolutionary influence in major segments of the economy during the next five years.
    • Eighty-four percent of developers feel they are paid what they are worth and more than half believe they will be a millionaire at some point.
    • With their rising importance in the workplace, 66 percent expect to get a raise in the next 12 months.
    • Three out of four developers feel they are more financially secure than their parents were at their current age.
  • Knowledge is Power: Developers are pragmatic and understand the value of both technology and government. While longstanding stereotypes misrepresent developers’ as withdrawn from business and society, the research reveals developers are informed and engaged participants in social and civic activities.
    • Developers nearly equally value the political power of technology (51 percent) and government leaders (49 percent).
    • Seventy-one percent of developers participated in political and civic activities in the last 12 months.
    • Developers find time to give back; the average software developer spends 50 hours per year volunteering.
    • Developers understand the demands for their skillset; 53 percent feel their time, for example donating coding skills, is a more valuable contribution than their money.
  • Empowered: Developers feel valued and empowered by their companies and in their profession. Viewing developers as tactical executioners is a thing of the past – developers understand they are at the epicenter of today’s digital economy and are qualified to initiate ripples of change throughout the enterprise.
    • Coders aren’t just taking orders. Ninety-three percent of developers frequently feel empowered to experiment and suggest changes to business processes, products, or services.
    • Although organizations are looking to developers to provide and foster innovation to drive growth and profits, the barrier to innovation may be this same need for growth and profits. Seventy-nine percent of respondents said that the pursuit of near-term profits may be holding their companies back from making long-term investments in innovative and unproven solutions.
    • Ninety-five percent of developers feel they are one of the most valued employees at their company and 89 percent say their company’s leaders see them as essential.
    • Developers feel that a talented software developer has more power to change society than a talented public speaker (63 percent versus 37 percent).

Supporting Quotes:

“Developers are the growth engine powering enterprises today. This research demonstrates this class is evolving as powerful, connected influencers who are mobilizing to shape more than code across business and society. We look forward to continuing to work with developers to equip them with the tools, resources and community support needed to thrive in this transformational period.”

-        Barry Crist, chief executive officer, Chef

“As the more and more of the economy shifts to being delivered online, software developers and systems administrators are uniquely suited to help businesses embrace this change. Rather than standing in the background and keeping the proverbial lights on, they are vaulting forward – to the front line of business development, strategy, and operations.”

-        Adam Jacob, chief technology officer, Chef

Additional Resources:

Tech Leaders Showcase Chef Support at #ChefConf 2014

Chef is Collaborating with AWS, Google, HP Cloud, IBM, Juniper, Microsoft, Rackspace, VMware, and Many More to Define how Businesses Deliver Software and Services to Customers

SEATTLE – April 14, 2014 – In advance of #ChefConf 2014, Chef, the leader in web-scale IT automation, today announced that the company is working with a diverse group of leading technology providers. Their goal is to enable enterprise businesses to better deliver their goods and services to customers. Chef’s comprehensive partner program provides technology partners with a direct route to tight integration with Enterprise Chef. The program includes joint marketing and sales opportunities and access to extensive training resources. Through a series of collaborative engineering, marketing, and sales efforts, Chef and its technology providers are turbocharging businesses.

Enterprises must move at a pace that anticipates customer expectations. Developers and IT operations teams need tools that allow them to satisfy consumer demand, and to react to changes in real time and at scale. A February 2014 Gartner report entitled, “Strategic Technology Trend: Web-Scale Singularity Means Goodbye to Conventional IT Wisdom,” (Cameron Haight) states that “The influence of DevOps on IT culture, tools, processes and organizational structure is resulting in the acceleration of application delivery and an environment of continuous experimentation.We believe this statement reinforces the fact that enterprises are looking to DevOps to help them move at ever-increasing speed.

Ecosystem Momentum:

With the enterprise demanding more speed and scale, Amazon Web Services (AWS), Google, HP, IBM, Juniper Networks, Microsoft, Rackspace, VMware, and others are working with Chef to empower enterprises by providing ways to achieve faster software delivery and to improve collaboration between development and operations teams, all in the name of ensuring delightful customer experiences.

-        Amazon Web Services: AWS uses Chef as the default automation engine for its application management solution, AWS Opsworks. Chef is also working with AWS Premier Consulting Partner, 2nd Watch, to deliver comprehensive support and services to customers by using Chef with AWS.

-        Google: Chef integrates directly with Google Compute Engine (GCE) to provide robust configuration management, resource provisioning, and application deployment capabilities for GCE customers. For technical information on using Chef with GCE please go here.

-        HP Cloud: Chef provides full integration with HP Cloud Services to deliver full stack infrastructure automation – from server provisioning and configuration management to continuous delivery of infrastructure and applications. Here is information on HP’s OpenStack™ technology-based public cloud.

-        IBM: Chef integrates with IBM SmartCloud to automate configuration management and application delivery in IBM’s OpenStack-powered cloud. IBM has released cookbooks for both SmartCloud and its WebSphere Application Server Liberty Profile. These provide reusable code for rapid resource provisioning and full application lifecycle management. In addition, Chef recently released test code that supports the IBM AIX operating system.

-        Juniper Networks: Chef recently released full integration with Juniper Junos OS for streamlining the configuration of networking infrastructure.

-        Microsoft: Chef provides a single platform for automating Windows and Linux systems on-premise and in Microsoft Azure. By delivering native integration with PowerShell and the Microsoft Azure cloud portal, Chef enables Windows users to easily manage on-premise workloads and to streamline migrations to Microsoft Azure. Microsoft Open Technologies has released its own collection of Chef cookbooks that offer the Chef community rock-solid code for automating the provisioning and management of compute and storage instances in the Microsoft Azure cloud. For detailed information on using Chef with Microsoft Azure, please watch this demo.

-        Rackspace: Rackspace is using Enterprise Chef for its DevOps Service. The Rackspace DevOps Automation Service now offers Enterprise Chef to automate the scaling of its hybrid cloud infrastructure and applications.

-        VMware: Chef integrates with VMware’s vCloud Automation Center (vCAC) to deliver self-service application deployment in vCloud environments. With Chef and vCAC, users can easily automate the configuration and delivery of resources and applications in vCloud environments.

Partners Talk Chef at #ChefConf:

#ChefConf 2014 features a comprehensive collection of technology providers speaking about Chef integrations and collaborations. Partner presentations include:

-        Jonathan Weiss, AWS, “AWS OpsWorks under the Hood”

-        Niek Bartholomeus, BMC, “Orchestration in Meatspace”

-        Nick Stinemates, Docker, “Chef and Docker”

-        Eric Johnson, Google, “Cooking with Google Compute Engine”

-        Moe Abdula, “Delivering at the Speed of Tomorrow with Cloud” (Keynote)

-        Tim Puyer, IBM, “IBM DevOps for Cloud Delivers Smarter Apps at Market Speed”

-        Mark Russinovich, Microsoft, Keynote

-        Jeff Mendoza, Microsoft, “Cooking in the Cloud: Chef and Microsoft Azure”

-        Matt Barlow, Rackspace, “Managing Many Customers, Many Chefs, and Tons of Cookbooks”

-        VMware, “VMware and Chef: How to Build, Deploy & Manage Modern Applications”

Supporting Quotes:

“The tremendous support we’re receiving from the most forward-leaning of today’s technology providers further validates Chef’s pole position as the de facto standard for web-scale IT automation. We appreciate so many great collaborators investing time and resources into Chef to help businesses move at the speed of customer demand.”

-        Ken Cheney, vice president of business development, Chef

Chef and Rackspace Collaborate to Expand Reach of Enterprise Chef

Open Cloud Leader Bakes Enterprise Chef into New DevOps Service, Empowering Users to Easily Deploy and Scale Applications

SEATTLE – April 14, 2014 – In advance of #ChefConf 2014, Chef, the leader in IT automation, today announced that Rackspace (NYSE: RAX), the open cloud company, is using Enterprise Chef for its DevOps Service. Rackspace DevOps Automation Service helps developers and IT operations teams automate the scaling of hybrid cloud infrastructure and the speedy deployment of fast-growing applications. As part of this service, Rackspace now offers Enterprise Chef as the automation platform for managing both your infrastructure and applications.

As enterprises today look for innovative ways to keep pace with the demands of customers, many turn to IT automation to continuously build, test and deploy new features and applications. The Rackspace DevOps Automation Service enables developers and IT departments to accelerate time-to-market by allowing them to deploy, test, and scale new configurations in minutes rather than days. Rackspace customers can use Enterprise Chef to automate the creation of development, staging, and production environments in Rackspace’s open, hybrid cloud. Enterprise Chef provides a shared repository of code for automating resources and applications. With it, development and operations teams can collaborate and move at the speed of customer demand.

News Highlights:

Rackspace using Enterprise Chef as part of its DevOps Automation Service is an important benchmark for IT automation:

-        Enterprise Chef sets the standard for cloud automation: Enterprise Chef allows you to interact with the Rackspace cloud, OpenStack, and PaaS environments via the knife command line, making it an ideal automation platform for cloud infrastructures. Enterprise Chef can scale to accommodate up to10,000 nodes under the management of a single Enterprise Chef server, which maximizes resource performance and investment for any cloud deployment.

-        Empowering developers is critical: As IT becomes the critical route to market in every industry, fast deployment of applications must be IT’s highest priority. Enterprise Chef provides IT operations and development teams with an automation platform capable of improving collaboration and accelerating software delivery.

-        Chef is building a new generation of IT skills: Chef and Rackspace both recognize that there is a gap between current IT skills and the skills demanded by the widespread adoption of large-scale compute infrastructures and their associated complexity. By providing tools and services that help develop new automation and collaboration skills, Chef and Rackspace are helping IT pros build better infrastructures, applications and careers.

Supporting Quotes:

“Enterprise Chef is an important part of our DevOps Automation Service, which is comprised of the same tools and best-practices that enabled us to push code to production in our public cloud more than 2,500 times last year. Working with Chef, we’re providing these same capabilities to customers so they can increase business velocity by leveraging our specialized expertise and creating new skill sets within their own organizations.”

-        John Engates, chief technology officer, Rackspace

“Rackspace shares our vision of an empowered IT workforce using DevOps strategies and automation technology to create delightful customer experiences. By making Enterprise Chef a core pillar of their DevOps Service, Rackspace is delivering on this vision and helping validate Enterprise Chef as the standard for automation in the cloud.”

-        Ken Cheney, vice president of business development, Chef

Additional Resources:

 

Postmortem: Chef Client Regressions and Heartbleed

Ohai Chefs,

Last Tuesday 04-08-2014 7:12 PM, we released Chef Client 11.12.0 which contained 3 regressions (CHEF-5198, CHEF-5199 and OHAI-562). Subsequently, we released a new version of Chef Client which addressed these issues on Wednesday 04-09-2014 6:28 PM. However, during the 24 hours in between, users of Chef were not able to:

  1. Mitigate the Heartbleed vulnerability.
  2. Make a Chef run which includes a file download from servers serving gzipped content that do not use chunked transfer. (A very common scenario)
  3. Reload custom Ohai 6 plugins via the ohai resource. (Particularly impactful to the users of the nginx cookbook.)
  4. Upload a cookbook that has return statement in it.

I’m aware that you rely on Chef in extremely critical situations. I’m sorry that Chef Client was not able to live up to your expectations for this time window especially during the critical Heartbleed vulnerability response.

In this blog post I’ll share the results of our postmortem about this incident.

Events & Response

Based on our original plans, the Chef Client 11.12.0 was expected to ship on Friday (04-11). However, with the announcement of the Heartbleed vulnerability, and in order to deliver the mitigation for this vulnerability more quickly, we pulled this release forward by 3 days.

CHEF-5198 was filed on 8:00 PM on Tuesday. At that moment we were working on the 10.32.0 release again in the context of the Heartbleed vulnerability. We’d seen some signs of the regressions but it was not until Wednesday morning that we were able to regroup and start investigating the reports. We responded to the filed issues on 9:48 AM Wednesday morning. At this time we had reproduced the test cases and started working on fixes for the issues.

At 1:00 PM we finished local testing of the fixes and hit the build button in our Continuous Integration (CI) pipelines. Due to some stability issues, we were only able to deliver this release to you in 5.5 hours at 6:30 PM.

Mistakes and Remediations

Vulnerability Mitigation Releases

The first mistake in the chain was to ship 11.12.0 in response to the reported vulnerability. If we had made a patch release (i.e. 11.10.6) with only the mitigation of the vulnerability, the first impact we caused would have been mitigated. We made this choice for speed as we were pretty confident on 11.12.0 and it also included mitigation for another vulnerability (libyaml (CVE-2014-2525)).

In order to prevent this from happening again moving forward as a policy we will make smaller scoped patch releases while shipping mitigations for security vulnerabilities.

Extensive Test Coverage

Obviously if we’d had test coverage for the specific regressions we encountered, all of the errors would have been prevented. We will be filling the exposed holes in our functional test suite; however given the variety of platforms Chef Client is running on and the depth of its functionality, we will continue to have holes in our functional test coverage.

This incident made us re-think our testing strategy for Chef Client and as a result we’ve decided to bump the priority of one of the improvements that we have been discussing for a while: “End to End integration tests for Chef Client”.

We will be working on building tests that will cover the common end to end scenarios and ensure that they are working in all of our releases. Ideally we would like to hook these tests into Travis as well so that they will also automatically run for your contributions as well.

One of the other major gates before our releases has been “Dogfooding”. We use release candidates of our software as much as possible. We update our pre-production infrastructure with all the Chef Client Release Candidates. However in 11.12.0 release we have skipped this step since we needed to cut off access to our pre-production infrastructure because of Heartbleed.

In order to help with the test coverage, moving forward we will increase the depth of our dogfooding efforts.

CI Infrastructure Stability

Lastly, taking 5.5 hours to release a build is unacceptable to us. Our awesome Release Engineering team have been re-engineering our CI infrastructure and building a world class Continuous Delivery system which will enable us to ship daily Chef Client releases with high confidence. We will talk more about this at ChefConf.

In the meantime in order to improve our response time in incident like this in the future we will be taking below steps to improve our CI infrastructure:

  • Improve the stability of our CI cluster with Chef.
  • Improve our Solaris testing speed with better hardware.
  • Investigate intelligent ways of running tests. (For instance, we don’t need to run the full knife test suite if the Solaris package provider is changed.)

Summary

Again, I’m sorry about the regressions and delaying our response to the Heartbleed vulnerability. We let you down at a critical time.

In addition to continuously striving to improve our incident response, we will take these steps to prevent the same things from happening again:

  • Vulnerability mitigation release policy
  • End-To-End Chef Client integration tests
  • Even more dogfooding
  • Stabilization of our existing CI cluster while building a new cluster for continuous releases

As usual we’re listening anytime you want to reach out to us:

Chef IRC, Chef-Hacking IRC, Chef Mailing Lists, File an issue for Chef

Update on Heartbleed and Chef Keys

Ohai Chefs! As most of you are already aware, the OpenSSL Heartbleed bug has exposed a giant hole in the security of the Internet over the past few days. Yesterday we released updates to the Chef Server (both Enterprise and Open Source) to address this bug, and we provided instructions on how to properly upgrade an affected server to address the compromised components of your Chef infrastructure. At that time, we incorrectly instructed you that the client keys used to authenticate with the Chef Server were safe:

Chef does authentication and authorization by signing each request, so you don’t have to worry about regenerating your client credentials.

This statement is incorrect, and I’ll tell you about the ways in which your client keys could have been compromised.

Heartbled Server – Key Generation

By default, the Chef Server generates the client private keys for all of the nodes in your infrastructure when they first register themselves. This private key is not persisted to the Chef Server, but it is transmitted across the network over a secure connection. In this case, that secure connection was provided by HTTPS using the OpenSSL library. Just like any other sensitive information in your Chef infrastructure, it is possible that this data was compromised by the Heartbleed bug. Since the private key is only transferred once, the chances of this information leaking is much lower than other objects, but that chance is still non-zero.

Heartbled Client

Another scenario in which the Heartbleed bug could potentially expose the client private key is when the Chef Client is connected to compromised server. A compromised server could be the Chef Server or any number of services that the Chef Client connects to, for instance when using the remote_file resource. In this case, a compromised server would be able to send a Heartbleed request back to the client and get access to 64KB of memory from the client. This memory can potentially contain the client’s private key.

Private Key Regeneration

Depending on your comfort level with the defense around your Chef Server, you may want to regenerate your client private keys. We’ve published a cookbook that automates the regeneration of client private keys. You can find it on the community site. Alternatively, you can perform this action via the Manage Console (both Enterprise and Open Source) for each individual client.

WARNING: If you use a tool such as chef-vault to encrypt sensitive information out-of-band, and that out-of-band encryption method relies on the public/private keypair of your client, you will need to re-encrypt those new secrets with your tooling of choice.

As we learn more about the Heartbleed bug, we’ll continue to update the community on the status and any additional steps needed to secure your infrastructure.

Management Console Enterprise Chef Add-on 1.3.1 Released

Hello!

We are pleased to announce the release of a new version of the Management Console for Enterprise Chef 11, 1.3.1. Please contact support@getchef.com for details on obtaining this release.

Security Updates

  • Update OpenSSL (CVE-2014-0160)
  • Update Rack::SSL (CVE-2014-2538)
  • Update libyaml (CVE-2014-2525)

About the OpenSSL Update

OpenSSL was updated to 1.0.1g to address The Heartbleed Bug. Management Console was not directly affected by this bug (it runs behind Enterprise Chef’s Nginx) but we’ve updated the dependency on OpenSSL as a precaution.

When running this release of the Manage Add-on you should also be running at least Enterprise Chef 11.1.3 and have followed the instructions to regenerate your SSL certificates linked here and in the EC 11.1.3 announcement.

Improvements

  • Log exception messages and backtraces
  • LDAP system recovery
  • Create validation key on starter kit download if none exists
  • Allow sign up without creating an organization
  • Improve starter kit download error messages
  • Separate sign in and sign up pages
  • Allow sign up to be disabled (Use the disable_sign_up configuration option. See the docs for more information.)

Bug Fixes

  • Fix regression in 1.3.0 where sign up button could be hidden
  • Make it so navigation does not disappear when window is too narrow
  • Fix for invalid usernames on password reset

Let us know if you have any questions or concerns,
Nathan Smith, Front End Team Lead

Companies Must Re-Build and Re-Design Their Businesses to Automate

I’m continuing my blog series on enterprise IT being too slow for business today. There are no half measures when it comes to automating a company’s technology. This post explains why.

Automation is a prerequisite for success today. There is simply no way of getting around it.  Those who master automated delivery of IT services are going to win.

But to automate properly, companies must re-build and re-design their businesses; that way, they can make the countless daily technology changes needed to keep up with customers. And this is only possible through the continuous delivery of code.

In a world where traditional everything is under attack – Amazon’s continued onslaught against brick and mortar retail and Uber’s competition with taxi cab services are two notable examples – businesses must make the change to continuous delivery of code now, or risk a slow death over the long-run.

Automation is an obvious solution to today’s speed challenge. This lesson goes as far back as the Industrial Revolution. Technology can perform some work at a rate that is impossible for humans.

And yet, according to Forrester data, nearly half the typical enterprise IT budget (47%) is spent on people. And almost 3/4 of that (72%) is spent on ongoing operations – work that is now overwhelmingly manual in most organizations. This means that approximately 1/3 (33.8%) of IT budgets now go to functions that are probably too slow.

Keeping that in mind, let’s look at three key categories for automation tools.

A task execution tool automatically performs a task that would otherwise require manual action; a process flow tool codifies processes; and a decision trigger tool helps determine when to take action, as opposed to the first two tools, which focus on how to act.

Task execution tools aren’t everywhere yet, but they’re getting close. This is good news for the evolution of IT automation because task execution is the foundation for broader automation. Task execution tools will be implemented and then unified with process flow technologies and, eventually, decision triggers to bring more intelligence to an organization’s overall automation tool chain.

Unfortunately, roadblocks to broader adoption of automation technologies and the whole concept of automation remain.

The top resistance force is cost. Many automation solutions are expensive, but more economical options are now available (Chef, for example)

The concerns about technology shortcomings are also valid, to an extent. Comprehensive automation is still out of reach for many enterprises, but plenty of reliable task execution utilities are already available.

The bottom line with every barrier to automation, though, is the issue of trust. And the whole automation field is now more an evolution of trust than it is an evolution of technology.

Getting technology people to work well together is a major part of the answer, as my next post will demonstrate.

Archives