puzzle_blog

4/2/2014 Management Console Impact

During the deploy of Chef Manage, there was a brief period where redirection was not behaving properly and visitors were presented with an invalid SSL certificate.

This happened because some configuration changes were not adequately tested or applied in all the correct environments as we had planned.

The SSL errors originated from a misunderstanding of which wildcard SSL certificates are used on which load balancers. We had initially set up Chef Manage to use manage.getchef.com, but the load balancer behind which it is hosted only uses *.opscode.com.

In our transition from being Opscode to being Chef, we’ve been conservative about changes we make in the interest of our users and the important jobs they have to do. New certificates that can be used with *.getchef.com have not yet been rolled out to these particular load balancers because we did not want to cause any unnecessary problems for anybody during this transition.

When leading the effort to get Chef Manage deployed, I did not do sufficient research and testing to become aware of this problem until it was too late, and it caused problems for our users and our team. I am sorry that I caused this to happen.

We were alerted to these problems during the deploy and took steps to notify users via our status page and fixed the problems.

At no point was the Hosted Enterprise Chef server API unavailable, and during the outage Chef Manage was available, but not behaving as expected. While this was not a total outage, we hate disrupting our customers’ work and expectations almost as much as we abhor not letting them get their work done because of an outage.

Chef Manage is now available at https://manage.opscode.com.

If you would like to contact me directly about this incident or anything else, you can reach me at [email protected] on on Twitter as @nlsmith.

Nathan L Smith
Engineering Lead – User Interface

Nathan Smith

Nathan Smith is the Engineering Lead for the Chef Visual Interaction team.