You are right, and I have some blame here of not including positive criticism in my comments. I do not have a script to offer, but I can go through the process I work by.

#1: A good password is a must. During installation, have a second computer generate a good password and either memorize it, or GPG encrypt it somewhere on the second computer. pwgen is decent in generating passwords.

2#, I fully agree with the article on automatic updates if its a personal computer. For others, one can have root mails sent if you are a fast and and read mail daily. This is how many people read about vulnerabilities before they reach the news.

#3: When installing large package like web services, I keep in mind of the long term prospect of each project. I ask the questions: Is there a deb package? Is it being maintained by a large group of independent developers? Is it mentioned in discussion at Serverfault? Are there any recent updates? What does the Wikipedia page have to say?

#4: read the man page, and check any section labeled security. Some man pages will say things like "we have this port open. Its completely insecure, and we expect either the local network to be safe or that you use a firewall". Through this just happened once for me, its still a good practice to check the man page with new services.

#5, avoid php themes/mods that require you to manually patch things. They won't be updated by Ubuntu, so things will either end with you uninstalling it or forgetting that it exist and thus get hacked. Sometimes ubuntu will just install over the mod, dealing with the issue for you.

Other than that, harderning depend on use case. A wiki/forum will need some form of anti-spam protection. A media center need access control or firewall to only allow local network. Unsecured protocols like nfs and nis need something like ipsec. A mail server needs authenticated smtp.

> have a second computer generate a good password

Why a second computer?

The Ubuntu installer does not ship with pwgen, and its good to have the password done right, right there while installing.

I'm throwing my hat in the ring. I took the author's original post and implemented it as an Ansible playbook in a little less than an hour.

http://practicalops.com/my-first-5-minutes-on-a-server.html

Happy to answer questions about it. :)

I don't know Ansible. I learned about it through this thread. It sounds very interesting. I'll have to experiment with it as I am getting ready to launch a service that will start out with one server but will probably grow quickly to the five to ten range with a few devs involved.

Towards the end of your page you talk about keeping a manual log file. That's exactly the way I've handled this sort of thing for years. I usually call mine "project-log.txt". These files usually have several sections and it looks like this:

Header: Project name, start date and other relevant details

Date entry: What I did on a particular date

Working on now: Before I do anything I make this entry. After an interruption I usually have no clue what I was working on. By making a habit of writing a note to myself about what I am going to be working on immediately I can task-switch to that phone call or unwelcome question and mentally come back to what I was doing quickly.

To Do: Self explanatory.

Questions: If questions come up I log them here so I don't forget.

Ideas: Sometimes as you are working you say things like "hey I should do x". I log them here and go back to what I was doing.

References: Links, etc.

After launch: There are things that might not be critical at all and can be done after launch. I try to only note must-have's in "To Do".

This file lives at a self-named directory: "~/project-log". I do this because I can also save other relevant files there and keep the entire thing under source control.

  <Project Name>
  <Date>
  <Description>
  <Notes>

  --------------------------------------------------------------------- 01MAR13
  <stuff I did on this date> format example:
  - Installed Ubuntu 12.04 LTS
  - Installed PHP
  - Installed MySQL
  - Installed Python
  --------------------------------------------------------------------- 02MAR13
  <stuff I did on this date>
  --------------------------------------------------------------------- 04MAR13
  <stuff I did on this date>

  --------------------------------------------------------------------- WORKING ON NOW
  - Configuring Virtual Hosting

  --------------------------------------------------------------------- TO DO
  - Configure vim for php development
  - Customize bash
  - Install imagemagick
  - Install php5-imagick

  --------------------------------------------------------------------- QUESTIONS
  - How to automate server bring-up and hardening?
  - Firewall or not?
  - Disable SSH?

  --------------------------------------------------------------------- IDEAS
  - Learn Ansible for automated deployment.

  --------------------------------------------------------------------- REFERENCES
  - Ubuntu 12.04 LTS docs
      https://help.ubuntu.com/12.04/  
  - Ansible
      http://ansible.cc/

There's nothing like good old plain text. :)

Nowadays I'm downright spoiled and use org-mode[1] to keep my systems journals. Org files are plain text as well, and org-mode takes care of setting up the tree by date. I can also add a journal entry from anywhere in Emacs with just a couple keystrokes, which makes it incredibly low-friction to use.

Like I said, the most important thing is to TAKE NOTES. Even pen and paper. It's one of Limoncelli's big points in Time Management for System Administrators.

Tooling doesn't really matter, the important part is being able to remember what the heck I did and when I did it. Invaluable for troubleshooting.

[1]: http://orgmode.org

The reason nobody shared their one true checklist for security is that it doesn't exist. Well secured systems get owned every day, and tons of bad ones never have trouble due to luck and lack of incentive.

Most extensive security procedures contain a lot of questionable advice and few will prevent human error which most compromise can be traced back to.

The only semi-universal list amounts to 1) use keys for ssh

2) block/turn off everything except your (web) service

3) automated security updates or some sort of failsafe procedure to install security updates very regularly (this is by far the most common error)

4) Learn about web application security threats especially sql injection and consider them at design and implementation and have some kind of regular review (this can be very difficult to get right)

5) Avoid storing anything that makes you a particularly desirable target like bitcoins or secret defense plans as anything can be hacked with enough incentive.

Most everything else probably won't make too much of a difference to you.

As far as one true checklists go, I really like NSA Guide to securing RHEL.

http://www.nsa.gov/ia/_files/os/redhat/NSA_RHEL_5_GUIDE_v4.2...

and TLDR version: http://www.nsa.gov/ia/_files/factsheets/rhel5-pamphlet-i731....

It is starting to get a little bit dated (RHEL 5 is quite old), but general rules still apply and usually they explain their reasoning.

> 3) automated security updates or some sort of failsafe procedure to install security updates very regularly (this is by far the most common error)

This isn't clear. Are you saying you think automating these updates is good or bad?

> 5) Avoid storing anything that makes you a particularly desirable

Well, in some cases your user's uid and pwd is the most valuable chunk-o-data a would be attacker wants.

> Most extensive security procedures contain a lot of questionable advice

List?

> and few will prevent human error which most compromise can be traced back to.

I think this would be the power of having a canonical auto-provision script on Github that many can review and contribute to. The script could certainly take the form of sections that could be commented out as needed. In other words, a well documented and reviewed set of recommendations that someone could edit based on pier-reviewed information in the comments and then use to automatically configure a server. That, I think, could be of value.

This is great! I was thinking exactly along those lines, and started looking for puppet/chef scripts that are readily available.

It seems that using puppet has a pretty steep learning curve, at least from what I could gather skimming the docs.

Actually what I would really like to see is an interactive script, which then guides me through the process of hardening a fresh ubuntu server install, offering sane suggestions along the way.

I am pretty sure such scripts do exist, but having them maintained and regualarly updated somewhere would be quite neat.

I'm in the process of automating the configuration of my VPS using Ansible. This tutorial is an excellent introduction http://lextoumbourou.com/blog/posts/getting-started-with-ans...

Nice, I'm on the same road. I'm also looking at Ansible, it's just that I haven't got around to it yet.

ansible looks pretty good. thx for the link.

About a year ago I spent several days trying to get a full puppet system. It was a real headache to be honest and many of the modules have no or very vague docs. Coming back to my setup it's very hard to read or trace what my setup is or how to do one off deployments.

Today I have started with ansible and have already gotten a huge amount done and deployed. Very happy with it. It's all in a small git repo and its very readable and concise.

It hasn't been updated too much recently, but I've used Bastille for this in the past, and it's in the Ubuntu repos. http://bastille-linux.sourceforge.net/

Yeah, I forgot about bastille. On their news page they claim that development has been restarted recently. Lets hope that it gains enough momentum again.

Last I checked, Bastille was broken enough that you could fix it easily (I made a few small patches myself), if you knew what you were doing. Which kind of defeats a lot of the power of Bastille, but at least it was still kind of a handy automated checklist for seasoned sysadmins.

I've been using a product called scriptrock (http://scriptrock.com) recently that collects these kinds of best practices and ensures that they actually exist on nodes. I.e, the testing of configuration rather than the execution. I think this is really interesting for two reasons. 1, it's like monitoring for security on my nodes (it runs by default every 5 minutes), and 2, they could build some kind of open repo for people to contribute these best practices (I think they were asking for beta users a while back). Plus it allows me to configure my infrastructure in a kind of TDD way.

Add https://fedorahosted.org/scap-security-guide/, which I will at some point attempt to warp into an Ubuntu checklist, since I have always thought concordance on at least some security posture settings should be distro-neutral, with distro-specific implementations.

Writing a canonical server security document is difficult because it depends so much on what the server is doing (what kind of application is it running? how is it accessed?) and where you're starting from. (barebones install? something pre-rolled by your VPS service?)

Sounds as though we need a decision tree app to write the scripts for us. It asks questions such as, "Do you have your own IP block?" and explains the question thoroughly for those who don't understand it, explains the answers offered (for questions such as, "Which of the following types of logins would you like to allow (check all that apply)? Password, public key, ...").

It would offer you explanations, deeper explanations, defaults, recommendations for when you would override the defaults, etc. for each item. It would be smart enough to prevent incompatible or contradictory settings.

Once this process was complete, it would generate the needed puppet / ansible / whatever script, which the admin would run and store for future reference and use. When installing another machine, the original script could be read in by the decision tree and the new script could be generated by modifying the original rather than starting from scratch.

I would start with Ubuntu 12.04 LTS on Linode and see what develops from there.

Shocking that no one has mentioned the grsecurity project.

The problem I had with the article was that it said what user accounts he set up, how their security was based, etc. Why!? Don't give away that info in a blog post! Even if it is something a hacker could easily find out if they wanted, by doing that, you paint a target on your back.

If you are walking past a dark alley in a big city, you don't say, "I am a blue-belt in karate, and I have a wallet with $300 in it." They have a gun, and they'll take that wallet, thank you.

I totally appreciate you putting all those links together, but I would hope that others remember that loose lips sink ships when it comes to security. And from a hacker's perspective, what's the fun in social engineering if someone just blabs it all in a post?

If your system can't survive an attack because the attacker learned from a blog post that you are using tool X then perhaps tool X is the problem and not the blog post.

Real would-be intruders are not dummies. They have a suite of tests they can run to "x-ray" your system to the extent it is possible and discover vulnerabilities.

To some degree it's like encryption code. The safest code has to be open source.

> The safest code has to be open source.

This is a commonly held belief that is not true. I've been doing open source development quite a bit over the last several years and have seen plenty of insecure open source projects that were even less secure than I would see in the private repository of every place I worked. Here's why:

* Open source code gets more eyes on it, when it is well-used. But there are loads and loads of projects that are hardly looked at, and they have a greater chance to be used before they are thoroughly vetted.

* Those projects were most likely thrown up there by a developer like me who just hacked something up quickly to solve a problem. Once done with the problem, the code stays up there and I just let it atrophy. That consists of 95% of my projects, at least.

You misunderstood. I was referring to encryption code. These are open source projects that have to survive the scrutiny of experts. For example, I guarantee you that if I attempted to contribute to an open source encryption code-base without a massive amount of work my submissions would be rejected. This is not my area of expertise. Could I do it? Sure. Nothing is impossible. However, I would have to devote a significant amount of time to fully understanding the state of the art before doing so.