Wednesday, November 6, 2013

Incorrect assumptions is the mother of all bugs

After digging deep through the bowels of fuel 3.2 I finally figured out what was wrong - they made the incorrect assumption that I would not use a non RFC 1918 IP address for my fuel bootstrap network.

This lead to ssh not picking up the /root/.ssh/bootstrap ssh key when trying to connect to the node and reboot it. After adding my network (x.y.z) to /root/.ssh/config I could get the nodes to reboot (and install) correctly.

Host slave-* controller-* compute-* storage-* 10.* 192.168.* 172.30.* 172.31.* 172.2?.* 172.1?.* x.y.z.*
CheckHostIP no
IdentityFile ~/.ssh/bootstrap.rsa
IdentityFile ~/.ssh/id_rsa
StrictHostKeyChecking no
UserKnownHostsFile /dev/null

Wednesday, October 30, 2013

Troubleshooting Fuel 3.2

I'm evaluating OpenStack using Mirantis Fuel 3.2, and got stuck on a problem where the bootstrap nodes aren't rebooting when I deploy my changes. To debug that I wanted to see what happened on the console, but since I'm viewing it through serial console over ipmi I got a big fat nothing, as the bootstrap node switches to graphical mode.

Luckily fuel uses cobbler, so it is easy to change that! Log on to the fuel node and run:

cobbler profile report --name bootstrap

Look for the line with the kernel options:

Kernel Options                 : {'url': 'http://x.x.x.x:8000/api', 'biosdevname': '0'}

Then run this to force it to stick to using serial output:

cobbler profile edit --name bootstrap --kopts "console=tty0 console=ttyS1,115200 url=http://x.x.x.x:8000/api biosdevname=0"

Kernel Options                 : {'url': 'http://x.x.x.x:8000/api', 'biosdevname': '0', 'console': ['tty0', 'ttyS1,115200']}

Now you can use serial over ipmi to view the bootstrap boot process!

You'll also have to update the provisioned OS, in this case CentOS:

cobbler profile report --name centos-x86_64

Look for the line:

Kernel Options                 : {'biosdevname': '0'}

And then update the profile:

cobbler profile edit --name centos-x86_64 --kopts "console=tty0 console=ttyS1,115200 biosdevname=0"

Then it should look like this:

Kernel Options                 : {'biosdevname': '0', 'console': ['tty0', 'ttyS1,115200']}

This will let you see the OS installation too.

Friday, October 25, 2013

Do you want to work on cool stuff?

When I joined Turn I didn't realize how cool the job really is - I knew about what the company did on a high level, but it took a while to sink in. Working with an advertising platform doesn't sound very appealing, but the technical challenges are indeed - Google deals with about 36,000 search queries per second, Turn gets over 1,000,000 queries per second! We compute a response in about 10 milliseconds - if you blink your eyes that takes 300 milliseconds. Pulling that off takes a lot of engineering skills!

We are working on really hard and challenging problems - Turn sees about one fourth of the Internet web traffic. So, unless you work at Facebook, Twitter or Google, you don't know what scale means! Our user database is over 1.5 billion, we process 100 billion data events every day, we have 20 trillion attributes stored for querying. If it were 2005, our current compute cloud would be the #1 supercomputing cluster in the world! Our Hadoop cluster is over 21 PB, and we continue adding more and more storage in order to keep up.

I work in the techops team, i.e. guys who keep the s*IT running. It is not your normal corporate IT operations job - we are constantly trying to automate ourselves out of the equation. Everything we do should be self-serve to our internal customers, so that we can keep the number of servers per admin at more than 1,000. We were among the first companies to look into SSDs (and found out they were too slow for us), and my group is working closely with hardware vendors to try out the latest technology.

Turn is currently running yesterday's automation: bare metal servers, puppet and a home grown application deployment tool, but we are moving to infrastructure as a service, with our own private OpenStack clouds (Turn has four data centers around the world), and due to our 10 millisecond response time constraint we are also looking at mesos where we can't afford the performance hit of virtualization. We need to change the way the application is deployed, and investigate SmartStack as a possible way to do dynamic service discovery. Turn has to stay on the bleeding edge of technology, and is looking for people who see throwing out the old ways as an interesting challenge, not as something scary.

If you think you have what it takes to work here, send me an email!

Wednesday, September 18, 2013

Installing gems in your home directory

When using a Ruby install where you can't write to the install directory, installing a gem will fail with:

$ gem install ipmi
Fetching: ipmi-0.0.1.gem (100%)
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions into the /usr/local/ruby/gems/ruby-1.9.3-p327 directory.

This can easily be worked around by telling gem to install into another directory:

Create the file $HOME/.gemrc with:

gemhome: /home/martin/.gems
gempath:
  - /home/martin/.gems
  - /usr/local/ruby/gems/ruby-1.9.2-p327

Then you just need to set your PATH to point to the gem bin directory so you can execute any programs it installs:

export PATH=$HOME/.gems/bin:$PATH

Monday, July 1, 2013

Rubocop rake task

Update: I have submitted a pull request to provide rubocop with a bundled rake task, which was merged into version 0.11.0, so now you just have to add the following to your Rakefile

require 'rubocop/rake_task'

Rubocop::RakeTask.new

If you use rubocop in your project to enforce a common Ruby style across your files, it is easy to make a rake task of it, without having to shell out.

Just add the following to your Rakefile

desc 'run rubocop'
task(:rubocop) do
  require 'rubocop'
  cli = Rubocop::CLI.new
  cli.run(%w(lib spec))
end

Tuesday, January 1, 2013

I love Ruby blocks

While implementing a change in the AWS CPI, I once again got to use a block in Ruby - a feature I really like.

This is a simplified version of the first shot at the change looked:

def user_data
    ...

    if has_dns?(spec)
      data["dns"] = get_dns_servers(spec)
    end

    ...
  end

  def has_dns?(spec)
    spec.has_key?("dns")
  end

  def get_dns_servers(spec)
    spec["dns"]["nameservers"]
  end

This works, but it doesn't follow the tell don't ask principle, i.e. it first asks about the state and then takes action on it if it is in the expected state, instead of just asking for things to be done.

By refactoring it to use a method that takes a block instead, this is now shorter and easier to read.

def user_data
    ...

    with_dns(spec) do |servers|
      data["dns"] = servers
    end

    ...
  end

  def with_dns(spec)
    if spec.has_key?("dns")
      yield spec["dns"]["nameservers"]
    end
  end

The only downside is that it isn't clear what value(s) are yielded when you just see with_dns(spec). If you name the variable(s) properly, you can remedy that for someone else who reads the code, but you still need know what is yielded. I do that by documenting the method using yard and the @yield tag.

Tuesday, December 11, 2012

DTrace in Ruby 2.0

Being a long-time DTrace user while working on Solaris, I really miss using it on our production systems. Luckily I use OS X, so I've got DTrace while developing, but there hasn't been "real" support for it until now!

Ruby 2.0 just got built in DTrace support as of commit 4c740bae, and to build your own Ruby (dev) with DTrace support in rbenv, follow these steps:

Start by installing OpenSSL, as Ruby 2.0 isn't compatible with the OpenSSL in OS X. The easiest way is to use homebrew:

brew install openssl

Next grab the Ruby 2.0 source from github (git is available in homebrew too in case you don't already have it installed).

git clone git@github.com:ruby/ruby.git

Now configure, make & install it:

$ ./configure --prefix=~/.rbenv/versions/2.0.0 \
  --with-opt-dir=`brew --prefix openssl`
$ make
$ make install

Now you can try it out in the current shell:
$ rbenv shell 2.0.0
$ ruby --version
ruby 2.0.0dev (2012-12-11 trunk 38301) [x86_64-darwin12.2.0]

Running DTrace


I'll cover more advanced DTrace scripts in a follow-up post, but here is something to get you started...

Note that DTrace require elevated privileges, and I don't know of any other way in OS X to do that, than to run it through sudo.

Counting number of created objects

This script will count the number of objects created in this very naïve Ruby program

count.d
ruby*:::object-create
{
 @objects[copyinstr(arg0)] = count();
}

hello.rb
puts "hello world from Ruby #{RUBY_VERSION}!"

You have to pass sudo the full path of the Ruby 2.0 binary (/Users/martin/.rbenv/versions/2.0.0/bin/ruby) as dtrace and rbenv doesn't play nice together.

$ rbenv which ruby
/Users/martin/.rbenv/versions/2.0.0/bin/ruby
$ sudo dtrace -s count.d \
  -c '/Users/martin/.rbenv/versions/2.0.0/bin/ruby hello.rb'
dtrace: script 'y.d' matched 2 probes
hello world from Ruby 2.0.0!
dtrace: pid 43289 has exited

  #<Class:0x007ff1728e8710>                                    1
  ARGF.class                                                   1
  IOError                                                      1
  Mutex                                                        1
  NoMemoryError                                                1
  SystemStackError                                             1
  ThreadGroup                                                  1
  Time                                                         1
  LoadError                                                    2
  Object                                                       2
  Gem::Specification                                           8
  Gem::Version                                                11
  Hash                                                        11
  Gem::Requirement                                            24
  Array                                                       96
  String                                                     250