Pick Our Brain

Open Source Contributions + Knowledge Sharing = Better World

  • Nested Variable Inheritance with Ansible: How to use a variable that refers to one of several lists of values

    Share this post:
    : Nested Variable Inheritance with Ansible: How to use a variable that refers to one of several lists of values

    So what do we mean by “nested variable inheritance” anyhow? We mean that we want to use the value of a variable, to refer to another variable which contains a list of values. We’re “inheriting” the name of the list, and then using the values contained in the list when evaluating an expression.

    In our use case, what we’d like to do is iterate over the list of values in a “with items:” expression, such as a list of usernames, or a list of file names. Instead of explicitly specifying the name of the list we’d like to use, we’ll use a variable who’s value is the name of the list. In this way we can write a playbook which can perform a task, but still iterate over different lists depending on which group of hosts the playbook is running against.

    First let’s define an example where this is useful, and later we’ll show how it can be done. Let’s say we have three departments in our company: Exec, Engineering and Sales. We want to use an ansible playbook “add-users.yml” ans the ansible users module to add the user accounts to systems, but only those accounts which are appropriate to the specific systems.

    A Basic Playbook

    Our standard vars file looks like: (vars can be defined in the playbook, stored under role/vars or added via “include_vars:” )

    user_list: [“user1”, “user2”, “user3”]
    password: [“myp@ssw0rd”]

    And our standard playbook looks like this:


    – name: Add user accounts
      user: name={{ item }} password={{ password }} shell=/bin/bash
      with_items: user_list

    But in this form, we’re stuck with only a single list “user_list” which contains all users which will be created. We need to use different lists for each of our departments. So let’s try again:

    Breakout users into groups

    Here’s our new vars file:

    exec_users: [“exec_user1”, “exec_user2”, “exec_user3”] 
    eng_users: [“eng_user1”, “eng_user2”, “eng_user3”]
    sales_users: [“sales_user1”, “sales_user2”, “sales_user3”]   
    password: [“myp@ssw0rd”]

     And our new playbook:


    – name: Add Exec user accounts
      user: name={{ item }} password={{ password }} shell=/bin/bash
      with_items: exec_users

    – name: Add Engineering user accounts
      user: name={{ item }} password={{ password }} shell=/bin/bash
      with_items: eng_users

    – name: Add Sales user accounts
      user: name={{ item }} password={{ password }} shell=/bin/bash
      with_items: sales_users

    So now we have a playbook, which will create accounts for three different sets of users, but we still need to distinguish which users we want to create on which hosts. We can add a hosts expression to our playbook to specify which hosts to run each task against.

    Limiting scope of user creation by groups


    – name: Add Exec user accounts
      hosts: exec

      user: name={{ item }} password={{ password }} shell=/bin/bash
      with_items: exec_users

    – name: Add Engineering user accounts
      hosts: eng
      user: name={{ item }} password={{ password }} shell=/bin/bash
      with_items: eng_users

    – name: Add Sales user accounts
      hosts: sales

      user: name={{ item }} password={{ password }} shell=/bin/bash
      with_items: sales_users

    The problem there is we are still statically defining which hosts will have which users. Also when the playbook runs, we’ll always have tasks being skipped when the hosts we are running against don’t match  the hosts expression. And there’s the larger issue that as we get more groups, our playbook gets larger and more complex as we specify more lists and more hosts groups.

    So, let’s try this another way. This time, we only want one stanza to add users, but we want to specify the list as a variable whose value is the name of the exec, eng or sales lists.

    Specify which group of users to create dynamically

    Our vars file is the same as before, but now we add entries to group_vars, setting the value of user_list to be one of the three lists for the appropriate group:

    i.e. in group_vars/exec:

     user_list: exec_users

    And now our playbook:


    – name: Add user accounts
      user: name={{ item }} password={{ password }} shell=/bin/bash
      with_items: user_list

    This looks like it will do the trick, unfortunately when we run the playbook, it iterates over the single value of the variable “user_list”, and creates users named “exec_users, eng_users and sales users”, rather than the lists we are trying to point to.

    Ansible will normally identify a variable when you use a non-quoted value. In this use case, it doesn’t. However there is a workaround that solves this issue. We need to define all of our our vars and lists in group_vars and then explicitly tell ansible that we want to use the value of the variable specified in group vars.

    The solution: Tell ansible to do what I meant, not what I said

    Here’s our group_vars/all :

    exec_users: [“exec_user1”, “exec_user2”, “exec_user3”] 
    eng_users: [“eng_user1”, “eng_user2”, “eng_user3”]
    sales_users: [“sales_user1”, “sales_user2”, “sales_user3”]
    password: [“myp@ssw0rd”]

    And in group_vars/exec :

    user_list: exec_users

     Our secret sauce is in our playbook vars file, here we tell ansible where to get the values from:

    user_list_name: “{{ hostvars[inventory_hostname][user_list] }}”

     And finally our playbook:


    – name: Add user accounts
      user: name={{ item }} password={{ password }} shell=/bin/bash

      with_items: user_list_name

    This finally gets us what we want, a simple generic playbook which creates user accounts. We have no extra tasks to skip or groups to re-define, and we only create the users we need on the hosts in the appropriate groups. When we add a new host, we assign it to the appropriate groups in our inventory file, and it gets all of the users which belong to the groups.

  • Setup vim for Development with Go, and JS

    Share this post:
    : Setup vim for Development with Go, and JS

    Description

    Notes from setting up vim with a handful of plugins for Go, Javascript development.

    Context

    I was a long time emacs user that migrated to Sublime Text (ST) Almost 3 years ago. I started with ST2 and changed to ST3 when I stopped using a Mac as my primary OS. I am very happy with ST. It is well worth the $70 (US). However, development is excruciatingly slow and there is zero communication about its future. The ST3 beta came out in January 2013. 18 months later we are still in beta. I suppose I could — and may — go on using the beta indefinitely. But what fun would that be.
    There are quite a few folks whom I respect which are from the other side of the editor wars — vim. I have long had it on my plate to learn more vim. This is an early step in that direction.
    Having never used vim as my editor I am starting from scratch; without
    ~/.vimrc even. That said I am not starting from scratch on my workstation so these notes may miss installing dependencies which already present on my system.

    Solution

    Install a handful of plugins with a plugin manager to make developing Go more efficient. The plugins installed are:

    1. Vundle
    2. go-vim
    3. YouCompleteMe
    4. tagbar
    5. vim-commentary
    6. delimitMate
    7. syntastic
    8. vim-javascript-syntax
    9. Vim Javascript
    10. vim indent guides
    11. tern for vim

    Using these plugins together with an otherwise vanilla vim provides code highlighting, completion, and hinting in Go, JavaScript, and Python. It also provides Syntax linting for Go and JavaScript. I stil need to find a good Python linter.

    Vundle

    Vundle is a plugin manager for vim. It downloads, installs, manages and updates vim plugin packages. NB: Vundle requires vim 7.4.
    First we need to create a place for vim bundles or plugins, checkout Vundle and generate a vimrc.

    $ mkdir -p ~/.vim/bundle # create dir
    $ git clone https://github.com/gmarik/Vundle.vim.git # clone vundle
    

    Then we need a .vimrc file setup as outlined in the Vundle README under bullet 3, as of this writing. A stripped down version might look like this:

    set nocompatible    " be iMproved -required
    filetype off               " required
    
    " set the runtime path to include Vundle and initialize
    set rtp+=~/.vim/bundle/Vundle.vim
    call vundle#begin()
    
    " let Vundle manage Vundle, required
    Plugin 'gmarik/Vundle.vim'
    
    " " Go
    Plugin 'fatih/vim-go'
    
    " " General
    Plugin 'Valloric/YouCompleteMe'
    Plugin 'majutsushi/tagbar'
    Plugin 'tpope/vim-commentary'
    Plugin 'Raimondi/delimitMate'
    Plugin 'scrooloose/syntastic'
    
    " " Javascript
    Plugin 'jelera/vim-javascript-syntax'
    Plugin 'pangloss/vim-javascript'
    Plugin 'nathanaelkane/vim-indent-guides'
    Plugin 'marijnh/tern_for_vim'
    
    " All of your Plugins must be added before the following line
    call vundle#end()            " required
    filetype plugin indent on    " required
    " To ignore plugin indent changes, instead use:
    "filetype plugin on
    " Put your non-Plugin stuff after this line
    

    Once the rc file is setup you need to let Vundle install the rest of the plugins for you. You can do that by calling vim with an argument: vim +BundleInstall. Alternatively you can start it and call PluginInstall:

    start vim
    :PluginInstall
    

    go-vim

    go-vim is a “Full featured” plugin for Go development. It is essentially a meta-plugin that installs go-code, go-imports and sets up a bunch of useful defaults. See the README at the above link for more information.

    YouCompleteMe

    YouCompleteMe is a fuzzy search code completion engine for Vim. It requires a coulple extra steps to compile and integrate completely.

    $ cd ~/.vim/bundle/YouCompleteMe
    $ # ensure the requirements are installed
    $ sudo apt-get install cmake python-dev build-essential # Present on OSX if XCode is installed.
    $ ./install.sh --clang-completer
    

    tagbar

    tagbar is a plugin that displays tags in a window, ordered by class, function, etc…

    vim-commentary

    vim-commentary is a plugin to make un/commenting code (out) easier and more efficient in Vim.

    delimitMate

    delimitMate is a plugin that provides something like electric-pair-mode in emacs.

    syntastic

    syntastic is a plugin that provides a plugpoin to run external syntax checkers on code within Vim.

    vim-javascript-syntax

    vim-javascript-syntax adds enhanced JS syntax to Vim

    vim-javascript

    Vim Javascript fixes the other JS syntax bits or something…

    Vim Indent Guides

    vim indent guides is a plugin that visually displays indent guides…

    Tern for Vim

    Tern for Vim is a plugin that provides Tern-based JavaScript editing support.
    Make sure you have node.js and npm installed (Tern is a JavaScript program), and install the tern server by running

    npm install 
    

    in the bundle/tern_for_vim directory.

    To Explore

    There are other plugin managers for vim which is left to the reader to compare if they desire. Similar setups are available in emacs and SublimeText — this is also left as an exercise for the reader.
    There are still a few items left that I would like to explore.

    1. Python – There are still a few things about Python that I need to get setup in vim. PEP8 linting is one major one. Possibilities to explore?
    2. angular – We are doing a lot of angular work, so it would be nice if I could get a little more editor love for it. Not a show stopper…
    3. tab settings – I am unclear on some of the tab settings. I think I have it set up in a way that makes me happy, but I am not positive.
    4. Lastly, I need to get my head around modal editing. I still have trouble remembering the syntax for doing most anything other than typing. That will come with time and use though.

    Wish me luck! Constructive feedback and suggestions welcome.

  • Slow DNS on Ubuntu?

    Share this post:
    : Slow DNS on Ubuntu?

    Experiencing 3 second lookup times from DNS? Me, too.
    Here’s how I fixed it.
    Edit /etc/nsswitch.conf and move dns before mdns4_minimal on the hosts: line. Like so:

    hosts:          files dns mdns4_minimal [NOTFOUND=return]
    

    And restart networking:

    sudo service network-manager restart
    

    That seems to have fixed it for me.

  • Adding Subfolders to My Drive in Google Drive

    Share this post:
    : Adding Subfolders to My Drive in Google Drive
    Adding Subfolders to “My Drive” in Google Drive / Syncing individual folders to Drive Desktop
    • Problem: Drive doesn’t (easily) allow you to ONLY add a subfolder to “My Drive”. This becomes problematic when you wish to sync only certain files using the Google Drive desktop application. Read below for the two workarounds (I personally prefer #2!).

    NOTE: This only works in the “Old” Drive. If you’ve turned on the “New” Google Drive, you may temporarily revert to the old one by clicking on the gear box in the upper right hand corner of Drive.
    • Fixes:
      1. If only one person would like to add the subfolder to “My Drive”, it is possible to change the owner of that subfolder to the individual wishing to have it sync to their “My Drive”  (Workaround #1)
        • Process:
          • Find the owner of the Subfolder
            • In Drive, find the folder where you would like to change owners
            • Right click the folder icon, and click Share… > Share
            • Scroll through the list of users that the folder is shared with and identify who is listed as “Is owner”
            • Contact that person to let them know you will need them to change ownership over to you
            • Press escape or click “Done”
          • Owner of subfolder:
            • Right click the subfolder and click Share… > Share
            • Scroll through the list of users, and find the user you would like to make owner
            • Press the dropdown next to their name
            • Change that user’s setting to “Is Owner”
            • The subfolder should now magically appear in the new owner’s “My Drive”

    1. If multiple people need to add the subfolder to “My Drive” (Workaround #2) *Reminder: See note at top of page regarding Old/New drive dependencies.
      • Process:
        • Create a new empty folder in “My Drive” where you’d like to have all the subfolders and documents you include synchronized to “My Drive”
          • Name it whatever you like, ie. “Synched Folders”
        • Locate the subfolder you wish to have synchronized/added to your newly created folder in “My Drive”
        • Right click the subfolder you wish to synchronize and select “Move to…”





    • Cmd+click (Mac) or Ctrl+click (Win) your newly created folder (named “Synched Folders” in this example)
      • When you perform this action, there will be a check mark that shows next to your newly created folder AS WELL AS the original parent folder within which the subfolder resides





    • Finish by clicking “Move”
    • Now everything you modify in your “My Drive/Synched Folders” directory will also be modified in the shared subfolder
      • Be extra careful so you don’t accidentally delete something!



  • Terminal Based Multi-Factor Authentication Token

    Share this post:
    : Terminal Based Multi-Factor Authentication Token

    Overview

    More and more sites are now offering Multifactor Authentication (MFA) authenticate with soft-token applications such as Google Authenticator or AWS Virtual MFA. Some websites that now allow MFA authentication are: Google Apps, AWS, Dropbox, etc…

    Here we will go over settting up a token generating app in a terminal. This saves me digging my phone out every time I log into something that has MFA enabled. With only a couple MFA enabled sites it isn’t too bad, but pass ten and you start looking for your phone pretty regularly.

    Dependencies and installation 

    To do this you will need to MFA enable an account and to install http://www.nongnu.org/oath-toolkit/. It is likely you’ll need a qr-code reader on your phone such as QR Barcode Scanner or use a command line tool/utility like libdecodeqr-simpletest to decode the key from the QR Code.

    Ubuntu

    $ sudo apt-get install oathtool
    ## Optional qrcode reader util
    $ sudo apt-get install libdecodeqr-examples

    OSX

    Installation may be supported by brew, fink, macports, or other things, but following are instructions for installing from source on OS X Sarah Pal^H^H^H^H^HMavericks. (It is left to the reader to ensure the signature verifies and to substitute up to date version numbers where applicable):

    $ curl 
    http://download.savannah.gnu.org/releases/oath-toolkit/oath-toolkit-2.4.1.tar.gz  
    | tar zxf -
    $ cd oath-toolkit-2.4.1
    ## disable here or figure out how to install XMLSec
    $ ./configure --disable-pskc
    $ make -j5
    $ make check ## I got failures but it worked to generate tokens
    $ sudo make install
    $ oathtool -h

    Enable MFA Somewhere

    Now we’ll walk through enabling MFA in an AWS IAM Account.

    Enabling MFA in AWS

    Login to your console

    • Go to the IAM service (/iam/home once logged in)
    • Then click users
    • Select your user
    • Select the “Security Credentials” tab
    • Click “Manage MFA Device”
    • Select “A virtual MFA device”
    • Click continue
    • Read the warning and click continue

    At this point a QRCode Appears with two fields in which to enter 2 consecutive qr codes. We want to add it to our phone’s MFA Virtual Token app and add it to our aliases so we can generate codes in our terminal…

    Take a screenshot of the image, you can toss this when you’ve extracted the key. You can also use this to add more tables and other devices later.

    • Use your phone app to add it to your Virtual MFA app by scanning the QR Code repeat if necessary
    • Use the QR Barcode Scanner to extract the key and save it as a text file.
    • Alternatively you can use the libdecodeqr-simpletest utility installed with libdecodeqr-examples which should look something like this:
    $ libdecodeqr-simpletest ~/AeroFS/totp-qr-code/cjp-aws-k-totp.png
    libdecodeqr version 0.9.3 ($Rev: 42 $)
    STATUS=2000
    otpauth://totp/me@myaws?secret=LALSKDJAOSFDADS3K5SADFHAKDSJNAKSDJN2OISD7USODIF33SOCVISOIVDVOID4
    • Note the secret in the output above, you will need it shortly
    • you can now enter 2 consecutive codes from your Virtual MFA Token app on your phone.
    • Click “Continue”
    • Click “Finish”


    Testing token generation on the command line

    We can run oathtool with the secret key we extracted from the QR Code above.

    $ oathtool --totp --base32 LALSKDJAOSFDADS3K5SADFHAKDSJNAKSDJN2OISD7USODIF33SOCVISOIVDVOID4

    NB: Sometimes the secret key is provided with spaces like so `sadq 3ine dsfs 4scw kmw2 ohac q4m4 h2vw`. In htis case we”ll need to quote it. Like this

    $ oathtool --totp --base32 "sadq 3ine dsfs 4scw kmw2 ohac q4m4 h2vw"

    You may wish to compare the tokens generated here with those on your phone. Or Perhaps you’d like to log out and log back in to test it. You’re choice. 

    Adding an alias for the command in ~/.bashrc

    NB: setting appropriate permissions and protecting the file with the aliases in it is an exercise left to the reader!!
     

    In $HOME/.bashrc add an entry like this

    alias myawsiotp='oathtool --totp --base32 LALSKDJAOSFDADS3K5SADFHAKDSJNAKSDJN2OISD7USODIF33SOCVISOIVDVOID4'

    In the case that there are spaces then something like this:

    alias myawsotp='oathtool --totp --base32 "sadq 3ine dsfs 4scw kmw2 ohac q4m4 h2vw"''

    Source your .bashrc and you should be ready to go:

    $ source ~/.bashrc
    $ myawsotp
    109345

    Additional Notes

    Extracting from Google Authenticator or AWS Virtual MFA 

    I tried pulling my previously keys out of Google Authenticator and AWS Virtual MFA, but couldn’t get them out of Google Authenticator at all. I did manage to ge them out of AWS Virtual MFA. See this page for 3 methods to try to get GA out.

    I did manage to get keys from AWS Virtual MFA by creating a backup then extracting the DB out from there like so:

    Install some utilities:

    $ sudo add-apt-repository ppa:nilarimogard/webupd8
    $ sudo apt-get update
    $ sudo apt-get install android-tools-adb android-tools-fastboot

    Create a backup:

    ## creates a backups called backup.ab
    $ adb backup -noapk -noshared -all -nosystem

    Then build the android-backup-extractor somewhere:

    $ git clone git@github.com:nelenkov/android-backup-extractor.git
    $ cd android-backup-extractor/lib
    $ curl http://downloads.bouncycastle.org/java/bcprov-jdk15on-148.jar -O
    $ cd ..
    $ sudo apt-get install ant
    $ ant
    $ sudo apt-get purge ant
    $ sudo apt-get autoremove

    Now we can use it to unpack the backup:

    $ java -jar ~/java/android-backup-extractor/abe.jar unpack backup.ab backup.tar  <password/code>
    $ sqlite3 dbs/com.google.android.apps.docs/f/fileinternal/<your_hash_here>/DB
    sqlite> .headers
    sqlite> select * from accounts;

    And it will squirt out your entries which you can then use like we did above.

  • AWS hijinx: Windows instances launched from AMI fail to get new password, password recovery only makes things worse

    Share this post:
    : AWS hijinx: Windows instances launched from AMI fail to get new password, password recovery only makes things worse

    More Windows, more password problems

    When a Windows Server 2008R2 instance is launched from a custom AMI, the Amazon EC2Config service is supposed to generate a new random password for the administrator account. Instead when trying to use the “Get Windows Password” option. the user is presented with the dreaded “Your password is not Ready” error shown below:

    What does this error mean? Well in a nutshell the EC2Config service did not generate a new password for this instance, and as a result the AWS console doesn’t have the new password to show you. This message always shows up immediately after a new windows instance is launched, but normally goes away after 15 minutes when the EC2Config service generates the new password, but if that new password is never generated you will still get this same error months after launching the instance.

    Ok, that looks easy, lets fix it

    A quick google search will tell you that the EC2Config service needs to be given permission to create the new password, and all will be right with the universe. This is done by editing the file “C:Program FilesAmazonEc2ConfigServiceSettingsconfig.xml”, to change the value of “Ec2SetPassword” to “Enabled”. Since you can’t login to the instance to allow this, you try the next search result which tells you to detach the root volume of the problem instance, and reattach to a windows instance which you can login to. Then you can mount the volume, edit the config.xml file, detach the volume, reattatach to the problem instance and then boot. Simple right?

    Not so fast! You now find yourself with an instance that starts, but doesn’t boot anymore. From the AWS console you will only know that the instance fails the instance status check, while passing the system status check. If you could see the console of this instance, you’d see something like this:

    Windows is telling you it can’t find a bootable partition, and is encouraging you to load the setup disc in the drive and perform a repair. I guess if I can find my AWS DVD drive, I can just… awww crud!

    So what do we do here? My boot drive which was working before is now not booting, and even when it was I couldn’t login because I didn’t know the password. I’m sure the Cloud OPS team at Microsoft planned to fix this bug, but the fix it never made it into the last beta.

    Taking the long way home

    All is not lost, your volume still contains all of your data, and your instance still can boot from it, as long as it were marked as bootable. For some reason the alternate Windows instance to which this volume was attached unmarked the bootable flag, and then treated it as a secondary volume. This renders is useless as a boot volume without some additional magic.

    What you need to do is reattach the volume to your alternate instance, and run a few handy utilities to reset the bootable flag on your volume. First utility is “bootsect.exe”, this is found on the windows 2008R2 setup disc in the boot folder. The other two “bcdboot.exe”and “bcdedit” are part of the standard windows installation, so you don’t need to go looking for them.

    Assuming that you are:

    1. running these commands on an alternate Windows 2008R2 instance
    2. trying to fix a boot volume for Windows 2008R2,  mounted on the D drive
    3. rot running some exotic setup where windows was not installed on the boot drive of your problem instance

    You need to run the following commands on your alternate instance to fix the boot volume for your problem instance:


    bootsect /nt60 D: /mbr
    C:windowssystem32bcdboot.exe D:Windows /s D:
    bcdedit /store D:BootBCD /set {default} device partition=D:

    bcdedit /store D:BootBCD /set {default} osdevice partition=D:
    bcdedit /store D:BootBCD /set {bootmgr} device partition=D:

    Each command should return a message indicating success. Once this is done, you can reattach the volume to your problem instance, and it should boot normally and reset the administrator password. You should now be able to get the new password from the AWS console and login to your instance.

    Was all that really necessary?

    No, it wasn’t. Why all this information explaining how to do something you don’t need to do? Well, the problem is that by the time you realized you couldn’t login to the new instance, you had already made a mistake. The way to avoid this problem entirely is to tell the EC2Config service it can reset your password before you create the AMI in the first place. This way when you launch an instance from that AMI the new instance can generate the administrator password and provide it to you in the AWS console. Simple…

    Here’s how you tell the EC2Config service to do that.  Connect to the instance before you create an AMI, go to Start -> All Programs -> EC2ConfigService Settings, then go to the “Image” tab and select “random” under “Administrator Password” settings as shown below. To save this setting, click apply.

    After saving these settings, you can create an AMI from this instance, and on relaunch the administrator password gets reset successfully. No hijinx necessary.

  • WordPress White Page of Death vanquished: database and template mangling when changing hosts

    Share this post:
    : WordPress White Page of Death vanquished: database and template mangling when changing hosts

    What are we working on?

    Our team is working on a contract to migrate a number of public and internal NASA web sites and application to the public cloud, Amazon Web Services in this case. Some of these we’re migrating almost verbatim, some we’re migrating from one application platform to another, and a few we’re reimplementing to leverage AWS-specific cloud services.

    One of the first applications we migrated was used as a blog and wiki platform; it was written in a proprietary system to which we had no access.  Our direction from NASA was to focus our deployment to a few dominant open source platforms. For this one, we chose WordPress.

    Setting up the WordPress platform wasn’t hard but migrating content and users (and integration with NASA’s internal single-sign-on) proved much more challenging.

    To run security scans and test new deployments, we want to be able to quickly clone production to development servers. AWS snapshots make this really easy: snapshot the boot volume, create a new AMI from it, and launch the AMI. Unfortunately, the change in hostname on the new box breaks WordPress — in our case in a subtle way that wasn’t addressed in the docs we found.

    This post explains why this happened, provides a couple solutions to fix it, and a lesson learned to avoid this problem in the future.

    WordPress Database Mangling

    When moving WordPress from one host to another, the change in hostname breaks WordPress.

    WordPress embeds the hostname of the original host in its database files, many of them. Some use the hostname, some use URLs, and other embed it in PHP-serialized strings which precludes global search and replace on an export file.

    We’re not addressing changing the WordPress folder location here since other docs discuss that in more detail.

    Many Locations of the Hostname

    Database table wp_blogs maps the multisite id to the name and path; the ‘domain’ is what you need to change.

    The wp_options table, and for multisite networks, wp_*_options contains name/value pairs that use the hostname. These include critically: siteurl, home, and fileupload_url — and these are all simple URLs. But there are also options like recently_edited, dashboard_widget_options, and adapt_theme_settings which contain PHP-serialized structures; you can’t edit these conventionally because string length changes will break the serialization. There are also _transient_feed_* options which I expected we could ignore.

    There are also wp_posts and wp_*_posts tables which contain the hostname in the guid entry, which WordPress claims should never change, except when moving attachment folders. If we’re moving a site to a new host, it’s probably OK to change but it may cause RSS subscribers to see feed entries from the old site as new entries. But there are other fields that get the hostname that we may want to change, and some of those have serialized representations.

    There are many, many more locations in the tables. We have 153 sites in our network, so we need some automation.

    Automated Search and Destroy

    The Codex describes some of what you need to do, pointing out that multisite is more complicated: http://codex.wordpress.org/Moving_WordPress

    It recommends using interconnectit’s Search and Replace script. The script goes through all the tables in the site and replaces the old hostname with the new, correctly handling serialized structures. We got the dreaded WordPress White Page of Death on our sites though the /wp-admin/ pages worked fine. We tried both version 2 and 3.0-Beta of the with the same result.

    We also tried the wp-cli tool which does the same thing (and more) and had the same White Page of Death syndrome.

    And we tried some custom Python + SQLAlchemy code, again with the same WPOD.

    The fact that all these mechanisms produce the exact same failure is — in some geeky way — vaguely comforting.

    White Page Of Death

    Other reports of White Page Of Death on WordPress complain of the entire site being blank, due to mangled updates or upgrades or extraneous blank lines after a close-php “?>” tag. But our /wp-admin/ pages were fine, for all sites. Something appeared wrong with the themes that the sites are rendered with since they fail, while the baked-in style of the wp-admin pages was OK.

    In our wp_*_options file, we see that ‘theme’ and ‘stylesheet’ were originally set to the same name as our site — wiki.nasa.gov — a pretty obvious choice. When the search-and-replace operations ran amok on the entire DB, it changed those to be the new hostname, and this caused the theme not to be found since it’s located in a directory with the old name.

    Progress: Break Theme, Repair

    To test our understanding, we changed our top multisite’s wp_options where the option_name is ‘template‘ or ‘stylesheet‘ and changed the option_value to “wiki_nasa_gov“. I got a White Page Of Death. Good, this is what I was expecting: that’s not the name of my theme directory!

    I then copied /var/www/wp-content/themes/wiki.nasa.gov to wiki_nasa_gov and got my site back. Excellent: now we can use the search-and-replace sledgehammers (searchreplacedb or wp-cli), and accommodate by renaming the theme directory (and presumably updating our code repo).

    Start-to-Finish: Change DB, Fix Theme

    Below is the step by step. Should take about five minutes.

    First Rule of the WordPress-Club: Backup the DB

    You should be able to restore this if the DB edits hose you.

    mysqldump -u root -p wordpress > wordpress.mysqldump


    Mine takes 25 seconds and results in a 106MB dump file.

    Change Site Config Hostname

    Edit: /var/www/wp-config.php to set DOMAIN_CURRENT_SITE to the new DNS hostname of our development box:

    define(‘DOMAIN_CURRENT_SITE’, ‘wiki.nasa.gov’);

    to

    define(‘DOMAIN_CURRENT_SITE’, ‘cshenton-wiki-webapp-1-stage-pub.example.com’);

    Global Search and Replace

    Download and install the wp-cli to your WordPress dir, rename it to ‘wp’ and make it executable. Use it to global search and replace our hostname in all tables, every column, preserving PHP-serialization; it takes about three minutes for our 153 site network:

    $ ./wp search-replace wiki.nasa.gov cshenton-wiki-webapp-1-stage-pub.example.com –network
    Success: Made 33478 replacements.



    Our …/wp-admin/ pages continue to work, but our site pages now give us the White Page Of Death. This is expected since it changed wp_options to ‘template’ and ‘stylesheet’ to be our new hostname.

    Repair the Theme: the hard way

    We can fix this by editing all the wp_*_options files to point ‘template’ and ‘stylesheet’ back at our old theme directory name ‘wiki.nasa.gov’. Neither the wp-cli nor the searchreplacedb2.php scripts allow us to target this carefully, only certain rows across every table. We’d have to write a bit of code to go through all the wp_options and wp_*_options files and make the change. The following would do the job: 
    <?php
    $db = new PDO(‘mysql:host=localhost;dbname=wordpress’, 
                  ‘dbuser’, ‘dbpasswd’);
    $tables = array();
    $query = “SELECT table_name FROM information_schema.tables 
              WHERE table_name=’wp_options’ 
              OR table_name LIKE ‘wp_%_options’”;
    foreach ($db->query($query) as $row) {
        $tables[] = $row[‘table_name’];
    }
    foreach ($tables as $table) {
        $update = “UPDATE $table SET option_value=’wiki.nasa.gov’ 
                   WHERE option_name IN (‘template’, ‘stylesheet’)”; 
        $affected = $db->exec($update); 
        echo “$affected : $tablen”;
    }


    It’s not really that hard and it’s something you can easily run from the command line. Of course you’ll have to do it again if you search-n-replace this database again.

    Repair the Theme: the easy way

    We can affect the change simply by renaming (or, safer, copying) the theme to the name the over-aggressive search-n-replace used:


    cd /var/www/wp-content/themes
    cp -r wiki.nasa.gov cshenton-wiki-webapp-1-stage-pub.example.com

    Lesson Learned about Theme Names

    While we anticipate it’s pretty common to name your theme after the FQDN of the site you’re hosting, you can avoid all this work by giving your theme a different name. In our case, we might have chosen “www_nasa_gov” for our theme’s directory name.  This will prevent any aggressive search-n-replace on the DB from breaking your theme.

Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors