Here at V! Studios, we try to use our resources as efficiently as possible, as long as it doesn’t interfere with getting the job done. One of those resources is obviously computing horsepower, especially when it comes to rendering 3D and 2D images and animations. While we do have a wonderfully auto-scalable cloud-based render farm, using it costs money. Before we reach out to the cloud for more computing power, we try to take full advantage of the existing hardware on-site. We’ve got about 300 GHz of CPU processing power and 276 gigs of RAM in our animator’s workstations that spends most of it’s time twiddling its thumbs. Basic UI only takes a few percent and much of the 2D and 3D interface is handled by the GPU. It’s really only while rendering that the hardware is fully engaged.
Our render farm has the ability to include all of our animator’s workstations, which each have considerable render power. Until recently, we only included those machines for after-hours render jobs, so as to avoid interfering with daily work. It can be hard to get anything done when your computer is cranking nearly 100% on a render job in the background. There’s just not a whole lot of foreground CPU cycles left to work with.
Enter a wonderful little piece of the Unix/Linux/Mac OS world called “niceness.” It has been around for more than 35 years and is simple and robust. Niceness is similar to the CPU Priority settings that can be manually set on Windows, but it has nearly six times the breadth and granularity of Windows Priority levels, and it is much easier to automate if you are at all comfortable working on the command line.
Niceness values run from -20 to 20, giving us 41 different levels of priority. The higher the niceness value, the more polite the process, where it will give up the CPU to any process with a lower niceness value. Most processes have a niceness value of 0 by default, letting the CPU manage resources as it sees fit. When your work and the background render have the same priority, the render is going to constantly fight for access to CPU cycles and RAM.
 |
| So Polite! |
There is a free GUI utility for the renice command for monitoring and manually setting the niceness value on OS X called Process ReNicer.
http://www.eosgarden.com/en/freeware/process-renicer/overview/
http://www.eosgarden.com/en/freeware/process-renicer/overview/
It hasn’t been updated since 2013, but still seems to work fine.
We use the cron utility that is built into nearly every *nix based OS. (There’s a link to a tutorial below.) We use it to set the nice values for specific render processes. Cron is a simple time-based scheduler that will execute commands or scripts automatically at specific times, dates or intervals. The cron utility looks at a text file called crontab (short for cron table). A few simple entries in that text file, and off you go.
In our case, we have two different render processes that gobble up CPU cycles like mad. One is MayaBatch, which manages Maya renders, and the other is Commandline, which manages Cinema 4D renders. (Even within Maya and C4D, there are multiple ways to setup renders so that the process names may be different.)
 |
| CPUs Blazing, No Problem Working |
The nice command launches a process with a particular niceness value, while renice changes the niceness value of an already running process. You can get more information about the uses and syntax of both commands by typing “man nice” or “man renice” in the terminal window. “man” is just short for “manual” it’s handy for all kinds of things in the *nix world. If you want to learn more about any command or utility, you can usually get a lot of info with man.
One problem with using the nice and renice commands is that they expect you to know the specific ID number of the process you want to change. Every time a process is launched, it receives a new ID number. To get around this, we use a command called pgrep. It is a variation of a common search tool, grep, which searches files for specified strings of characters. In our case, we just want to search the currently running processes, which is exactly what the “p” in pgrep does. If you know the name of the process you are looking for you can find that processes ID number simply by typing
pgrep [ProcessName]
in the terminal, and it will return the ID number of the named process.
For example, I might enter
pgrep MayaBatch
and get the returned value
216
To change the nice value of a process we can enter
renice [new niceness value] -p [ProcessIDNumber]
For example, I might enter
renice 19 -p 216
This would set the niceness of the currently running process with ID 216 to a value of 19. The -p tells renice that the next item is the Process ID number.
Since we don’t necessarily know the Process ID number, and we don’t want to have to manually use the pgrep command every time we want to change a processes niceness, we can incorporate pgrep right into the renice command itself like this
renice [new niceness value] -p $(pgrep [ProcessName])
This tells renice to use the results of the pgrep command, which is a $ or character string, as the Process ID number.
So I might use the following command
renice 19 -p $(pgrep MayaBatch)
and any currently running process named MayaBatch will have its niceness value set to 19, without me having to look up the Process ID number.
Great. Now I just need my machine to look out for any new process named MayaBatch, and when it sees one, change the niceness value.
Here’s where cron and crontab come in. I can add that renice command into my crontab file and tell it how often to execute that command. For a simple command like renice, cron can execute it lightning fast, in a tiny fraction of a second, using virtually no resources. It has no noticeable effect on the user experience, working on the computer. Because it has so little impact on the computer’s performance, I run it once every minute. That way a render job will never hog resources for more than a minute and generally a lot less.
Here’s a short tutorial on the basics of cron and crontab on Mac OS X.
https://ole.michelsen.dk/blog/schedule-jobs-with-crontab-on-mac-osx.html
It works much the same on any *nix based system.
It works much the same on any *nix based system.
Based on the commands and syntax discussed above, we use a crontab file with the following contents:
*/1 * * * * renice 19 -p $(pgrep Commandline)
*/1 * * * * renice 19 -p $(pgrep MayaBatch)
This runs once every minute, looking for processes called either Commandline or MayaBatch. If it finds one, it sets the niceness value to 19.
I first ran this cron job on my own workstation as a test. My machine hosted several renders throughout the day without any noticeable impact on my own work, both in 3D and 2D. I’m sure having plenty of RAM and a very fast drive makes a difference. Ideally, you’d want to have enough RAM that the entire render job can be held in memory, along with whatever other work you are doing. We average about 48 gigs of RAM per machine and 8 to 12 multi-threaded cores, and that has been plenty.
We have now deployed the same cron job across all our animator workstations, and so far have been enjoying the additional productivity without any obvious negative side effects.