Happy 7th Birthday to Thomas

Installing DSpace in Amazon AWS Part 2: Handle Service

One of the things I didn’t cover at all in my first post on this topic was how to get the CNRI handle service working. The DSpace instructions are good enough for many installs, but I found myself at a loss as to why the local handle proxy server wouldn’t start. In this write-up, I will outline what went wrong and how the support at CNRI helped me fix it. And for good measure, I will also walk you through the setup.

Before I get started, it’s worth pointing out that you don’t have to use the CNRI handle service. DSpace makes its own handles, and these can be whatever you want them to be. Simply change the dspace.cfg file to reflect your local instance if you don’t plan to use this.

Okay, so the steps to get this working are as follows.

Make Handle Config
Send sitebndl.zip to CNRI
Edit config.dct
Edit dspace.cfg
Start Handle Server
Open EC2 Ports
Migrate Old Handles
Troubleshooting

Make Handle Config

As the dspace user, or one that has write privileges to ${dspace.dir} (usually /dspace/), run this script:

${dspace.dir}/bin/make-handle-config ${dspace.dir}/handle-server

Be sure, of course, to replace ${dspace.dir} with the actual location, like so:

/dspace/bin/make-handle-config /dspace/handle-server

Of particular importance in this script is the IP address. Make sure you use the EC2 instance’s public-facing IP address in this step! This is important! The address must be public-facing here. If you did everything I outlined in the previous post, you will have allocated an Elastic IP to this instance. Use that IP address.

Further in the script, you have an option to disable UDP. I do not know whether UDP is better or worse, or whether it even works here, but I have it disabled, so I only have TCP ports in use for the handle server. Choose whichever you like, but take note of it. It affects how you open ports later.

And finally, after you set the explicit configurations, you are presented with the generation of administrator and server keys. Choose to encrypt these, but when it offers you to enter a passphrase, leave it blank. I was not able to get this to work with a passphrase, and so if you don’t know how to do so either, just leave it blank.

This step will have generated everything you need in ${dspace.dir}/handle-server to run the server.

Send sitebndl.zip to CNRI

Now that everything is generated properly, find the file ${dspace.dir}/handle-server/sitebndl.zip file and email it to CNRI as per the instructions you should have received upon purchase confirmation.

From this, the CNRI administrators will set up your prefix and notify you.

Edit config.dct

Once you have received your prefix assignment, you have to make some changes in the ${dspace.dir}/handle-server/config.dct file.

First, all instances of YOUR_NAMING_AUTHORITY in the lines “300:0.NA/YOUR_NAMING_AUTHORITY” need to be replaced with your new prefix. There are three of these.

Second, as per the DSpace installation instructions, you have to add the following lines somewhere in the server_config section:

"storage_type" = "CUSTOM" "storage_class" = "org.dspace.handle.HandlePlugin"

I put mine just inside the server_config element:

"server_config" = { "storage_type" = "CUSTOM" "storage_class" = "org.dspace.handle.HandlePlugin" "server_admins" = ( "300:0.NA/YOUR_NAMING_AUTHORITY" ) ...

And finally, you remember how in the first step, the one that generated this file, you specified the public IP address of the EC2 instance? You should see that in three locations in this file. Now you need to manually change these to reflect the actual, private IP address of the EC2 instance. If you do not, the handle server will be unable to bind to the local ports, because it simply doesn’t know anything about its public IP. This took a good deal of time to figure out, and I wouldn’t have figured it out without help from CNRI, probably. See the Troubleshooting section below for the things I ran into.

Edit dspace.cfg

At this point you can edit your ${dspace.dir}/config/dspace.cfg file.

The dspace.cfg file has three entries of relevance:

# Canonical Handle URL prefix handle.canonical.prefix = http://hdl.handle.net/


# CNRI Handle prefix

handle.prefix =

# Directory for installing Handle server files handle.dir = ${dspace.dir}/handle-server

The only one you will absolutely need to change is the prefix, which in an OOTB DSpace is 123456789. You will set this to whatever prefix you are assigned by CNRI. If you used a different location for handle-server, set that here too.

Start Handle Server

If you’ve gotten everything configured, you should be ready to start the server. For this you will run another script from ${dspace.dir}/bin, called start-handle-server:

/dspace/bin/start-handle-server

Errors are written out to the error.log- file, so if you encounter any problems, look there first.

Open EC2 Ports

The last thing you’ll want to do is go to your Amazon AWS console and find the Security Groups you had applied to this instance. It’s either default or quick-start-1 or something like that. I am using quick-start-1. Click the applicable group and then choose the Inbound tab, where you will see the port mappings you can add and remove.

Add two custom TCP rules to allow port 8000 and port 2641 to be open.

aws-custom-ports

Apply changes once you’ve added the new rules.

Migrate Old Handles

The final step is to migrate any existing content you’ve already uploaded to the new handles. If you haven’t uploaded anything to your repository, then skip this step. Otherwise, run the ${dspace.dir}/bin/dspace update-handle-prefix to perform the migration:

/dspace/bin/dspace update-handle-prefix 123456789

And that’s it! Let me know if I am missing something or you encounter any problems with this tutorial.

Troubleshooting

Watch your log files for the following and related kinds of error messages. This, incidentally, is what I encountered when I had the handle server trying to bind to the public IP address of the EC2 instance. Suffice it to say that the virtual machine does not know its public IP address, and you have to re-bind the server to the local private address for the server to startup.

"2013/03/20 07:49:14 UTC" 100 class net.handle.server.HdlUdpInterface: Error setting up server socket: java.net.BindException: Cannot assign requested address "2013/03/20 07:49:14 UTC" 100 class net.handle.server.HdlUdpInterface: Error setting up server socket: java.net.BindException: Cannot assign requested address "2013/03/20 07:49:14 UTC" 100 class net.handle.server.HdlTcpInterface: Error setting up server socket: java.net.BindException: Cannot assign requested address "2013/03/20 07:49:14 UTC" 100 class net.handle.server.HdlTcpInterface: Error setting up server socket: java.net.BindException: Cannot assign requested address "2013/03/20 07:49:14 UTC" 100 class net.handle.server.HdlHttpInterface: Error setting up server socket: java.net.BindException: Cannot assign requested address "2013/03/20 07:49:14 UTC" 100 class net.handle.server.HdlHttpInterface: Error setting up server socket: java.net.BindException: Cannot assign requested address

Test post

This is a test post. Disregard.

Installing DSpace in Amazon AWS

So you’ve decided you need a digital repository, and you’ve chosen DSpace. Here’s how to host it in Amazon’s cloud.

DSpace is a reasonable choice, and it happens to be the least finicky and most modern out of its handful of competitors. But where to host it? Because it requires a Tomcat or other Java servlet environment, you can’t just toss it up in your average commodity hosting service. The next best bet is to get a cheap virtual machine, and Amazon is the place I chose. Here’s what I did.

Note: You could probably do this with equal success on Rackspace. If you do, your process will be different, of course.

The steps for getting DSpace installed and running on an Amazon EC2 instance involve:

1. Choose an EC2 instance type (and launch it, of course)
2. Configure your EBS storage
3. Configure the AWS security and Elastic IP
4. Setup an SSH connection
5. Install the DSpace prerequisites
6. Tweak some environment bits in preparation for DSpace
7. Download, configure, and build DSpace
8. Copy the desired webapps and start Tomcat

What isn’t covered here

I am not going to cover any kind of customization of DSpace. It’s complicated and well beyond the scope of a write-up on hosting. If you want your DSpace to do something that’s not completely obvious from the comments in the configuration files, you will have to do some research on your own. I am also not going to cover things like load balancing, disaster recovery, and the like, mostly because I haven’t done any of this myself yet. And finally, I am not going to do any hand-holding with regards to the Amazon AWS management console. I am assuming you know how to, for instance, launch a new instance and such. These are fairly intuitive if you’ve been around systems administration at all, I think. I will describe steps that pertain directly to the choices you have to make to get AWS configured for DSpace, though.

A note about cost

How much will it cost? That really depends on which choices you make above, but in general you only pay for what you use. Let’s just say if you’re looking to get by on a REALLY tight budget, you can (at the time of this writing; Amazon changes prices periodically) have a basic repository up and running for $16-$40 per month, depending on precisely how much storage space you’re using. But a non-enterprise class decently-performing system with, say, up to 1 TB of storage will run you on the order of $200ish a month, assuming you have used the entire terabyte of space. In my estimation, this is a pretty good price to showcase the benefits of a digital repository, after which you might want to spring for something a bit more robust. Aside from a few additional scaling considerations, though, I don’t think there is any significant difference between what I did here and what you might have to do in an enterprise setting.

Step 1: Choosing an Instance Type

If all you need to do is get something basic up and running, with no real care for performance and stability, a T1 Micro instance is probably sufficient. I have run one of these now for a number of months with few issues, but see below.

If you do choose a Micro instance, though, keep in mind you have pretty limited memory. If you don’t mind having your DSpace instance encounter occasional memory problems, terminate unexpectedly, or sometimes just perform badly, this option is for you. Do not take from this that it’s impossible or impractical to run DSpace this way. I just wouldn’t run it this way for anything you intend to use for production. As a development environment, this is perfectly acceptable.

Now, when you first choose to launch a new instance, Amazon will ask you what snapshot you want to use. Basically this is where you choose things like machine architecture (32 or 64 bit) and operating system. Unless you are experienced with a particular flavor of Linux and prefer it above all others, the Amazon AMI (64 bit) is recommended. It’s at the top of the list. It is a RedHat style Linux OS, which means you’ll be using yum for package management.

Move through the options, or accept the defaults, which appear to be sane for most purposes. During the instance setup, you have the option to configure Elastic Block Stores (EBS), which is extra disk storage for use with the instance. You may set this up as part of the instance creation, or you may do it later. If you do it later, you will just need to make sure that the EBS gets attached to the instance you created. I trust you can figure out where your volumes are listed.

The last thing you’ll do before launching the instance is to create and download your key file. This is important, because it’s the way you connect to your new VM. Save the file off somewhere where you can remember it. You will need it for Step 4.

Step 2: Configure EBS Storage

When I launched my first EC2 instance for DSpace, I didn’t configure a separate EBS for use with it, and I accepted the default 8 GB root disk size. For a number of months, this was not a problem. It became a problem when my team and I began testing out import procedures on larger volumes of content, and I ran out of disk space. Resizing the root disk for an EC2 instance is possible, but it is midly painful and requires significant downtime. Instead, I recommend, even for development machines, that you place the DSpace assetstore on its own EBS separate from the root disk. This will ensure that you can resize with less disruption, especially if you’ve also allocated an Elsatic IP (see Step 3 below).

Create your EBS with any size you like. With EBS, you are only paying for what you use, not what you have allocated. So if you choose to make the EBS 1 TB (the max), you won’t incur the maximum charge (currently $0.10 per GB per month, or $100 per month in this case) unless you actually used that much space. Allocate what makes sense to you. I have mine set at 100 GB, which gives me some room, but not too much. After all, I want a cap on the potential cost of these services, and limiting the disk space is one way to do this.

You’ll need to designate the device under which your instance can use the EBS. Typically these look like /dev/sdb or something like that. Pick something and remember what you picked.

Step 3: Security and Elastic IP

You need some security groups. These can be shared across all of your instances. I am using quick-start-1 to hold my custom security rules. This allows me to set network access policies so I can run and publicly access particular services. In particular, you will want SSH access so you can manage the instance from within the OS, and to install and configure things. I have additional ports open to support FTP, but you may not need or desire those. And finally, I have ports 80 and 8080 open for HTTP traffic. DSpace, because it’s a Java web application, will be accessible through port 8080.

Next up is Elastic IP. Each EC2 instance is supposed to include an Elastic IP allocation for free. This is useful because, if you have to take your instance down for maintenance (e.g., disk resizing), when it comes back online the underlying instance address might change. The public-facing Elastic IP will not. So you could, for instance, use a link shortener with a custom name (something TinyURL allows) to point to the public IP address without worrying about the shortened URL becoming inoperable. It’s a consideration, anyway.

Allocate and associate your Elastic IP. There’s not much else to configure.

Step 4: Setup SSH

The Linux snapshot I recommended earlier has an SSH server installed by default. You will connect to it through a single account, ec2-user, but to configure your connection, you will have to tell your SSH program where to find the authentication key. On Windows, you are probably using PuTTY for your SSH needs. I don’t know how to configure the key for use in other systems, so you’ll have to search around to figure that out if you’re using something else. Presumably, if you’re using Mac or another Linux machine, you’ll just specify the key on the command line or something.

Anyway, with PuTTY, you need the related PuTTYGen program, available at the same site you download PuTTY. Go get it, run it, and then choose Load to load an existing key file. Browse to where you saved the .pem file, show All files (*.*) and select the .pem file you had previously downloaded. Open it, then Save private key. Put it in the same location, or somewhere where you can find it again.

Now open PuTTY and in the Session pane, enter your AWS instance address, or the Elastic IP address you associated, in the Host Name field. Expand the SSH menu and choose Auth. Browse to and select the .ppk key file you saved with PuTTYGen. Go back to Session enter a name for this session (aws, perhaps), and click Save. The new session should appear in the list. Double-click your session to open the connection, and at the login prompt, enter ec2-user. If all was configured correctly, you will have successfully authenticated. Now you can finish the other preparations for the system to receive DSpace.

Step 5: Install DSpace Prerequisites

Most of this part is, or should be, straightforward and is mostly documented at the DSpace site. In general, you will need to make sure that your environment has the following components to build DSpace:

Java 6+ (Java 6 JDK is included with the Amazon AMI we are using, but you can also install Java 7 with no issues. I will go through the Java 7 install below)
Postgresql 8.4+ (If you use Oracle, you’re on your own; I will be using pg 9x)
Tomcat 5.5+ (we’ll use Tomcat 7, but Tomcat 6 seems to be better on low-end machines; use either)
Apache Maven (we’ll use 3x, no problems)
Ant

I am assuming you are running this as root or a privileged user. To become root:

sudo su -

Java 7 JDK

yum install java-1.7.0-openjdk

Once you do that, though, you need to change your default java binary. These are listed in /etc/alternatives and are symlinked. I have changed my symlink to the following.

lrwxrwxrwx 1 root root 59 Feb 25 20:21 java -> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre/bin/java

You can do that by running the following:

cd /etc/alternatives unlink java ln -s /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre/bin/java

Postgresql

yum install postgresql-server

There are some additional tweaks to make in a bit, but we can wait a few minutes to get the rest of the stuff installed.

Tomcat

yum install tomcat7 tomcat7-admin-webapps

Don’t start Tomcat yet. There is still work to do.

Apache Maven

Download it from here. Get the latest version, probably. I am using 3.0.5 with no problems. Use the binary tar.gz.

Untar and gunzip it, then copy the resulting folder to someplace like /usr/local/apache-maven/

In a moment there are some environment variables to set, but we don’t want to do this for root.

Ant

yum install ant

That’s all of the prerequisites. All of these are documented in the DSpace documentation. What’s not documented very well are items I cover in Step 6.

Step 6: Tweak the Environment

We have all of the prerequisites in place, but the environment still needs a few things for DSpace to build.

We need a DSpace Linux user to own the DSpace installation. Create one.

useradd -m dspace

Now let’s initialize the database.

service postgresql initdb

This creates the files for the database. By default, the database is created in /var/lib/pgsql9. I don’t know what the size considerations for large DSpace installations look like, but from what I have experienced to date, you will be safe if you keep the database on the root disk. If you think you want it somewhere else, the additional EBS store is not a bad place, but make sure you have a dedicated place for it.

I encountered a baffling setback when I tried installing DSpace on a Postgresql 9 database, in that by default it wants to use ident. Looking on a different system, I noticed that Postgressql was configured to trust both local and host (localhost) connections. So I set that up by editing the pg_hba.conf file generated in the previous step.

vim /var/lib/pgsql9/data/pg_hba.conf

Edit it to match the following. All other lines should be commented out.

local all all trust host all all 127.0.0.1/32 trust

I don’t know if there is anything insecure about this, but this does work, and previous installs of DSpace relied on it being this way.

Briefly, become the postgres user, so you can create our DSpace database and user. Enter each command and follow any directions. Note the password you set in the createuser step. You will need it later.

su - postgres createuser -U postgres -d -A -P dspace createdb -U dspace -E UNICODE dspace exit

Next, it’s time to mount the EBS created to hold the DSpace code and assetstore.

vim /etc/mtab

Add the following line to the mtab file. Note the device name you picked earlier, when you created the EBS and attached it to the EC2 instance. Here we have /dev/sdf but if this is the first EBS you’ve created for a non-root disk, it could be /dev/sda

/dev/sdf /dspace ext4 rw,noatime 0 0

Next go to the /dev directory so you can make the filesystem. Make sure the device you named and mapped in /etc/mtab is present in this directory.

cd /dev ls mkfs.ext4 sdf

Now mount the device.

mount /dspace

Now it’s time to change some group memberships. This is not strictly necessary, but it might help mitigate potential permissions problems. I added the tomcat user to the dspace group and the dspace user to the tomcat group.

vim /etc/group

dspace:x:501:tomcat tomcat:x:91:dspace

Next let’s change the /dspace directory to be owned by dspace.

chown -R dspace /dspace

Now become the dspace user so you can add the environment variables.

su - dspace vim ~/.bash_profile

Add or change the following lines:

M2_HOME=/usr/local/apache-maven/apache-maven-3.0.5/ M2=$M2_HOME/bin PATH=$PATH:$HOME/bin:$M2

Here we’ve pointed M2_HOME to the location where we installed the Maven binary, set the M2 variable to its bin directory, and added the bin location to our path. With the environment variables in this file, these get set every time the dspace user logs in. So we need to login again to make sure they are correctly set.

exit su - dspace

Step 7: Download, configure, and build DSpace

This entire step assumes you are the dspace user, which you will be after following the previous steps.

Download the DSpace code from dspace.org. Since it comes from SourceForge, you’ll have to click to download, then copy the direct link, pasting it in your SSH window.

For example:

wget http://downloads.sourceforge.net/project/dspace/DSpace%20Stable/3.1/dspace-3.1-src-release.zip?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fdspace%2F%3Fsource%3Ddlp&ts=1361825630&use_mirror=iweb

Unzip the file and perform the following actions.

unzip dspace-3.1-src-release.zip cd dspace-3.1-src-release vim build.properties

Follow the directions in build.properties to get things set up. The comments are pretty helpful, but you can also refer to the DSpace documentation if you need assistance. What these get set to may depend on your local needs.

Pay particular attention to the db.password, which should match what you set for the dspace user above.

Many of the defaults are fine to use.

After saving and exiting the build.properties file, you can finally begin the compile process.

cd dspace mvn package

Wait for the mvn package command to finish. It can take a while. Once it’s completed successfully, you will see a big SUCCESS message. Now move into the target directory and do the actual install.

cd target/dspace-3.1-build/ ant fresh_install

This creates the database schema, adds default data, copies the DSpace code, and such. Watch to make sure this is successful as well. It takes some time.

Step 8: Copy the desired webapps and start Tomcat

Finally you can copy your webapps to where Tomcat can use them. If you know which particular webapps you want to copy, you can copy them one by one, or you can just copy all of them.

One by one, for example:

cd /dspace/webapps cp -R xmlui/ /usr/share/tomcat7/webapps/

cp -R /dspace/webapps/* /usr/share/tomcat7/webapps/

Start Tomcat:

/etc/init.d/tomcat7 start

It takes Tomcat a while to start, sometimes as long as 20 minutes if you aren’t very lucky. Mostly it takes around 2-5 minutes.

Wrapping it up

That’s it! You should now have a functioning DSpace. Other things you’ll want to do include creating an administrator account, testing the web application, and such. Try following along in the DSpace manual or reading up on the various command line tools available to find out what else you can do, including advanced configurations and such.

Discuss this with me on Google+: https://plus.google.com/u/0/102013352492248128705/posts/Ly1LsBhksSv

Arduino HVAC Thermostat: Update

Project page

Well, I think I am finished with this project. I have one more night of testing to make sure all of my assumptions bear out, but even if they don’t, it’s just a matter of refining from here.

When I originally set out to build this thing, I had plenty of examples online that used regular Arduinos. Had I followed suit, I guess I would have ended up having to solder something. I was lucky that the Esplora came out recently, and while it seems to have generated mixed reviews, I love that it contains enough features to get someone started doing real things with it. I was able to use the slider (a potentiometer) to effect three control states, the thermistor, which seems to be sensitive enough for my purposes, and I even tied into the light sensor to automatically switch from day to night.

This will probably be the last big post on this topic, unless I decide to expand my skillset to include a TFT touch screen interface, or the much crazier idea of adding voice controls. Perpahs, though, I will provide a one month update. Anyway once winter is over, it will be a while before I need my HVAC at all.

Arduino HVAC Thermostat

Since I live in a NYC apartment, I decided it would be fun to try my hand at the kind of HVAC thermostat outlined in this Instructable. This is my inaugural post on the subject. The project page is here: http://stuff.aaron-helton.com/doku.php?id=long_projects:arduino_hvac_thermostat

For the kickoff post, I am really just gathering up supplies and figuring out how things work. So far I have acquired an Arduino Esplora and a Parallax Standard Servo. I managed to get them working together despite some difficulty.

Esplora + Parallax Servo

One of the issues I encountered immediately was that the Esplora has outputs that are compatible with TinkerKit modules, while it turns out the Parallax Servo is not wired quite the same way. Because I am pretty much clueless when it comes to electronics, this fact escaped me, resulting in several hours of bafflement when the servo didn’t work.

If you notice the wire order running into the servo housing, it is Black, Red, White. TinkerKit assumes that the Red and White wires are swapped, which you can see at the connector end. Once I did that, I was able to plug in the servo to the Esplora and have the Esplora turn the motor, no problem.

Code

As soon as I finished that, I got to work on the code. There is still much to do, but the basic code works. The reason I didn’t re-use code from the original Instructable is because I am using a built-in temperature sensor and a completely different servo, but I think the principles are the same. The code is available on the project page.

That’s as far as I got this weekend, but there will be more to come.

2012 World Maker Faire

I’m planning to go, are you? Click the logo to lean more!

Maker Faire NYC 2012 Logo

If you’re willing to hand out and/or post promotional materials in your NYC neighborhood, you could score free tickets. The deadline has been extended to September 7 (was August 31): http://blog.makezine.com/2012/07/23/call-to-join-the-world-maker-faire-street-team/

A Book Closes

A story of my house on 13205 Rochester Ln in Austin, TX, which can be yours for the paltry sum of $160,000. Listing here: http://www.texashomesforsale.com/austin/pflugerville/home/13205-Rochester-Ln,-Austin,-TX-78753/8374925

Work in Progress

We never imagined, when we bought the place, that this day would come and we would actually sell our house, but after we left Austin to appease our wanderlust, it became all but inevitable. The sensation is something like reading a compelling book, one so enjoyable that putting it down is not an option, even though you know you are hurtling from the front cover to the back cover, and when you reach it, it will be over. Personally, I find this both exhilarating and saddening because, while the house has been something of a friend to us (we occupied it for four years before renting it out), the end of this particular book is in sight, inevitable and unavoidable. We cannot stop now.

In the heady days of 2005, home ownership finally seemed a reachable goal and a desirable prospect. We had moved to Austin at the end of 2003 with no real job prospects beyond S’s having found a long-term substitute teaching position within Round Rock ISD. I began collecting unemployment and food stamps at the end of December, 2003, after I finished my terminal leave in the Army and my Release from Active Duty (REFRAD) came through. To top it off, we had a baby (DP) on the way, due sometime in late June or early July of 2004. If 2005 was the time to buy a house, it was only because early 2004 was so much the opposite. Things didn’t start looking much better even once I started a job in mid-March of 2004 as a contractor in Dell’s call center. My $13.25 per hour paycheck gave us breathing room, but only just. Even so, long-term career prospects within that segment of Dell are hard to come by, and there was a distinct possibility I would have been laid off 9 months later in a downsizing exercise.

That changed a couple months later when a previous interview panned out and landed me another contract job, this time doing phone support and problem analysis at a company called Hire.com, one of those scarred survivors of the Dot Com era. Even though it was still a contract job, I had no hesitancy in taking it: it paid $25 an hour, which for the first time since leaving the Army gave me more than enough to take care of my family. I started in May, and DP finally was born in late June, which was fine by that point because now we could afford some health insurance.

By April of 2005, I had been in my job a year and was set to convert from a contractor to a full-time employee, along with a pay raise, which meant now I could have the company pay for some of my health insurance, and my position was much more secure. This is what I mean by heady days. At the time, I’ll admit I felt pretty good about things. A promotion in June further solidified this, but by that point I had already bought my house and moved in with grand plans for things I wanted to improve, repair, or otherwise change about it.

Because I am somewhat reserved by nature, I do my best to keep emotions from clouding my vision. And this time was no different. Looking back, I can see that I bought my house near the peak of the market, give or take. In the years since, the market value hasn’t really gone anywhere, and one year the tax appraisal came in far below our original purchase price. If I had given in to emotion, made the kinds of unwise decisions that were so common then, this might be a completely different story. Having been unemployed works a kind of magic on you, such that you always fear ending up back there, and buying a house is the kind of commitment that we know now carries with it grave consequences should one buy at market peak more house than one could afford.

I knew that, if I bought too pricey a house, should I fall into unemployment again, there would be no way I could afford to pay the mortgage on unemployment payments. Further, I wanted to ensure that if something happened to me, the mortgage could be paid on an even lower income. This little bit of upfront wisdom was the difference between keeping my house and being devastated by foreclosure when I did eventually find myself back in the grip of unemployment.

In the meantime, things in my household had never looked so good. Within a month of my promotion at Hire.com, the company was acquired by a slightly larger company in a similar product space, a company called Authoria. As with all such acquisitions, this one came with layoffs and employee attrition, but I came through just fine. That event alone was not enough to shake my confidence too much, but after another year in my position, I began to put out feelers for what else the company had to offer. I applied for acceptance into a graduate program at St. Edward’s University and began that in mid-2006, using up what money I still had from the GI Bill and supplementing that with tuition assistance from Authoria.

At the same time, I was moving down a pretty radical path of self-improvement beyond just a graduate degree. By the end of 2007 I had shed the nearly 50 pounds I had gained after leaving the Army (and I’m proud to say I’ve maintained my weight since), saw an orthodontist for the second time in my life (the first time I decided against braces), signed up for braces, and generally began more critically examining the world I lived in, including where I worked. I’ll admit all of this growth made me start to chafe where I was. From what I could tell, there was no proper career path for me at Authoria, and so if I wanted to use the growing body of knowledge I was collecting, I was certain I was going to have to do it somewhere else. Add to this my growing suspicion and distaste for the normal business cycle that includes intake and mass layoffs, and you can see why I eventually decided to get out, moving instead into government and international civil service, where I can chart an actual course for myself. In the process, I can hopefully help make the world a better place, even if it’s just in small ways.

And so it goes. The rest of my time in Austin I captured pretty well on my blog, I think. But now it’s time to close the book. My family and I enjoyed the house and put it through its paces. We painted walls and I laid out new tile here and there. We potty-trained two children in that house and laughed, learned, and loved there. So while I am glad I won’t have to hold on to the house now that I’ve cast myself adrift in the world again, I do admit feeling it’s a bittersweet parting.

I hope whoever buys the house enjoys it as much as I and my family have. It’s really a nice house.

ImageMagick Woes

This post is aimed at those who use a shared hosting provider like 1&1 (or, so this can be searchable, 1and1), have some sort of PHP application, and need to use ImageMagick, specifically the convert utility to create thumbnails and such.

Background

In pursuit of a place to catalog and store my personal digitized and digital items, I went through a quick evaluation of various Digital Asset Management applications that run on a typical LAMP/ish stack. After trying out several, I finally settled on Omeka, which uses PHP5 and MySQL and runs best on a Linux system. Omeka was the easiest of the applications to get working and keep working. I have it running now so I can build out a personal digital library (warning: it’s not ready for production use, at all).

There were some problems even with this application, however, with the use of the convert utility (thumbnail creation failed) and it took me several days to track down the source and apply a fix. The rest of this post talks about the problems and my solutions. Part 1 deals with the internals of Omeka specifically (largely so you can get good error messaging in the first place). Part 2 has some general lessons for anyone else using PHP applications on providers like 1&1 and making use of ImageMagick. Skip ahead if you are really only interested in the more general PHP/ImageMagick stuff.

Part 1: Omeka Internals

When you first install Omeka, there won’t be any logging capability turned on, and so if you run into problems, you need to make sure that’s the first thing you do. In OMEKA_ROOT/application/config there is a file called config.ini. Edit that file so that the Logging section has log.errors = true:

;;;;;;;;;;; ; Logging ; ;;;;;;;;;;;

; log.errors ; Log errors and other information. ; default: false ; ; Errors, exceptions, and other messages will be logged to ; application/logs/errors.log (this file must be writable by the web ; server if logging is enabled). log.errors = true

Your application error logs will appear in OMEKA_ROOT/application/logs/errors.log

Unfortunately, even with logging enabled, sometimes it’s not at all clear what’s going on when something fails. Since this post is about ImageMagick and thumbnail creation, I am only going to look in the part of the application responsible for thumbnails, Image.php. You can find this script in OMEKA_ROOT/application/libraries/Omeka/File/Derivative/

Note that CASE MATTERS here.

The main function in Image.php that handles image creation is called createImage(). It builds the parameters string for the convert command and then executes it, passing back the value of the $newFileName if everything works.

If it doesn’t work, the best Omeka can do out of the box is write in the log (or on the screen if you decided to configure it that way, but don’t do that if you can avoid it):

‘Something went wrong with image creation. Please notify an administrator.’

Followed by a stack trace. Instead of wondering at the folly of telling someone to notify an administrator through log files, let’s see if we can’t do something about the error message. Like make it useful.

Turns out we can modify the command being called through the exec function. The original command looks like this:

exec($command, $result_array, $result_value);

But if we modify it like this:

exec($command.' 2>&1', $result_array, $result_value);

We now have the full set of output held in $result_array, success or fail. We only care, of course, about fail.

So now we have to look at the part of the same function that tests for success:

if ($result_value == 0) { return $newFileName; } else { //throw new Omeka_File_Derivative_Exception('Something went wrong with image creation. Please notify an administrator.'); }

If something goes wrong with the image creation, that’s all that we see in the logs. Something went wrong. Now that we have some additional things in our $result_array, we can put THAT in the logs just like this:

if ($result_value == 0) { return $newFileName; } else { throw new Omeka_File_Derivative_Exception("Tried to run command: \n$command\n\nGot result: \n$result_array[0]\n$result_array[1]"); }

It’s not the most beautiful code in the world, but it has the advantage of providing information I can act on. If you run into errors anywhere in your Omeka install, and you keep getting uninformative errors like “Something went wrong,” try tracking them down to the file they occur in and see about pumping out any variable values you have whenever the error happens. That’s how I found this anyway (well, that, and by following along with a different approach suggested in one of the Omeka forum threads).

Part 2: PHP and ImageMagick

Once I got Omeka to give me better errors, I was able to track down this little gem from OMEKA_ROOT/application/logs/errors.log:

2012-06-22T21:26:01-04:00 ERR (3): exception 'Omeka_File_Derivative_Exception' with message 'Tried to run command: /path/to/convert 'OMEKA_ROOT/tmp//cfee5f76ba0f296eb2f1853c8c5aaad5.jpg' -resize '800x800>' 'OMEKA_ROOT/tmp/fullsize_cfee5f76ba0f296eb2f1853c8c5aaad5.jpg'


Got result:

libgomp: Thread creation failed: Resource temporarily unavailable' in OMEKA_ROOT/application/libraries/Omeka/File/Derivative/Image.php:245

Thing is, ANY PHP application on 1&1 can run into this if it makes use of ImageMagick. The solution was uncovered by Drupal user Ben Mango and posted in the ImageAPI Issues comments. Because ImageMagick has some memory issues, 1&1, along with probably most other shared hosting providers, put limitations in place that keep it from getting out of hand. If you don’t take the necessary precautions, your scripts will fail like this. The one simple thing you can do to keep this from happening is to, somewhere where your script can make use of it, add the following line:

putenv("MAGICK_THREAD_LIMIT=1");

I figured the best place to put this was inside one of the top-level functions in Image.php:

public static function createAll($originalFilePath, $fileMimeType) { // Don't try to make derivative images if we don't give a path to // ImageMagick. if (!get_option('path_to_convert')) { return false; }


        //Do this on 1and1
        putenv("MAGICK_THREAD_LIMIT=1");

self::checkOmekaCanMakeDerivativeImages(); return self::isDerivable($originalFilePath, $fileMimeType) ? self::createDerivativeImages($originalFilePath) : false; }

Solving this took me far too long, but with a bit of luck, others who have this problem or a similar one won’t have to spend as long looking for answers.

Discuss on G+

Google Currents vs Flipboard

I didn’t really know I was waiting on anything in particular before writing this post, but it turns out I was. I was waiting on Google to announce its API partnership with Flipboard. So now it’s time to revisit my earlier review of Google Currents.

For those of you who don’t want to read further, let me leave you with this: use Flipboard instead. It’s better.

Read on for more details on my updated Currents conclusions, starting with what was fixed.

Availability

When Google Currents was first released in December 2011, it was for US users only. When Google updated the app in April 2012, it finally opened to an international audience. This was absolutely the right move, but unfortunately availability wasn’t the biggest problem, at least not for me.

Google+ Integration

I admit to being pretty harsh in my assessment of the level of integration the first version of Currents had with Google+. Part of that was sour grapes over Google’s squashing of social features in Google Reader, a product I had grown incredibly dependent on. At its launch, Currents was able to add public +1s to items. And that was it. I still had to go to a computer to post it to G+. This was underwhelming, of course.

Google fixed this in the April update, (well, mostly; and then fully). Great, I thought, they fixed at least some of the things I had complained about. Unfortunately…

Google Reader Integration

This was still suboptimal. I am not going to repeat the same rant here, because there’s no point. Suffice it to say I still couldn’t get Currents to use the folders I had created to categorize my Reader feeds. Without that, there was no way I was going to pick through my feeds to see which ones I should add individually to my Library. This is the kind of task I do maybe a couple times a year, and it’s not pleasant to do from a mobile device. So again, even after the April update, there was no compelling reason to use Currents.

Enter Flipboard

I met the news of Flipboard’s API partnership with Google+ with a bit of puzzlement and a lot of excitement. And my excitement only grew once I discovered that Flipboard actually does, quite as expected, integrate with Google Reader and my feed folders. So guess what? I’m passing on Currents and using Flipboard instead. Flipboard has already been doing what Currents seeks to do, and it does so better than Google’s own app. Google is going to have to really make Currents sing for me to contemplate it again.

My conclusion? Use Flipboard. Discuss?

.:aaron.helton:.

musings.of {'husband', 'father', 'technologist', 'international civil servant', 'humanist', 'skeptic', 'gamer', 'dreamer'}

Happy 7th Birthday to Thomas

Installing DSpace in Amazon AWS Part 2: Handle Service

Test post

Installing DSpace in Amazon AWS

Arduino HVAC Thermostat: Update

Arduino HVAC Thermostat

2012 World Maker Faire

A Book Closes

ImageMagick Woes

Google Currents vs Flipboard

Share this:

Share this:

Share this:

Share this:

Share this:

Share this:

Share this:

Share this:

Share this:

Share this: