Saturday, 15 October 2011

A bad weekend of Apple, Ubuntu and me

Over the weekend I have had 2 technical problems. And a third one which was the emotional consequence of the 2 technical ones.

So the first problem was a result of iOS5 update. My iTouch 4th gen became unstable. I liked my iTouch, iPad and iXXXXX because they were always responsive and reliable. No lags no glitches no errors. But after I have installed iOS5 on my iTouch. My oh my!!! Lags are there and makes me feel like on Windows.
The other issue seems to be a bug. You can test it by yourself.
Start "App Store" - Select "Updates" - Select "Purchased" - Select "Not on this iPod" - Swipe right to left on any item and then select the "Hide" button.
In my case "App Store" always does exit.
Steve Jobs have died just one week before the release of iOS5 and here we are. Everything I really like for iOS devices have gone. Think about it.

The other problem is more serious. And to be honest more depressing to me.
I have upgraded my Ubuntu 11.04 to Ubuntu 11.10 on my "office" notebook.
And for first look I hit bug "Booting system without full network configuration..."
What does it mean you may ask? A normal user who runs into this bug can not work anymore on the machine. So this is THE experience a normal user can not tolerate and will change to something else. Good question what the next choice will be. Back to windows?

To me the conclusion was that instead of companies making products more stable -  Apple and Linux like - Apple and Linux are making themselves more of a Microsoft like. And sorry to say but that is really-really depressing me.

I beg you Apple and Ubuntu to follow your own path and forget the Microsoft one.

Friday, 10 June 2011

Why ec2-ami-tools - ec2-upload-bundle - hangs

Me and my colleagues had been fighting an issue for some days. Since it made me really-really f**** upset I decided to investigate it now.

The issue: is that if you are behind a proxy (not necessarily) - http or socks does not matter much - you might have problem with ec2-ami-tools. When you start ec2-upload-bundle it does nothing just looks like hanging. We had been waiting for some minutes but nothing happened. Actually it is trying to do something but can not start upload.
In contrast on my home network ec2-upload-bundle almost immediately starts uploading and showing a message to indicate it.
So what is wrong? Why this is happening from behind a proxy? Is it proxy related?

We have tested many possible workarounds without any luck.
First we have specified http_proxy and https_proxy of course but it did not work.
Then we also tried to socksify the process. Without any luck.
At that point we decided to eliminate the proxies. We got an access to a machine in DMZ and we made a dynamic port forwarding and tried to use that as a socks proxy server. And that was really surprising because even that did not work.
Everything else - Firefox, ssh - was working fine but not ec2-ami-tools.
Here we ran out of the ideas for a while.

But  of course problem happens to be solved.

So I made a tcpdump, had a look and saw that the process is trying to access a special IP address 169.254.169.254 . This is a link-local IP address and this is used by amazon within the cloud to fetch some instance specific data. It is unclear why ec2-upload-bundle is trying to reach it. It is much more unclear how the program is written to be able to hang on it. I mean it is also visible that it makes a retry before the TCP timeout (0,3,9 sec) but than it gives up and hangs.

Workaround: to the problem can be to configure this 169.254.169.254 IP on the ssh server or http/socks proxy server. Also you can try somehow locally - with a firewall - reject these connections. But you have to prevent the connection to timeout since if the connection times out ec2-upload-bundle will hang. See the note for an alternative workaround.

Note: This is working on my home machine because normally my Ubuntu Linux has a 169.254.0.0/16 route. And this is enough to prevent the timeout. But only if you are not using proxy. If proxy is in between than your own routing is not used to reach this IP but the route of the proxy is used. So you have to add this route to the proxy/ssh server if you can.

route add -net 169.254.0.0 netmask 255.255.0.0 dev eth0 # Replace eth0 with your interface.


Comment: I have less motivation now to investigate it further but probably I will check what is wrong in the code and why ec2-upload-bundle hangs in such a case.

Update 1: I have checked the source code of ec2-ami-tools and it seems the error is in instance-data.rb. The initialize method is checking if the meta data is available or not and set @instance_data_accessible according. But there is no timeout defined here so you have to wait ... It seems even the TCP timeout is not realized by the open method. A Timeout::timeout(10) { } can solve this error.

Update 2: Later in instance-data.rb the read_meta_data(path) method should call the open() method only if @instance_data_accessible is true. But this is not the case. I do not know ruby at all but this seems to be a coding error. I have just added some print command to see what was going there but do not understand the code really. Besides the open() methods here also hanging indefinitely. So this is a complete hang here.

Saturday, 30 April 2011

How to create a custom RedHat / CentOS Amazon EC2 / Cloud image

Note: Cloud computing is currently a very dynamic segment of IT and relevant changes - regarding technologies, available features and licensing - might happen anytime.

First of all you may ask why to build your own AMI when you can find many existing  - CentOS and other free - Amazon EC2 images.

The following sentences are taken from an official amazon document.

"You launch AMIs at your own risk. Amazon cannot vouch for the integrity or security of AMIs shared by other EC2 users. Therefore, you should treat shared AMIs as you would any foreign code that you might consider deploying in your own data center and perform the appropriate due diligence.

Ideally, you should get the AMI ID from a trusted source (a web site, another EC2 user, etc). If you do not know the source of an AMI, we recommend that you search the forums for comments on the AMI before launching it. Conversely, if you have questions or observations about a shared AMI, feel free to use the AWS forums to ask or comment."

So what if you do not trust those public AMIs?  And where are the official RedHat images? (Old document is here)

Update: When I started to write this post - 2011.04 - RedHat did not have any available public AMIs in amazon. Besides that RedHat licensing was really-really restrictive and using your own image required special RedHat subscription and transferring 25 subscriptions to the cloud for at least half a year (at least as much as I understood the license).
 
Some historical documents:
RedHat Cloud Computin(Had been removed)
RedHat Cloud FAQ (Archived)


Licensing in the cloud is another general problem. Amazon released EC2 beta in 2006 and went public in Q4 2008. Although 3-5 years have gone OEM licensing is still a very big problem in most cases. OEMs are trying - or seeing the results probably not really - to adapt licensing and changing the cloud (amazon) licensing very frequently. Sometimes releasing paid AMIs and covering license fees such a way, sometimes requires special licenses and sometimes allows you to use your existing licensesSo probably if you can not find something now or license is not available currently, it will be available in a month, in week or even in a day. 

Update: RedHat has released some new payed AMIs. For some additional hourly fee you can use these instances and you will not have licensing problem. Search for the owner 309956199498 in the list of available AMIs and you can see RedHat AMIs. 

See RedHat pricing here and standard pricing here for comparison.
Other historical RedHat announcements:
RedHat 5.5 32/64
RedHat 5.6 32/64
RedHat 6.0 32/64
RedHat 6.1 32/64
Common RedHat AMI updates
RedHat on Amazon EC2

So if you still want to build your own image - because of security, licensing or because you have no time to wait till it is officially released by the OEM - amazon has a GOOD description how to build your own image.

BUILDING YOUR OWN IMAGE IS ALWAYS A RISK FROM LICENSING POINT OF VIEW!!!
MAKE SURE YOU HAVE THE PROPER LICENSES BEFORE USING YOUR OWN IMAGE!!!
OR BETTER USE FREE SOFTWARES IN THE CLOUD.
(Debian, Ubuntu, CentOS)

I can only add some probably hopefully useful information.

If you want to create CentOS image via yum you have to use a pre installed CentOS operating system as a build server. (CentOS is free to use in the cloud and have a central repo even for install and update.)

If you want to create RedHat image via yum you have to use a pre installed RedHat operating system as a build server. (Licensing of RedHat might be a problem. And upgrading from the cloud via RedHat Network [RHN] is probably not the best idea.)

I recommend to use a 64bit OS version as a build platform since you can use that to build both 32 and 64 bit images. On a 32bit OS however you can build only 32bit images.

I also recommend to use the latest version of the OS. It is possible to install older version of the OS on the latest version. (It is most probably possible to install latest version of the OS on an older version but might have some unavailable new features - for example missing file system support.)

I prefer to use original kernels from RedHat and CentOS and not using the available amazon AKIs.

You can make your life easier by disabling SELinux - or at least running in permissive mode. After you could build a running image you can try to enable SELinux again if you need. (And I would suggest to use it if possible.)

When you are installing the base OS with yum the amazon document shows how to create a yum configuration file by hand. CentOS has a so called "release" file which contains the yum config. Using that it is more automatic and version safe. RedHat also has a release file but yum config is needed there.