[ FEED ]

When looking for a solution to store offsite backups during my travels, I was interested in uploading incremental, as well as encrypted backups to Amazon S3, since the API is simple to use, and the price model for their service is affordable, especialy when using Amazon's Lifecycle management to transfer S3 objects to Amazon Glacier automaticaly). When looking deeper into this topic and evaluating available tools I discovered that most of the available S3 clients where not able to stream data you pipe into them via STDOUT / STDIN directly (using the RAM) to the S3 bucket, but need to store the whole data as a file on your harddrive before being able to upload. The main reason for the lack of this feature is that Amazon's S3 API expects the the size (Content-Length) of the data to be stored, before receiving the data itself. After doing further research for tools facing this problem I found js3tream which tries to get around this problem by splitting the inputstream into chunks of fixed size and storing them as seperate objects in the given S3 bucket. But when trying to work with this tool I got nothing but a bunch of exceptions, and even when my Java is acceptable I was not able to spot the root of the problem. Anyway I've seen that Amazon, since 2010 supports a more suitable way to upload files in multiple chunks, than js3tream tends to use. Amazon's S3 API provides the opportunity to upload an object in multiple parts, and expects the Content-Length of each part uploaded instead of the Content-Length for the whole object. Using this mechanism, it is possible to split the inputstream into chunks and upload each chunk until the end of the inputstream is reached. When searching for tools supporting this feature, I only found clients being able to read files, instead of reading from STDIN. While s3cmd is able to stream stored data from S3 to STDOUT it is not able to stream data from STDIN to S3, even though they announced to integrate this feature. Since I could not find any client suitable for my needs, I wrote some small python script, using boto's S3 interface to upload data read from STDIN chunkwise to S3 using Amazon's multipart upload API.

The script can be found on github. To use it for incremental backups you can make use of the tips given in the tar HowTo of the js3tream project. To encrypt your backups you can pipe the data generated by tar through gpg.

# tar -g /etc/backup/home/diff -C / -vcpjO /home | \
> gpg -r com-example-backup-home -e | \
> 2s3 -k /etc/backup/home/aws-key -b com-example-backup-home -o backup.0

This example assumes that the file, keeping track of the changes in your filesystem, as well as the file containing the AWS credentials for the access to your S3 bucket are stored in /etc/backup/home/, and you generated a GPG keypair named com-example-backup-home in the users GPG keyring (# gpg --gen-key).

By running this command and incrementing the objects name (backup.1, backup.2, etc) every run, you can construct a incremental, encrypted backup of your user's home directories.

To restore your backup, you can read the S3 object(s) from S3 using s3cmd and pipe the data through GPG and tar the other way round.

# s3cmd get s3://com-example-backup-home/backup.0 - | gpg -d | tar -g /dev/null -C / -xvj
# s3cmd get s3://com-example-backup-home/backup.1 - | gpg -d | tar -g /dev/null -C / -xvj

Before using s3cmd, you have to configure it to connect to your S3 bucket. Have a look into the s3tools documentation for further information.

Your comment

I had a tough way from Pécs to Croatia. I walked the approximately 40km across the border and took a bus to Osijek. I just arrived a few hours ago and already met nice people guiding me to a nice bakery for some food. Today the whole country is full of red candles due to the memorial of the end of the Battle of Vukovar. I'm going to read a bit about this before falling into dreams.

Your comment

I finally managed to publish some of the pictures I took on my way. It is just a small selection so far, but now where it is easier for me to add new photos I hope to do it more regularly! I hope you will enjoy some of the impressions!

Your comment


Since some people where afraid that I got arrested, fell down a canyon or got eaten by a bear, I am back here to tell you: I'm alive! A lot of exciting things happened since my last post in June. After I left Krakow I went down to the polish border and first spend some time in Zakopane, which polish call the winter capital of Poland. Zakopane lies at the foot of the Tatra Mountains, a mountain range that forms the border between Poland and Slovakia. It is even possible to hike from Poland to Slovakia (and vice versa), crossing the peak of Rysy, which lies at the border between Poland and Slovakia and at the same time forms the roof of Poland. Since Zakopane appeared to be a quite touristic place (even during the summer season) I decided to leave this place after some days and explore some place I heard about back in Olomouc: Ždiar.

The small village is located just about 15km behind the Slovakian border, and provides a amazing view on the Belianske Tatras, the eastern part of the Tatra Mountains. It is a very good base to explore the High Tatras. The Tatras provide a lot of well maintained, but still demanding hiking trails to explore the beauty of the mountains. In between people built smaller huts (Chatas), including a restaurant with traditional slovak food (e.g. Bryndzové halušky, Cesnaková polievka or Guláš) and usually providing accommodation, from private rooms, a bed in a dormitory, to just a place on the floor for you and you mattress. I spend one week hiking through the Tatras, mostly sleeping open air with mattress and sleeping bag, sometimes with tent, and the last night at the Chata pod Rysmi, before attacking Rysy next day, and enjoying a amazing view from the top of Poland.

Overall I spend about 4 month in that small village and had the opportunities to work in a hostel and get to know to a lot of people from all around the world, to make friends, learn to cook some tasty dishes and even learn a few words of Slovak. It was an amazing time I had over there, and it's no question that I will be back one day to see my beloved left behind!

But it was time to move on and see some other places. The idea is to escape the cold of the winter and move south. For now I made it down to Pécs, a small town in south-west of Hungary. I was surprised to find it full of Germans coming here to study medicine. Since in Germany it is almost impossible to study medicine due to the insanely high Numerus clausus, highly motivated people come to Pécs for the first 4 semesters of their studies, and then often go back to Germany to finish.

I will spend some time here to do some maintenance on my equipment and finally do some more on this blog before heading down to the next place. So, more on this channel soon - Stay tuned...

Your comment