Computing | Ady's Blog

Adding Storage To Lenovo Laptop

Before purchasing anything, I would usually do a lot of research about reliability, pricing, and support. Another factor is upgradability.

I wanted to buy an IdeaPad laptop directly from Lenovo, but it was not as customizable as I would like it to be.

The laptop has multiple configuration options, there are models with an SSD and HDD, and some models with only SSD. I wanted to buy one with only a single 512 GB M.2 2242 SSD and upgrade in the future. When I contacted Lenovo sales via chat, they told me that it is impossible to add a new drive, that it would void the warranty.

I decided to buy one with a single M.2 SSD anyway.

The interesting this is that a disk caddy (including cables) is included indicating that we are allowed to add a disk on our own.

I haven’t tried to open the bottom panel yet, but I guess it’s normal these days to be able to add a disk into the disk bay.

There you go, I should be able to add a drive when needed 👍🏼

Restricting xmlrpc.php

Earlier today, I saw some spikes on the load graph for the new server (where this site is hosted).

Upon checking the logs I saw a lot of these:

134.122.53.221 - - [01/May/2020:12:21:54 +0000] "POST //xmlrpc.php HTTP/1.1" 200 264 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
134.122.53.221 - - [01/May/2020:12:21:55 +0000] "POST //xmlrpc.php HTTP/1.1" 200 264 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"

198.98.183.150 - - [01/May/2020:13:44:24 +0000] "POST //xmlrpc.php HTTP/1.1" 200 265 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"
198.98.183.150 - - [01/May/2020:13:44:25 +0000] "POST //xmlrpc.php HTTP/1.1" 200 265 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"

I’m not mentioning the source IP owner, technical readers can look it up if they are interested. However, the second IP comes from a IP range that is rather interesting.

Searching the Internet, I found out that many people consider the xmlrpc.php as a problem. For those who are not familiar with WordPress, this file is responsible for external communications. For example when using the mobile application to manage your site, and also when you use Jetpack.

There are plugins to disable XML-RPC such as this one, but I use the app from time to time, so I would like to keep xmlrpc.php working.

The official Jetpack website provides this list for whitelisting purposes.

I have been restricting access to my /wp-admin URL for ages, using Nginx. I think it is a good idea to do the same for xmlrpc.php.

location ~ ^/(xmlrpc\.php$) {
    include conf.d/includes/jetpack-ipvs-v4.conf;
    deny all;
 
    include fastcgi.conf;
    fastcgi_intercept_errors on;
    fastcgi_pass php;
}

The simple script to update this IP list into Nginx configuration, that is consumed by the configuration above:

#!/bin/bash
 
FILENAME=jetpack-ipvs-v4.conf
CONF_FILE=/etc/nginx/conf.d/includes/${FILENAME}
 
wget -q -O /tmp/ips-v4.txt https://jetpack.com/ips-v4.txt
 
if [ -s /tmp/ips-v4.txt ]; then
  cat /tmp/ips-v4.txt | awk {'print "allow "$1";"'} > /tmp/${FILENAME}
 
  [ -s ${CONF_FILE} ] || touch ${CONF_FILE}
 
  if [ "$(diff /tmp/${FILENAME} ${CONF_FILE})" != "" ]; then
    echo "Files different, replacing ${CONF_FILE} and reloading nginx"
    mv -fv /tmp/${FILENAME} ${CONF_FILE}
    systemctl reload nginx
  else
    echo "File /tmp/${FILENAME} match ${CONF_FILE}, not doing anything"
  fi
fi
 
rm -f /tmp/ips-v4.txt

It can be periodically executed by cron so that when the IP list changes, the configuration gets updated.

Now, if any IP other than Jetpack tries to access /xmlrpc.php it will receive Error 403 Forbidden.

Have fun!

Flask + GitLab OAuth

I’m back. A lot of things have changed since I last wrote and one of that is my go-to language.

Earlier today, I needed to write a simple Flask application using GitLab as the OAuth2 provider.

I immediately turned to Flask-OAuth to do the job, but it keeps on failing with:

SSLHandshakeError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

It seems to be a problem with httplib2.

After struggling for quite some time, I found Flask-OAuthlib that claims to be a replacement for the outdated Flask-Oauth. It worked like a charm.

GitLab’s documentation on consuming its OAuth2 is quite basic. Below is a basic implementation that works.

All you need to do is change the gitlab.example.com to your GitLab server, and add the consumer_key and consumer_secret. If successful, the main page will display a JSON with the logged on user’s details.

from flask import Flask, render_template, redirect, url_for, session, request, jsonify
from flask_oauthlib.client import OAuth
 
app = Flask(__name__)
app.debug = True
app.secret_key = 'development'
oauth = OAuth(app)
 
gitlab = oauth.remote_app('gitlab',
    base_url='https://gitlab.example.com/api/v3/',
    request_token_url=None,
    access_token_url='https://gitlab.example.com/oauth/token',
    authorize_url='https://gitlab.example.com/oauth/authorize',
    access_token_method='POST',
    consumer_key='',
    consumer_secret=''
)
 
@app.route('/')
def index():
    if 'gitlab_token' in session:
        me = gitlab.get('user')
        return jsonify(me.data)
    return redirect(url_for('login'))
 
 
@app.route('/login')
def login():
    return gitlab.authorize(callback=url_for('authorized', _external=True, _scheme='https'))
 
 
@app.route('/logout')
def logout():
    del session['gitlab_token']
    return redirect(url_for('index'))
 
@app.route('/login/authorized')
def authorized():
    resp = gitlab.authorized_response()
    if resp is None:
        return 'Access denied: reason=%s error=%s' % (
            request.args['error'],
            request.args['error_description']
        )
    session['gitlab_token'] = (resp['access_token'], '')
    return redirect(url_for('index'))
 
@gitlab.tokengetter
def get_gitlab_oauth_token():
    return session.get('gitlab_token')
 
if __name__ == "__main__":
    app.run()

I hope it saves someone some time.

CrashPlan 3.5.3 Headless Upgrade

A headless installation of CrashPlan will fail when it tries to update itself.

This short post assumes that you already have it setup and successfully running before, and is targeted only to help you save some time by identifying important files to copy.

Running the installer again will also work, but we actually spend more time to fix the scripts and the identity file might get overwritten causing more time to figure out what happened.

So here goes. This is how we extract the tar archive and the cpio archive within it.

# CrashPlan_3.5.3_Linux.tgz
# cd CrashPlan-install
# cat CrashPlan_3.5.3.cpi | gzip -dc - | cpio -i --no-preserve-owner

Changed files for 3.4.1 to 3.5.3 (thanks to rsync) are:

lang/txt.properties
lang/txt_sv.properties
lang/txt_th.properties
lang/txt_tr.properties
lang/txt_zh.properties
lib/com.backup42.desktop.jar
lib/com.jniwrapper.jniwrap.jar
lib/com.jniwrapper.winpack.jar

All I did was replace those files, and my CrashPlan installation is working fine.

If you actually arrive here to find information on installing for the first time, this post can help you if you’re using a Dlink DNS-32X series. Follow it from start to end (with some adaptation to the paths) and you’ll be fine.

However, you might have to change paths and also do extra steps to get it working. At one point, CrashPlan will run fine but you’ll see that it’s not uploading files.

This post can help you troubleshoot the Java issues by replacing libraries.

From the top of my head I remember having to insert a new library with the correct architecture inside jna-3.2.5.jar, replace libmd5.so, and replace libjtux.so. I also had to link /ffp/usr/local/crashplan/libffi.so.5 to a location accessible by the system loader.

Good luck!

Inbox Zero: Gmail vs. Mailbox

Mailbox is really cool for someone who has a lot of emails like me.

inbox-zero-600

It was worth the wait although I didn’t have to wait long (registered in the queue weeks ago). Have you been activated?

Build LFTP on Mac OS X Lion

If you’re a seasoned Linux SysAdmin, you’ll miss LFTP. It’s a really powerful FTP client. Yes, you can also install it using MacPorts or Fink but right now, this is much quicker for me.

Here’s how I built LFTP 4.3.6 on my MacBook Pro. For the record, I’m on 10.7.3

Prerequisite: Apple developer tools (Xcode)

1. Download The GNU Readline Library (The library that came with OS X will not work). This is how I built Readline 6.2:

$ cd /where/readline/was/extracted
$ ./configure --prefix="/usr/local" --disable-shared && make && sudo make install

This will build a static library and install in it /usr/local

2. Make sure the library (libreadline.a) was built successfully in /usr/local/lib

3. Download LFTP and build it:

$ cd /where/lftp/was/extracted
$ CXXFLAGS="-O0 -Wall -fno-exceptions -fno-rtti -fno-implement-inlines" \
LDFLAGS="-Xlinker -search_paths_first -L/usr/local/lib" \
CPPFLAGS="-I/usr/local/include" \
./configure --with-openssl --disable-shared --disable-nls
$ make && sudo make install

In the lftp configure line, the “-Xlinker -search_paths_first” is necessary so that the linker will not prefer the system shared /usr/lib/libreadline.dylib to the static libreadline.a we just compiled.

The final binary will be installed into your /usr/local/bin/lftp and if your build was successful you should be able to run “lftp” and get the prompt:

lftp :~>

My LFTP version:

LFTP | Version 4.3.6 | Copyright (c) 1996-2012 Alexander V. Lukyanov
 
LFTP is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
 
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.
 
You should have received a copy of the GNU General Public License
along with LFTP.  If not, see <http: //www.gnu.org/licenses/>.
 
Send bug reports and questions to the mailing list lftp at uniyar.ac.ru.
 
Libraries used: Readline 6.2, Expat 2.0.1, OpenSSL 0.9.8r 8 Feb 2011, libiconv 1.11, zlib 1.2.5

Good luck!

Perhaps some of you arrived by searching for “error: â€˜rl_kill_full_lineâ€™ was not declared in this scope”. Yep, the above steps should help you.

VirtueMart Custom Login Module in Joomla!

Here’s a short article on making a simple module to include on your Joomla! pages that displays login / logout links. I’m a Joomla! newbie so there might be better ways to accomplish this.

I was helping a friend-client to accomplish redirection to the same page after logout.

Here’s the basic code that you need to have in a module with the Jumi extension:

< ?php
    $user=& JFactory::getUser();
    if (!$user->guest)
        echo '<a href="index.php?option=com_user&task=logout&return=Lw">Logout</a>';
    else
        echo '<a href="index.php?page=account.index&option=com_virtuemart">Login</a>';
?>

The code above will redirect users to the root or uppermost level of the website.

Let’s say that your website is http://www.yourwebsite.com/ and your shopping page with VirtueMart is installed at a subfolder http://www.yourwebsite.com/shop/

The question is simply where you want your user to end up after logging out. If you need your users to end up at http://www.yourwebsite.com/ then you’re good to go. If you want your users to be redirected to the shop or a thank you page, here’s where you need to be a little creative.

You need to replace that “Lw” in the logout link to a different string. “Lw” is the base 64 representation of the character “/”. So this means that the user will be redirected to / which is http://www.yourwebsite.com/

A solution I came out with:

< ?php
    $redirect_to = '/shop/';
    $user=& JFactory::getUser();
    if (!$user->guest)
        echo '<a href="index.php?option=com_user&task=logout&return=' . base64_encode($redirect_to) . '">Logout</a>';
    else
        echo '<a href="index.php?page=account.index&option=com_virtuemart">Login</a>';
?>

So only line 2 needs to be changed. Let’s say you want users to be redirected to http://www.yourwebsite.com/thankyou.html here’s how you will change line 2:

2	$redirect_to = '/thankyou.html';

If you’d like your users to simply be redirected to the same page where they clicked the logout link, here’s what you should do to line 2:

2	$redirect_to = $_SERVER['REQUEST_URI'];

That’s simply it. At first, I totally forgot that I can use PHP’s base64_encode so I ended up confusing my friend with an online encoder so that he can replace the “Lw”.

One annoying thing that I wasn’t able to solve is the login page always displays the error message:

Error: You do not have permission to access the requested module.

I think Joomla! is trying to load VirtueMart too early and I can’t make it go away. Looking at the Internet a lot of other websites has this message displayed. If you know how this message can be removed without hacking the CSS or source code, please let me know and I’ll give you credit.

References:

Leverage Browser Caching

In the previous post I wrote about enabling compression for your pages so that they would load faster to the visitor. Today I’m going to write about how you can make use of browser caching to save some bandwidth.

Some people told me that their ISP or hosting provider requested that they upgrade the hosting plan or subscribe for more bandwidth. Since this site doesn’t have that much traffic, I wouldn’t know.

However recently I was able to help on a website which has a lot of visitors compared to this site. Around 14-18 visitors per minute on a working day and the bandwidth usage was very high, more than a gigabyte per day.

For the website, I saw that there were many requests for images (photos). The images aren’t that big anyway, around 100KB each but the amount of request made it significant.

Armed with knowledge of mod_expires, I added the following clauses to .htaccess while hoping that the server has the module installed. The following configuration is minimal, and Google Pagespeed actually suggests for 1 week.

<ifmodule mod_expires.c>
        ExpiresActive On
        ExpiresByType image/gif "access plus 2 hours"
        ExpiresByType image/png "access plus 2 hours"
        ExpiresByType image/jpg "access plus 2 hours"
        ExpiresByType image/jpeg "access plus 2 hours"
        ExpiresByType text/css "access plus 2 hours"
        ExpiresByType application/javascript "access plus 2 hours"
        ExpiresByType application/x-javascript "access plus 2 hours"
</ifmodule>

Although I know why Google Analytics set its expiry to 2 hours, it’s kind of amusing since the suggestion comes from another Google product. Oh well I am allowed to be amused right?

So let’s get to the results. Here are the bandwidth graphs from both days. I enabled mod_expires at around 6PM on 5 January 2012.

We can’t really see the difference by looking at the graphs. Google Analytics shows that there are at least 200 more visits on 6 January 2012. The numbers? Here you go:

At least 400MB were saved by this technique. You can actually put specific settings for each folder in your website. For example 2 hours is nice for cosmetic images which may need to be changed frequently but not for photos. For example if you run a photography website, you can even make your photos to expire in 1 year!

What mod_expires does is actually telling the browser that the resource (images) will expire on a specific date. It’s flexible enough to set the date from the access time. Here is the link to the official manual page for mod_expires.

Please be careful to note that this is not a quick solution for the lazy. You must think hard enough to set the proper amount of time before the images expire otherwise normal users will not see your changes or updates to the image until the cache on their browsers expire!

Good luck!

Let Apache Compress Your Website

Website speed is one of the most important factor to make people like to visit more. In 2008, I wrote Compressing WordPress Output and this is done by adding one line to index.php

The problem with that approach is that when you upgrade WordPress you have to manually add the line into index.php, and the technique does not improve loading time for administration pages.

Compressing is done on the fly by Apache, and it helps improve loading time because your browser receives smaller files. While some browsers were broken (they do not know how to handle compressed content), today’s browsers are much more efficient.

If your webhost allows you to add your own .htaccess then you can use this technique. Please note that you can do this for any type of website, not limited to WordPress.

<ifmodule mod_deflate.c>
     AddOutputFilterByType DEFLATE text/text text/html text/plain text/xml text/css application/x-javascript application/javascript
</ifmodule>

As simple as that. The directive above will ask Apache to compress text, html, xml, css, and javascript files before sending them to the browser. We don’t compress images because we don’t save much on image compression plus it will be too stressful for Apache.

You can use Firebug or Chrome to actually verify whether your content is now zipped. You might have to clear your browser cache first otherwise the files will not be requested from the server. You should look for the content-encoding header and it should say “gzip”. To satisfy yourself, you might want to capture the size of the page (there should be a column for that) before and after you add the directives into .htaccess and you should be able to see a significant reduction in size.

If you don’t see gzip then the possibilities are either your webhost does not allow .htaccess or mod_deflate is not installed on the server. To see whether or not mod_deflate is installed on the server you can remove the 1st and 3rd line in .htaccess and refresh the page. If it’s not installed then you will see an error.

For Joomla! users, there’s an easier way to do it in the Global Configuration section:

Simple eh?

For WordPress, one might argue that they can change the parameter gzipcompression in the semi hidden configuration page http://www.yourblog.com/wp-admin/options.php but I tried it in WordPress 3.2.1 and it did not work. I read in WordPress forums, they removed this feature because Apache can handle it much better.

Go ahead and try it out. Your visitors will be thankful.

Sparrow and Shortmail

I’ve been pushing the limits of my mid-2010 13″ MacBook Pro lately, running all sort of applications and leaving them running. There has been occasions where the machine will freeze while it struggles to schedule the CPU time to all of the applications.

I use a lot of email, and by a lot I don’t mean 10 emails per day. No joke. The features of Apple Mail 5.0 are actually enough for my daily use but it does feel a little bit sluggish with 5 IMAP accounts.

Recently, I signed up to Shortmail via my Twitter account. The idea is nice. 500 characters max per email, straight to the point.

While casually browsing through my Twitter stream last night I saw people talking about setting up Sparrow to access Shortmail and sending attachments via Dropbox.

I was thinking, Shortmail is short, Sparrow is light. Combining them together should be nice… and since all of my email accounts are now running via IMAP, why not give Sparrow a try with Shortmail before using it for serious communications.

I downloaded Sparrow through the website to try it out, instead of buying it from the Mac App Store for $9.99

I was pleased that it only asked for my email address and password so I entered my Shortmail account details. It was nice. Shortmail’s logo was displayed on the screen. But it took ten minutes to sync my 3 messages. I don’t know what it was doing and I’m not sure whether the waiting time is caused by Shortmail servers or Sparrow itself.

One thing I hate the most is how markup is displayed in the preview lines in Sparrow. It’s not in Facebook’s email but markup are there for the first 2 emails.

I can understand if there are glitches in Shortmail as it’s still fresh but I don’t expect glitches from Sparrow. I wonder how long it would take for it to sync my gigs of emails from other email accounts. I can’t even imagine and I can’t afford to waste so much time.

Looks likes it’s going to be a while before I am convinced to try Sparrow again. I am glad I didn’t rush to buy the app for $9.99 in the Mac App Store. For now, I’m leaving it to AppZapper to cleanup!

Update 16 December 2011 11:15 AM GMT+8

Less than an hour ago, I received a shortmail from Shortmail telling me that they are working on a new version of their IMAP service that should make a difference. This is great news for those who prefer using Shortmail service via regular email clients. From @Shortmail‘s timeline I can see that they are responding to many users about improving the IMAP service.

Keep up the good work! By the way, my Shortmail address is romantikaname [at] shortmail [dot] com

Thanks for the shortmail! This is nice, and it shows that you are serious about what you do!

For the record, I was more disappointed with Sparrow displaying ugly markup on the preview pane.

Shrinking VirtualBox Disk Images

On my MacBook Pro with minimal disk size (250GB) I had to install VirtualBox in order to run any kind of Windows, for emergency cases. I was able to install Windows Server 2008 in a 20GB dynamically expanding virtual disk image (VDI).

A couple of months back I also made a clone of my HP NX9010 hard disk, a Seagate Momentus 80GB drive using the dd utility and attach it to the same virtual machine. I don’t have a lot of time at hand so this will enable me to selectively back up the files on that drive, when I have some free time. I use that old laptop to play videos for my child.

After some time, the Windows Server 2008 VDI grew up to 11GB and the Momentus VDI stayed at 74GB because dd was copying it bit by bit. I needed to free up more space. My target was the Momentus VDI because it’s huge and I have already deleted (backed up) a lot of files in it.

As their name suggest, they are dynamically expanding so they will not shrink themselves. In VMware, you can easily stumble upon the “shrink” button in the tool. In VirtualBox, we need to use the command line utility VBoxManage.

It’s as simple as:

VBoxManage modifyhd Seagate\ Momentus\ 80GB.vdi --compact

*VBoxManage is run on the host machine, while the virtual machine is powered off (not suspended).

Running it without any preparation work will shrink the VDI a bit, too insignificant to even notice.

And so I used SDelete to zero all the unused space on the VDI, and ran VBoxManage again. SDelete needs to be run inside the virtual machine while it’s running.

sdelete -c d:

I managed to get the 74GB VDI to shrink to 23GB. Now that’s a significant reduction in size.

Still, I was greedy. I wanted to know whether fragmentation has any effect on the shrink-ability of the VDI. So I went into the virtual machine, ran defrag, ran sdelete, then I executed VBoxManage to compact the VDI.

Below was my final result. From 74GB to 5.1GB I was fully satisfied. I was also able to shrink the 11GB VDI that contains the OS to 9.7GB.

If you have questions/suggestions/feedback, please leave a comment.

Good Bye MyBlogLog

I have just received an email from Yahoo! announcing the end of MyBlogLog service. It’s going to be discontinued effective May 24, 2011. Nice of Yahoo! to give us some time before terminating it for good.

However the link for help pages redirected me to this page: http://help.yahoo.com/l/us/yahoo/ysm/ll/featured/index.html, which I think is a mistake because it’s for “Local Featured Listings”.

Nevertheless, we thank you Yahoo! for a great service.

Google Authenticator Rocks

OK, “Google Authenticator” is actually the name of the smartphone apps provided by Google. What is it all about? It’s two factor authentication similar to the ones banks use to authenticate users to log in or to perform transactions. Corporate users might also have the same security method using smart card or RSA SecurID to log in to machines or connect to VPN.

The basic idea is simple, you will need your password and the code provided by the apps so that if your password is compromised the malicious user will not be able to log into your account. Plus, you’ll always have your phone with you so you will have this code when you need it.

Here’s how the app looks like on the iPhone:

The code changes in a few seconds (there’s a timer pie on the left that moves to show how long will the code will stay).

This 6 digit code, will be asked when you log in to Google services from the web browser. Don’t worry it will not be every time if you select it to remember trusted machines. It’ll be remembered for 30 days and after that I think it will be asked again (I have just started using it today so I am just guessing the behavior).

Before you Begin

If you use Google services in various places such as in mobile applications (e.g. Google Mobile App), desktop applications (e.g. Google Talk), and other places where two factor verifications aren’t coded yet, you will need some time to set up application specific passwords for each of them so that they will still work. More on this below.

How To Start

It’s easy to activate. One will need to navigate to https://www.google.com/accounts/SmSAuthconfig and follow the on screen instruction.

Official step by step instruction from Google is available here.

It’s a very simple and quick process. Once activated, all other applications not using two factor authentication will automatically be denied access (login fails) and they need to use the application specific password discussed below.

What if I lose my smart phone?

Google thought of everything. Upon signing up, you will be receiving 10 emergency codes to keep safe. These codes need to be kept securely, but accessible, for example in your wallet. Each of them can only be used one time.

You’ll also have the option use a backup phone number for an emergency code to be sent to you via SMS or an automated voice service. Cool eh?

Application Specific Passwords

While setting up two factor authentication takes maybe 2 or 3 minutes, entering application specific passwords may take some time. They each are 12 characters in length. They are used for applications such as Thunderbird, Outlook, mobile mail, GTalk, IM+ mobile app, etc. and the list is endless.

But it’s not that painful, compared to the security it provides (says a paranoid man).

For pictures from other devices (BlackBerry and Android) please head here: TechCrunch: Google Rolls Out Two-Factor Authentication For Everyone. You Should Use It.

So what about you? Are you jumping on the bandwagon? You should. Tell me your experience below.

Mac OS X: Modifying Stubborn Files Transferred From Windows

This is related to the previous post, where I had a lot of checked out Subversion working copies in my disk. They came from a Windows machine I used before I bought myself a Mac.

For your information, in a Subversion working copy, there will always be a hidden folder named .svn in each directory, and a bunch of files in them. Many files in there are read-only on Windows, and on UNIX or Mac systems they have read permissions set (-r–r–r–). They are supposed to be that way and only a Subversion client can work on them.

When attempting to update my working directory, my Subversion client refused to update the files because it can’t write to .svn/entries

Tested on: Snow Leopard 10.6.6
Known to work as early as: Tiger 10.4.x

The issue is not related to permissions or even Subversion. For some kind of unknown reasons, when read-only files are transferred from Windows, there is a special flag set on Mac OS X – the user immutable flag. When this happens, you will be receiving these:

$ chmod u+w entries 
chmod: Unable to change file mode on entries: Operation not permitted

As root:

$ sudo chmod u+w entries
Password:
chmod: Unable to change file mode on entries: Operation not permitted

However, root can delete the files. But I don’t want the files to be deleted. I merely wanted to update (and later add new and modified files to the repository) my working copy.

The simple solution to this is to use the chflags command to unset the user immutable flag:

$ chflags -R nouchg *

As simple as that, and you will be able to operate on the files again. In my case, my “svn update” operation went as smooth as silk.

Remember, the permissions of the file does not even matter. Even if it’s world-writable -rwxrwxrwx

For the record, I transferred the file via a USB drive (NTFS file system), and I have received reports that this is the same case if you transfer from a file share (Samba or Windows File Sharing).

I hope this saved you some time.

Starting Subversion Service on Demand in Mac OS X

Since I moved to a new home I didn’t let my Linux servers run 24/7 so that I can save on electricity bills. It’s not so much about moving homes but it’s about the tariff increase. My 500W and 400W power supplies can easily reach the max usage of around 300W each which translates to around 432kWh per month, costing me around RM123.55 (source: TNB Tariff). Money that can be better spent for my gadgets, craft items for my wife, or toys for my kid.

The last time my Subversion repositories were used, was in Feb 2009. Since the repositories are just files, I can easily transfer them to my Mac to use locally. For me, file history and ability to revert changes are more important that using the repository as backup. My Mac is backed up using Time Machine anyway so I am pretty much safe. I am obsessed with file history and proper branching in source code, sometimes I think I have a mild version of OCD. Seriously.

OK, now on to the technical part. I am unsure on how my machine is pre-installed with CollabNet’s version, but you can install it on your Mac using one of the available packages listed in Apache’s official Subversion packages list.

Tested on: Snow Leopard 10.6.6
Known to work as early as: Tiger 10.4.x

My version:

svn, version 1.6.5 (r38866)
   compiled Jun 24 2010, 17:16:45
 
Copyright (C) 2000-2009 CollabNet.
Subversion is open source software, see http://subversion.tigris.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).

Please take note that the method I am going to show here is to allow access to Subversion as a service via the network. I like to prepare for future use for example allowing access from other machines. You can always opt for local file based access, http, https, and svn+ssh (and you will not need to do the steps below).

On Mac, you can have services launched as a permanent process or on demand using launchd – System wide and per-user daemon/agent manager.

In this discussion we will be running svnserve on demand, similar to running services via inetd on Linux.

It is fairly straightforward. You need to create a .plist file similar to the one below. I named mine org.apache.subversion.svnserv.plist

< ?xml version="1.0" encoding="UTF-8"?>
< !DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" 
"http://www.apple.com/DTDs/PropertyList-1.0.dtd"> 
<plist version="1.0">
<dict>
    <key>Debug</key>
    <false />
    <key>Disabled</key>
    <false />
    <key>GroupName</key>
    <string>staff</string>
    <key>Label</key>
    <string>org.apache.subversion.svnserv</string>
    <key>OnDemand</key>
    <true />
    <key>Program</key>
    <string>/usr/bin/svnserve</string>
    <key>ProgramArguments</key>
    <array>
        <string>svnserve</string>
        <string>--inetd</string>
        <string>--root=/Users/adyromantika/SVNRepository</string>
    </array>
    <key>ServiceDescription</key>
    <string>SVN Code Version Management</string>
    <key>Sockets</key>
    </dict><dict>
        <key>Listeners</key>
        </dict><dict>
            <key>SockFamily</key>
            <string>IPv4</string>
            <key>SockServiceName</key>
            <string>svn</string>
            <key>SockType</key>
            <string>stream</string>
        </dict>
    <key>Umask</key>
    <integer>2</integer>
    <key>UserName</key>
    <string>adyromantika</string>
    <key>inetdCompatibility</key>
    <dict>
        <key>Wait</key>
        <false />
    </dict>
</plist>

Things that need to be changed:

Line 17: /usr/bin/svnserve needs to be changed to reflect your local installation.
Line 22 needs to be changed to use your own repository root. This is not the name of the folder with “conf”, “db”, etc. folders but one folder up, so that your svnserve can serve multiple repositories.
Line 11 is where you need to put the group name you want the svnserve process to run as. As you can see, I am lazy so I used the default group “staff”.
Line 41 is where you need to put the username of you want the svnserve process to run as. I used my own user id which is not the best security practice but as I mentioned earlier, I am lazy.

Now that it’s done, copy the file to /Library/LaunchDaemons/ and run the command:

sudo launchctl load /Library/LaunchDaemons/org.apache.subversion.svnserv.plist

You are all set. Please note that since it’s launched on demand, you will not see the svnserve process running unless you are connected to the repository. You can simply use telnet to verify you get some kind of response:

adymac:~ adyromantika$ telnet localhost 3690
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
( success ( 2 2 ( ) ( edit-pipeline svndiff1 absent-entries commit-revprops depth log-revprops partial-replay ) ) )

If you don’t get a response or “Unable to connect to remote host” you may want to check the file /etc/services and see whether these two lines are commented:

svn     3690/udp    # Subversion
svn     3690/tcp    # Subversion

Good luck. If you are having issues please comment below and I will try my best to help.