LOPSA-NJ News Aggregator
Unix Rosetta Stone
Judging from the number of Delicious bookmarks it has, it's pretty well know, but I figured that I couldn't be the only person in the dark, and I figured someone might get some info from it.
Wacky SSH Authorized Keys Tricks
What you might not have caught was in the comments, where Ben Cotton mentioned a trick I hadn't heard of, namely specifying the allowed remote commands in the authorized_keys line. He said there were even more features available, just waiting on the manpage. I replied that if he wrote it, I'd link to it.
Well, Ben put his money where his mouth is. He goes into nice detail and provides some good links and suggestions. This is really fascinating stuff, and I'm looking forward to using it in my own organization.
Therek over at Unix Sysadmin jumped in the fray, too. He's got three neat tricks for your ssh needs that you should really check out. I had no idea SSH key auth could be bent in these directions!
I've said it before, but I'll keep saying it. I love having visitors to my blog who enjoy what I write, and it really brings it home to interact with everyone like this. I couldn't ask for a better bunch of readers, though to be honest, I'm worried about Ben's longevity. I can't imagine what his cholesterol level must be ;-)
Ben, Therek, thank you both very much! I know my readers will really enjoy these articles. And as for everyone else, the same offer goes for you. If you've got something to share, let me know, I'll be happy to link to your blog entry or host it here if you've got the urge to write.
On Remote Workers and Working Remotely
I’ve been on both sides of the remote worker relationship. On the manager side, I’ve managed some good-sized projects using an all-remote work force. Indeed, I’ve hired, managed, fired, and promoted workers without ever knowing what they look like. On the worker side, I do most of my work remotely, and I have for some time now. Judging by the amount of repeat business I get, I’d say that I’m more than acceptably productive working remotely.
In dealing with various clients, recruiters, prospective employers, business owners, and talking to friends who manage people for a living, I’ve heard pretty much every excuse/reason there is for not wanting to deal with a remote work force. I’ve heard and experienced successes with remote workers as well, and they all have a few key things in common, which are missing from the stories of failure. I’ll talk about them in a minute.
I first want to just say that I’m not some kind of fanboy who thinks remote workers are the answer to every problem. There are valid reasons for not having remote workers. For example, it’d be hard to build cars with a remote work force. Some things (some!) just require a physical presence. Whoever maintains the printers at your company really has to be around to change out ink cartridges and stuff like that.
There are certain classes of jobs, though, that are well-suited to working remotely. There are even classes of jobs that are necessarily performed remotely to some degree (field sales and support technicians for example), that could be made 100% remote with the proper tools and processes in place.
So what makes a remote worker success story different from a story of failure?
Always be prepared…The number one difference I’ve seen between success and failure in managing a remote work force is that successful managers spent the time to prepare the managers, the team, the department, the organization, and the remote workers themselves to work remotely.
If you don’t prepare for a remote work force, you will fail miserably. As a result, I’m a big advocate of treating “Let’s go remote!” as an internal project with goals and milestones just like any other project. Preparing an organization to manage a remote work force takes a good deal of forethought, with a focus on communication and collaboration tools, reporting, accountability, scheduling, etc. In addition, you have to prepare the remote workers themselves, to insure they know what’s expected of them in terms of reporting their status, scheduling, communication, etc. They also need to know *about*, and *how to use* the tools they’ll be expected to use from home.
You have to plan this. You have to prepare, or you’re going to be like the HR manager who told me their company no longer allows for remote workers because “we tried it once and the guy made a complete mess of things”. When I asked the HR manager why he attributed that to the geographic location of the worker, he said “good point, he could just as well have made a mess here in the office”. You need good workers no matter where they’re going to work. The workers need expectations and goals from the manager, and the manager needs feedback and communication (and results!) from the worker. Tools help to facilitate these things. This is already a long post, so I’ll probably make a tools list in another post.
Communicate, and set expectationsBefore the tools come other higher-level decisions and communication. For example, one problem I’ve heard more than once about remote workers is “we can’t hire a remote worker full-time, because then everyone will want to work from home”. As if they didn’t already all want to work from home! Everyone would love to have the option! Even if they didn’t take advantage of it, they’d consider it a really cool perk! They’d tell all of their friends about it, because it would make them jealous, and guess who their friends will contact first when they start to look for other opportunities?
You have to start somewhere, and you can’t just swing the barn doors open and let everyone go their own way on day 1. If you have an existing corporate structure in place with assets and services and regular meetings and the like, then you have to decide who can make the most benefit from a remote situation the soonest, make them the pilot group, and manage the expectations of the rest of the organization while the pilot group prepares to move to a remote workspace.
1, 10, 100, 1000A common software application rollout strategy is to make it accessible to 1 user, then 10, then 100, then 1000, then… move up from there. In preparing your organization or department, you might consider a similar strategy.
I work for a client right now where I’m the “1″. If I can work effectively with the rest of the team (in the office), if I can produce results, remain accessible as-needed during working hours, manage the expectations of my team with regards to my presence (appointments happen), and overall be an asset to the team, then the management may decide that it can work on some larger scale - even if ‘larger’ means 2 instead of 1. It might also be useful to do a ‘remote rotation’ so that glitches can be caught early before making a physical presence in the office optional.
Success, of course, means getting together with the team and figuring out what tools will be used to best emulate an office working environment. We use IRC for 99% of our communication, falling back to email when we need to cc managers, we have a wiki for documentation and status updates, we have a trouble ticket system, everyone has everyone else’s phone number, blackberry PIN, or whatever. We’re a technical group doing system administration. It’s working wonderfully.
“But if the sysadmins work from home, the developers will want to work from home!” Maybe so. That’s where you have to manage expectations, and communicate with your workers to let them know that the company’s ‘office optional’ project is in an early alpha stage, that it’s being tested on the group most familiar with the technologies involved, and most capable of exploiting those technologies successfully to produce results. Once the geeks work out the shortcomings, and management is able to evaluate the effectiveness of the plan, the tests will become more widespread.
Really, it’s not a whole lot different from doing anything else that affects the whole company: changing payroll providers, healthcare options, software and desktop hardware upgrades and replacements… it just takes communication. The process has to be managed, just like every other process.
There’s more than one way to do it!There’s no one solution out there. When I joined php|architect Magazine in 2003, it was run by Marco Tabini, and I was a remote editor. A couple of months after joining, I became editor in chief, and was in charge of remotely managing the magazine. I did it differently from Marco, but he still remained involved and engaged through good communication.
Python Magazine was created and managed by me, and for the entire lifespan of the magazine, I have not seen anyone else involved in its production in person. Ever. Design, production, web site admin, executive administration, tech editors, authors, accountants… time lines, budgets and planning documents… all remote, and mostly delegated. I started the magazine with the thought that at some point someone more engaged in the community and with Python should take charge — I was just a “temp” to get the vision off the ground. Sure enough, when I handed the magazine over to Doug Hellmann, he did things differently from me, and it’s working out wonderfully for him as well!
Everyone has their own management style. Don’t think that just because your management style is a little unique you can’t handle remote workers. Good managers are creative, and aren’t afraid to execute on creative solutions.
addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F11%2F18%2Fon-remote-workers-and-working-remotely%2F'; addthis_title = 'On+Remote+Workers+and+Working+Remotely'; addthis_pub = 'jonesy';Great tool for network diagramming
After reading this book pretty much cover to cover, I decided that I needed to logically map out the various relationships of my services, to figure out the inheritance policies (Nagios supports multiple inheritance in configuration objects).
I started looking for a good free diagramming tool, first on Windows then on Linux. Windows was hopeless. I found lots that looked promising, but ended up being shareware. I don't have MS Office Pro on my personal laptop, so I didn't have Visio handy, and I wasn't going to buy a piece of software when I was sure that something good and free existed.
Giving up, I booted into Linux to see if anything I didn't know about was in synaptic. Of course not. The best diagramming solution in Linux is Dia, and I'm sorry to say it, but it's ugly. Really ugly. I'll use it if that's the only thing available and I'm just looking for something quick, but I won't like it.
I kept looking, and finally out of desperation I did a search for online applications, and I hit the jackpot. I found Gliffy. It's a flash diagramming application with built in stencils for all sorts of things, and the ability to add your own clipart. It'll even export to Visio.
I was impressed. It's free for personal use up to 5 public diagrams. You can pay $5/month for unlimited drawings and removing the ads, and there are corporate versions that have built in collaboration. It's easy to use, and it helped me a lot. Here's a drawing of some of my nagios groups:
If you're in the market for a cross-platform diagramming solution, you could do a lot worse than Gliffy.
Keating College
Some time ago I spoke to Craig Keating about his plans for a new secondary school in the center of Melbourne. His plan was to focus on the core academic areas and cater to academically gifted students. He had some interesting ideas for his business, one of which was to pay teachers rates that are typical for private schools (higher rates than government schools) but not have any sport programs in the evenings or weekends (private schools typically require teachers to work every Saturday and one evening every week in coaching a sport). This would therefore give an hourly pay rate that was significantly higher than most private schools offered and would thus allow recruiting some of the most skilled teachers.
One of his ideas was to intentionally keep the school small so that every teacher could know every student. One of the problems with most schools is that they take no feedback from the students. It seems that this serious deficiency would be largely addressed if the teachers knew the students and talked to them.
He pointed out that in the history of our school system (which largely derived from the UK system) the private schools had a lot of sporting activities as a way of filling time for boarding students, given that few schools accept boarders (and those that do have only a small portion of the students boarding) the sports are just a distraction from study. This is not to say that sports are inherently bad or should be avoided. He encouraged parents to take their children to sporting activities that suit the interests of the child and the beliefs of the parents instead of having the child be drafted into a school sport and the parents being forced to take an unwilling child to sporting activities that they detest (which I believe is a common school experience).
My own observation of school sport is that it is the epicentre of school bullying. There is an inherent risk of getting hurt when engaging in a sport. Some children get hurt every lesson, an intelligent person who ran a school with an intensive sports program might statistically analyse the injuries incurred and look for patterns. Children who are not good at sport are targeted for attack, for example when I was in year 7 (the first year of high school) one of my friends was assigned to the “cork bobbing” team in the swimming contest - this involved a contest to collect corks floating in the toddler pool for the students who were really bad at swimming. At that moment I knew that my friend would leave the school as the teachers had set him up for more intensive bullying than he could handle. Yet somehow the government still seems to believe that school sports are good!
This is not to say that physical activity is bad, the PE 4 Life program [1] (which is given a positive review in the movie Supersize Me [2]) seems useful. It has a focus on fitness for everyone rather than pointless competition for the most skilled.
I have just seen a sad announcement on the Keating College web site [3] that they will not be opening next year (and probably not opening at all). The Victorian Registration and Qualifications Authority (VCRA) announced in November that the application to be registered as a school (which was submitted in March) was rejected.
The first reason for the rejection was the lack of facilities for teaching woodwork and metalwork. As the VCRA apparently has no problems registering girls’ schools that don’t teach hard maths (a teacher at one such school told me that not enough girls wanted to study maths) it seems unreasonable to deny registration to a school that doesn’t teach some crafts subjects and caters to students who aren’t interested in those areas.
The second reason was the lack of facilities for sport and PE. Given the number of gyms in the city area it seems most likely that if specific objections were provided eight months earlier then something could have been easily arranged to cover the health and fitness issues. When I spoke to Craig he had specific plans for using the city baths, gyms, and parks for sporting activities, I expect that most parents who aren’t sports fanatics would find that his plans for PE were quite acceptable.
The third reason is the claim that 600 square meters of office space is only enough to teach one class of 24 students. That would mean that 25 square meters is needed for each student! I wonder if students are expected to bring their own binoculars to see the teacher or whether the school is expected to provide them. :-#
The government has a list of schools that work with the Australian Institute of Sport [4]. These schools provide additional flexibility in studies for athletes and probably some other benefits that aren’t mentioned in the brief web page. I don’t object to such special facilities being made available for the small number of students who might end up representing Australia in the Olympics at some future time. But I think that a greater benefit could be provided to a greater number of students if there were a number of schools opened to focus on the needs of students who are academically gifted. This doesn’t require that the government spend any money (they spend hundreds of millions of dollars on the AIS), merely that they not oppose schools that want to focus on teaching.
Currently the government is trying to force Internet censorship upon us with the claim that it will “protect children” [5]. It seems obvious to me that encouraging the establishment of schools such as Keating College will protect children from bullying (which is a very real threat and is the direct cause of some suicides). While so far no-one has shown any evidence that censoring the net will protect any child.
- [1] http://www.pe4life.org/
- [2] http://etbe.coker.com.au/2006/09/22/supersize-me/
- [3] http://www.keatingcollege.com.au/
- [4] https://secure.ausport.gov.au/asc_internet/ais/scholarships/ais_athlete_career_and_education/ais_ace_-_what_we_do/educational_guidance
- [5] http://etbe.coker.com.au/2008/11/18/other-reasons-for-not-censoring/
Reunion with Drupal, Break from Django
I started using drupal maybe 3-4 years ago. At the time I wasn’t all that impressed. I liked it better than Joomla (Mambo, at that time), and it was a little more featureful than PHP-Nuke. But even back then I hated that this thing was really making some sweeping, grand assumptions about what I would be using my Drupal site for. I used Drupal for LinuxLaboratory.org, and it was ok. I left Drupal once, to give MediaWiki a shot, but the truth is I didn’t want a wiki, so I went out and tested a bunch of other applications, and wound up back at Drupal. The 5.5 release was quite a bit better, and it got the job done.
About 2 weeks ago (maybe less?) I downloaded version 6.6. I poked. I prodded. I looked for new themes and found lots of them, and they were pretty cool. I looked for theme and module-building tutorials, and there were lots of them, and even entire books were published on each of the topics - even specifically for version 6 of Drupal. I looked for modules, and found a few useful ones who actually showed a trend of following the Drupal releases pretty closely. I also found that a couple of things I had used as modules in earlier releases were now built-in.
I fiddled on and off for a few days and was able to get a site together for my company’s web site that’s way, way better than the wordpress site that was there before. I’m also redoing the main LinuxLaboratory.org site using Drupal.
What about Django?I know that lots of you were encouraging me to keep moving ahead with Django. I *will* be moving ahead with Django at some point, but what I found is that doing example projects using the dev server and deploying a real application using Apache are such vastly different beasts that doing the former doesn’t really help make you qualified to perform the latter. When I had my site ready to go, and I had it working on my locally-installed dev server, I found myself completely lost when it came time to get it working on my webfaction account. It really shouldn’t be that hard, but it is. Or it was for me.
You can all take comfort in knowing that I still hate PHP and consider it a necessary evil. For the moment, though, I have a couple of projects involving PHP coming up. By the time those projects end, I hope I can be more skilled with Django, and with Django deployment. I’m not even going to mess with the dev server anymore. It’s just a damn tease. I’m going to sit down and spend some time with Django on Apache with mod_* and finally come up with answers to all the questions I had that nobody anywhere seemed to have any reasonable answers to. When I figure them out, I’ll post here and you can all flame me or learn something new, perhaps depending on your own skill level
In the mean time, while I don’t typically do book reviews, I’d recommend that anyone using Django 1.x stay away from the book “Practical Django Projects”. It’s specifically non-1.0, and you’ll be tripped up from the very first sample app, and it doesn’t get better from there. If you want to learn from the book (and there’s learning to be had from it), download 0.96.x, and use that to go through the book. When you’re done with the book, read the release notes for Django 1.0. You’ll have to make some alterations before moving your apps to 1.0, but overall you’ll be just fine.
addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F11%2F17%2Freunion-with-drupal-break-from-django%2F'; addthis_title = 'Reunion+with+Drupal%2C+Break+from+Django'; addthis_pub = 'jonesy';Other Reasons for not Censoring the Net
Currently there is a debate about censoring the Internet in Australia. Although debate might not be the correct word for a dispute where one party provides no facts and refuses to talk to any experts (Senator Conroy persistently refuses all requests to talk to anyone who knows anything about the technology or to have his office address any such questions). The failures of the technology are obvious to anyone who has worked with computers, here is an article in the Sydney Morning Herald about it [1] (one of many similar articles in the MSM). I don’t plan to mention the technological failures again because I believe that the only people who read my blog and don’t understand the technology are a small number of my relatives - I gave up on teaching my parents about IP protocols a long time ago.
One of the fundamental problems with the current censorship idea is that they don’t seem to have decided what they want to filter and who they want to filter it from. The actions taken to stop pedophiles from exchanging files are quite different from what would be taken to stop children accidentally accessing porn on the net. I get the impression that they just want censorship and will say whatever they think will impress people.
I have previously written about the safety issues related to mobile phones [2]. In that document I raised the issue of teenagers making their own porn (including videos of sexual assault). About four months after writing it a DVD movie was produced showing a gang of teenagers sexually assaulting a girl (they sold copies at their school). It seems that the incidence of teenagers making porn using mobile phones is only going to increase, while no-one has any plans to address the problem.
The blog www.somebodythinkofthechildren.com has some interesting information on this issue.
Two final reasons for opposing net censorship have been provided by the Sydney Anglicans [3]. They are:
- Given anti-vilification laws, could religious content be deemed “illegal” and be filtered out? Could Sydneyanglicans.net be blocked as “illegal” if it carries material deemed at some point now or in the future as vilifying other religions? If it’s illegal in Vic say, and there isn’t state-based filtering (there wont be), will the govt be inclined to ban it nation wide?
- Given anti-discrimination laws, if Sydneyanglicans.net runs an article with the orthodox line on homosexuality, will that be deemed illegal, and the site blocked? You can imagine it wouldn’t be too hard for someone to lobby Labor via the Greens, for instance.
So the Sydney Anglicans seem afraid that their religious rights to discriminate against others (seriously - religious organisations do have such rights) will be under threat if filtering is imposed.
I was a bit surprised when I saw this article, the Anglican church in Melbourne seems reasonably liberal and I had expected the Anglican church in the rest of Australia to be similar. But according to this article Peter Jensen (Sydney’s Anglican Archbishop) regards himself as one of the “true keepers of the authority of the Bible” [4]. It seems that the Anglican church is splitting over the issues related to the treatment of homosexuals and women (Peter believes that women should not be appointed to leadership positions in the church to avoid “disenfranchising” men who can’t accept them [5]).
It will be interesting to see the fundamentalist Christians who want to protect their current legal rights to vilify other religions and discriminate against people on the basis of gender and sexual preference fighting the other fundamentalist Christians who want to prevent anyone from seeing porn. But not as interesting as it will be if the Anglican church finally splits and then has a fight over who owns the cathedrals. ;)
A comment on my previous post about the national cost of slow net access suggests that Germany (where my blog is now hosted) has better protections for individual freedom than most countries [6]. If you want unrestricted net access then it is worth considering the options for running a VPN to another country (I have previously written a brief description of how to set up a basic OpenVPN link [7]).
- [1] http://www.smh.com.au/articles/2008/11/11/1226318639085.html?page=fullpage
- [2] http://doc.coker.com.au/safety/mobile-phone-safety/
- [3] http://www.sydneyanglicans.net/forums/viewthread/3673/
- [4] http://www.smh.com.au/articles/2008/06/22/1214073053814.html
- [5] http://www.abc.net.au/worldtoday/content/2004/s1213397.htm
- [6] http://etbe.coker.com.au/2008/11/17/national-cost-slow-net-access/#comment-16729
- [7] http://etbe.coker.com.au/2008/05/24/ipsec-is-pain/
Building and designing systems: Is the cart pulling the horse?
The short of it is that opposed to typical complex engine designs, where each individual part was tested independently and then together in subassemblies, and then again when the unit was complete, the space shuttle was pretty much designed, assembled, then tested. The better method has the advantage of weeding out all the really bad decisions in the small scale, then when you get to the point that you put them together, it generally works rather than flying apart at high speed.
While Code Monkeyism is primarily centered on software development, the points that Stephan make are readily applicable to us as infrastructure engineers, particularly in a growth phase where we're engineering new solutions and trying to implement them.
I'm as guilty of putting the cart in front of the horse as anyone. My debacle with the cluster was a prime example. When you're given a job to do, the equipment to do it with, and no time to learn, these kinds of things happen. Particularly when you're working with shoddy tools anyway.
I shouldn't have attempted to have the very first cluster I created be a production system, first. More due diligence in researching solutions was called for, and I probably would have learned beforehand that RHCS wasn't ready for prime time. I have learned from the experience, though, so all is not lost. Using the knowledge and experience I've gained, the next time will be more solid.
Is this something that everyone has to learn on the job, or was there a class or memo that I didn't get?
The National Cost of Slow Internet Access
Australia has slow Internet access when compared to other first-world countries. The costs of hosting servers are larger and the cost of residential access is greater with smaller limits. I read news reports with people in other countries complaining about having their home net connection restricted after they transfer 300G in one month, I have two net connections at the moment and the big (expensive) one allows me 25G of downloads per month. I use Internode, here are their current prices [1] (which are typical for Australia - they weren’t the cheapest last time I compared but they offer a good service and I am quite happy with them).
Most people in Australia don’t want to pay $70 per month for net access, I believe that the plans which have limits of 10G of download or less are considerably more popular.
Last time I investigated hosting servers in Australia I found that it would be totally impractical. The prices offered for limits such as 10G per month (for a server!) were comparable to prices offered by Linode [2] (and other ISPs in the US) for hundreds of gigs of transfer per month. I have recently configured a DomU at Linode for a client, Linode conveniently offers a choice of server rooms around the US so I chose a server room that was in the same region as my client’s other servers - giving 7 hops according to traceroute and a ping time as low as 2.5ms!
Currently I am hosting www.coker.com.au and my blog in Germany thanks to the generosity of a German friend. An amount of bandwidth that would be rather expensive for hosting in Australia is by German standards unused capacity in a standard hosting plan. So I get to host my blog in Germany with higher speeds than my previous Australian hosting (which was bottlenecked due to overuse of it’s capacity) and no bandwidth quotas that I am likely to hit in the near future. This also allows me to do new and bigger things, for example one of my future plans is to assemble a collection of Xen images of SE Linux installations - that will be a set of archives that are about 100MB in size. Even when using bittorrent transferring 100MB files from a server in Australia becomes unusable.
Most Australians who access my blog and have reasonably fast net connections (cable or ADSL2+) will notice a performance improvement. Australians who use modems might notice a performance drop due to longer latencies of connections to Germany (an increase of about 350ms in ping times). But if I could have had a fast cheap server in Australia then all Australians would have benefited. People who access my blog and my web site from Europe (and to a slightly lesser extent from the US) should notice a massive performance increase, particularly when I start hosting big files.
It seems to me that the disadvantages of hosting in Australia due to bandwidth costs are hurting the country in many ways. For example I run servers in the US (both physical and Xen DomUs) for clients. My clients pay the US companies for managing the servers, these companies employ skilled staff in the US (who pay US income tax). It seems that the career opportunities for system administrators in the US and Europe are better than for Australia - which is why so many Australians choose to work in the US and Europe. Not only does this cost the country the tax money that they might pay if employed here, but it also costs the training of other people. It is impossible to estimate the cost of having some of the most skilled and dedicated people (the ones who desire the career opportunities that they can’t get at home) working in another country, contributing to users’ groups and professional societies, and sharing their skills with citizens of the country where they work.
Companies based in Europe and the US have an advantage in that they can pay for hosting in their own currency and not be subject to currency variations. People who run Australian based companies that rent servers in the US get anxious whenever the US dollar goes up in value.
To quickly investigate the hosting options chosen for various blogs I used the command “traceroute -T -p80” to do SYN traces to port 80 for some of the blogs syndicated on Planet Linux Australia [3]. Of the blogs I checked there were 13 hosted in Australia, 11 hosted independently in the US, and 5 hosted with major US based blog hosting services (Wordpress.com, Blogspot, and LiveJournal). While this is a small fraction of the blogs syndicated on that Planet, and blog hosting is also a small fraction of the overall Internet traffic, I think it does give an indication of what choices people are making in terms of hosting.
Currently the Australian government is planning to censor the Internet with the aim of stopping child porn. Their general plan is to spend huge amounts of money filtering HTTP traffic in the hope that pedophiles don’t realise that they can use encrypted email, HTTPS, or even a VPN to transfer files without them getting blocked. If someone wanted to bring serious amounts of data to Australia, getting a tourist to bring back a few terabyte hard disks in their luggage would probably be the easiest and cheapest way to do it. Posting DVDs is also a viable option.
Given that the Internet censorship plan is doomed to failure, it would be best if they could spend the money on something useful. Getting a better Internet infrastructure in the country would be one option to consider. The cost of Internet connection to other countries is determined by the cost of the international cables - which can not be upgraded quickly or cheaply. But even within Australia bandwidth is not as cheap as it could be. If the Telstra monopoly on the local loop was broken and the highest possible ADSL speeds were offered to everyone then it would be a good start towards improving Australia’s Internet access.
Australia and NZ seem to have a unique position on the Internet in terms of being first-world countries that are a long way from the nearest net connections and which therefore have slow net access to the rest of the world. It seems that the development of Content Delivery Network [4] technology could potentially provide more benefits for Australia than for most countries. CDN enabling some common applications (such as Wordpress) would not require a huge investment but has the potential to decrease international data transfer while improving the performance for everyone. For example if I could have a Wordpress slave server in Australia which directed all writes to my server in Germany and have my DNS server return an IP address for the server which matches the region where the request came from then I could give better performance to the 7% of my blog readers who appear to reside in Australia while decreasing International data transfer by about 300MB per month.
HOWTO: Debian and SCSI multipathing with multipath-tools
After getting iSCSI working on Debian Etch the next thing to do is to set up multipath to get redundancy in case one path from the SCSI client to the SCSI target fails.
First, let’s digg a bit more in depth about what a path is, what can go wrong and what we can do to prevent it. Usually in a simple iSCSI environment there are two network interfaces dedicated to the remote storage, each one connected to a distinct ethernet switch and each switch connected to the a distinct ethernet interface in the host SAN. Then here you have two separated controller cards (let’s call them A and B) which connect to the same logical volume (a RAID array.. so here redundancy is already covered). I repeat, this is the simplest redundant scenario, in which you can have redundancy, a good fault-tolerance and can parallelize via round-robin the requests from the initiator to the host target.
So, let’s imagine we have configured both interfaces (and all the needed connections) in our server (the initiator) and we send an iSCSI discover request on both interfaces:
:~# iscsiadm -m discovery -t sendtargets -p 172.16.1.10
172.16.1.10:3260,1 iqn.2002-10.com.infortrend:raid.sn7612996.101
:~# iscsiadm -m discovery -t sendtargets -p 172.16.11.10
172.16.11.10:3260,1 iqn.2002-10.com.infortrend:raid.sn7612961.112
as said, both interfaces are connected to the exact same data volume(s), because we want some fault tolerance in case one path fails. But wait… what’s happening in kernel land?
:~# dmesg|grep “SCSI device”| grep -v sda # sda is the local disk
SCSI device sdb: 1638400000 512-byte hdwr sectors (838861 MB)
SCSI device sdb: drive cache: write back
SCSI device sdb: 1638400000 512-byte hdwr sectors (838861 MB)
SCSI device sdb: drive cache: write back
SCSI device sdc: 11717947392 512-byte hdwr sectors (5999589 MB)
SCSI device sdc: drive cache: write back
SCSI device sdc: 11717947392 512-byte hdwr sectors (5999589 MB)
SCSI device sdc: drive cache: write back
SCSI device sdd: 1638400000 512-byte hdwr sectors (838861 MB)
SCSI device sdd: drive cache: write back
SCSI device sdd: 1638400000 512-byte hdwr sectors (838861 MB)
SCSI device sdd: drive cache: write back
SCSI device sde: 11717947392 512-byte hdwr sectors (5999589 MB)
SCSI device sde: drive cache: write back
SCSI device sde: 11717947392 512-byte hdwr sectors (5999589 MB)
SCSI device sde: drive cache: write back
We have two volumes exported from the SAN but our server is detecting four volumes, two pairs of identical volumes to be correct. This is quite normal… we are exporting the same volumes on every path so our initiator detects four distinct volumes. One solution could be to mount sdb and sdc and then, if something goes wrong, manually mount in the same point sdd and sde. But obviously this is something we should avoid because it would create unwanted downtime. So, you need multipath.
In Debian, you can install it with a simple:
aptitude install multipath-tools
and have a very very basic configuration editing /etc/multipath.conf with something like this:
blacklist {
devnode “sda”
}
defaults {
user_friendly_names yes
}
Restart the multipath-tools service and then you’ll get your new devices as /dev/mapper/mpath*. These are absolutely ordinary block devices, so you can partition, format and mount them as they were a normal local disk.
When a path fails, multipathd will automatically exclude it from the dispatching algorithm and you won’t notice the failure happened.
As a final side note, remember that you can use multipath with any number of block devices and they haven’t to be iSCSI devices… it could be a failovered DAS as well, for example (Dell M3000 comes to my mind).
Posted in Debian, High availability, Howtos, Linux, StorageHost to host security with SSH Keys
Check the ~/.ssh directory for the user you want to ssh as. There's probably a "known_hosts" file, which keeps track of the machines that user has contacted previously, and there's probably an id_dda and id_dsa.pub. These are the private and public keys of the user, respectively. You might instead see similar files, but with "rsa" instead of "dsa". These are keys that have been created with another encryption method. See more information here.
We have the keys now, so what we want to do is make the remote machine aware of them, so that our account on the source machine which has the private key can connect without authenticating with a password. To do this, we install the public key (the id_dsa.pub) in the ~/.ssh/authorized_keys of the remote account we want to connect to, on the remote host. So, we have
Machine A:
User: msimmons
Machine B:
User: msimmons
machineA$ cat ~/.ssh/id_dsa.pub
[text output]
machineB$ vi ~/.ssh/authorized_keys
[insert text output from machineA]
Ensure that the permissions on the authorized_keys file are not world-writable or the ssh daemon will probably refuse to connect. It should also be noted that your sshd config (probably in /etc/ssh/sshd_config) should be setup to allow key based authentication. The manpage should help you there.
At this point, you should be able to connect from one account to the other without a password. This allows you to use rsync to transfer things automatically, through the cron. It would look a bit like this:
machineA$ rsync -e ssh -av /home/msimmons/myDirectory/ msimmons@machineB:/home/msimmons/myDirectory/
Read the manpage for (many) more rsync options.
There is a weakness to this method, though. Anyone that obtains a copy of the private key (the one in machineA called id_dsa) can pretend to be you, and authenticate as you to machineB (or any other machine that has your public key listed in the authorized_keys). This is potentially a very bad thing. Particularly if you have your private key on your laptop, and the laptop gets stolen. You wouldn't want a thief to get their hands on your private key and compromise the rest of your network. So how to get around not needing a password, but not wanting someone just to be able to use your private key if they get a copy. The answer is to use a pass phrase on your private key.
Through proper use of the ssh-agent and ssh-add commands, you can set up passwordless communication from one machine to another. I could explain the common usage of these, but it would just be duplicating this fine effort from Brian Hatch: SSH and ssh-agent. He talks about setting up ssh-agent and ssh-add, but if you're like me, you've already got existing SSH keys laying around without passphrases. The answer to that is to simply run ssh-keygen -f [keyfile] -p to reset it.
Now that you've got a working secure key and a way of not having to type your passphrase every time, lets figure out how to get your servers to take advantage of the same technique. At the very least, you're going to have to type the user's passphrase once, either the first time you want to connect, or (more likely) when the machine boots up. That is not to say that you'll require a password to boot the server, just that before your cron jobs run, you'll need to start the ssh-agent.
Once you start the ssh agent on the remote machine and add the key (per the instructions above), how do we keep that information static? Well, remember those variables that ssh-agent setup that tell 'ssh' the socket and PID to talk to the agent with? It turns out that you can put those (and any other variables you need to be static and universal) in the crontab at the top:
msimmons@newcastle:~$ crontab -l
SSH_AUTH_SOCK=/tmp/ssh-sBrpd11266/agent.11266
SSH_AGENT_PID=11267
48 10 * * * ssh root@testserver.mydomain uptime > ~/uptime.txt 2>&1
This will allow any of the scripts being called by the cron daemon to access the variables SSH_AUTH_SOCK and SSH_AGENT_PID, which in turn allows your scripts to ssh without using the passphrase. All that is required is updating the crontab when you reboot the machine and/or restart the agent.
On my desktop, since I ssh a lot, I add the same variables to my .profile in my home directory so that I only need to type in the passphrase once. If you find yourself connecting to other machines frequently from the server, you might want to do the same thing.
I'm sure I messed up the explanation in some parts, so if you have any questions, please don't be afraid to ask in the comments. I hope this helps someone set up their key-based authentication in a more secure manner.
Tips for an initial buildout
I know the installers that built my most recent rack at the colocation really appreciated the
layout that I made to show what was going where in the rack. I also prepared spreadsheets listing all the cables and where they went. The colocation also needed all serial numbers to all equipment that I was bringing in, which is good for me to have anyway and is probably a good practice to have on hand.
Has anyone else got any tips for a one-time build out that would help?
mii-tool is deprecated, use ethtool
Title says it all. If you want to check (or set) your ethernet NIC configuration or status in Linux, people used to use mii-tools. But there’s a lot more powerful and modern tool that obsoletes it: ethtool
You can install it with your favourite package manager, if it’s not already present in your system.
In Debian/Ubuntu, you can issue
# aptitude install ethtool
Here it is an example:
# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0×000000ff (255)
Link detected: yes
To PyWorks Speakers and Attendees
I’ve gotten several emails over the past couple of days about meeting up at PyWorks. This is reasonable, since I was the main engine initiating contact with prospective speakers back in the June-August timeframe. However, much to my disappointment, I won’t be attending PyWorks this year.
The story is basically that, in September, the startup I was working for was bought out, and I was thrust into managing projects to migrate the various components of our infrastructure to their new home with the new company. I was also executing a good number of migration tasks myself. This is not to mention the meetings and phone calls and long email threads, all while still maintaining the production site on a day-to-day basis, which is no small challenge.
Life just became a little nightmarish there for a while, and for the good of the conference and my sanity, I told Marco (Tabini, of MTA, who runs the conference, and Python Magazine, and php|architect… the list goes on) to turn over the conference organizer responsibilities to someone who could commit themselves fully to the tasks at hand.
So, I won’t be there. I’m not happy about it, but in the end it was better to turn over the conference than to destroy it altogether.
I’ll see you next year, and I guess there’s still a small chance I’ll make it to PyCon. I expect things to be settled before then.
addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F11%2F12%2Fto-pyworks-speakers-and-attendees%2F'; addthis_title = 'To+PyWorks+Speakers+and+Attendees'; addthis_pub = 'jonesy';Sysadmin Extorts company for better severance package
This isn't the first disgruntled sysadmin story we've seen this year. Please, I beg of you, spare the servers in your rampage. ;-)
Where to put your system monitoring
My current setup is that I have 3 “data sites”, which I consider to be physical locations where servers are kept. The primary site, the backup site, and the soon-to-be-primary site. When the new site becomes primary, the current primary will become backup, and the backup site will go away. Here's how they're setup:
They are geographically diverse. and as you can see, there is limited bandwidth between them.
Nagios is currently set up at the Backup site, and has remained unchanged for the most part since the backup site was the primary (and only) data site. This is not ideal, for a bunch of reasons.
Because of the way Nagios queries things, it is at the mercy of the networking devices between it and the target. If the router in-between goes down, then Nagios sees everything beyond that router as down. You can alleviate the most annoying side effect (dozens or hundreds of alerts) by assigning things beyond the router to be "children" of the router, in which case Nagios will only let you know that the parent is unavailable.
Aside from not having status checking on entire segments of our network in the event of an outage, what if the segment with no network access hosts your mail server? I've had this happen before, and it's disturbing to suddenly receive 2 hours worth of 'down' notifications at 3am. Not a good thing.
To circumvent this type of behavior, I'm going to be employing one nagios at each location:
In the event that one of my sites loses network access, I've still got another host to send messages.
If you monitor, how do you guys arrange your monitoring? If you don't, any plans to start?
HOWTO: the definitive guide to Debian Etch open-iscsi (take 2)
I guess the fact I’m here writing again on this topic goes for that definitive I put in the title the first time :) So obviously it was not so definitive, and here we are again with a, I hope, better and improved version.
This time we are going to use th backports repository and the Etcn’n'half kernel, cause they provide a better and far more stable support for iSCSI under Debian (Etch).
So, first of all add the backports repository:
echo “deb http://www.backports.org/debian etch-backports main contrib non-free” >> /etc/apt/sources.list
and do some basic stuff:
# aptitude update
# aptitude install debian-backports-keyring
# aptitude update
Now, let’s install the newer 2.6.24 kernel from the Debian Etch’n'half project (note: it’s present in the officila Debian repository, it doesn’t come from the backports.org one)
# aptitude install linux-image-2.6-amd64-etchnhalf # remove amd64 if you’re on x86_32
now here, if you are a Broadcom NeteXtreme 2 user (lsmod|grep bnx2), be careful and remember to install these NEW package before rebooting, or you will have an unpleasant surprise
# aptitude install firmware-bnx2
This is due to a change in newer Linux versions
Then reboot, cross your fingers and then install the newer open-iscsi package:
# aptitude install -t etch-backports open-iscsi
Everything should be ok and this time you should have all the config files in the right place, a proper script to mount/unmount iSCSI target devices at boot time and so on…
Anyway, I still prefer the old-school config file, so usually I replace the Debian stock one with something like this:
node.active_cnx = 1
#node.startup = manual
node.startup = automatic
#node.session.auth.username = dima
#node.session.auth.password = aloha
#node.session.timeo.replacement_timeout = 15
node.session.timeo.recovery_timeout = 15
node.session.err_timeo.abort_timeout = 10
node.session.err_timeo.reset_timeout = 30
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Wait = 0
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.MaxConnections = 0
node.conn[0].iscsi.HeaderDigest = None
node.conn[0].iscsi.DataDigest = None
node.conn[0].iscsi.MaxRecvDataSegmentLength = 65536
I have highlighted one line because that parameter is used to choose the timeout after which an iSCSI device is considered dead, and thus that path discarded (we’ll talk about paths later).
So, time to discover new devices now:
# /etc/init.d/open-iscsi restart
# iscsiadm -m discovery -t sendtargets -p $SAN_IP_ADDRESS
# /etc/init.d/open-iscsi restart
check out your dmesg output and look for new /dev/sdX devices.
Some partitioning and formatting later, you can edit your fstab with something like this
/dev/sdb1 /mnt/files ext3 defaults,auto,_netdev 0 0
and you should be done!
Posted in Debian, Howtos, Linux, StorageEncrypted Filesystems out of the box on CentOS
Anyway, I've been looking for ways to encrypt the drives I transport. It looks like the "best" way is to use TrueCrypt for encrypting the entire device. It's cross platform (Windows, MacOS, and Linux) and has a great interface and is pretty easy to script.
My problem is that it is a comparative pain in the butt to get running on my platform of choice (CentOS/RHEL5). If you look, the only supported Linux versions are Ubuntu and SLES. Yes, I can compile from the source, and I have to test things, but I don't want to have to manually recompile things on production servers. I suppose I could compile it once and package an RPM if I had the time and knowledge (and the time to acquire the knowledge). Instead, I decided that it wasn't the solution for me, unless it was the only solution available. So I kept searching.
Today I chanced upon what I think is a great solution. Using the dm-crypt software along with built in loop devices, it's possible to encrypt a device without using any non-native software.
In the (hopefully) unlikely event that the link I pointed to goes away, here is the (much abridged) process:
If you're using a file, rather than a device (to have an encrypted volume sitting on an otherwise unencrypted filesystem), create the file, here using 'dd':
dd of=/path/to/secretfs bs=1G count=0 seek=8
Setup the loop to point to your file/device:
losetup /dev/loop0 /path/to/secretfs
Create the encrypted volume with cryptsetup:
cryptsetup -y create secretfs /dev/loop0
Create the filesystem on the device:
mkfs.ext3 /dev/mapper/secretfs
Mount the encrypted filesystem:
mount /dev/mapper/secretfs /mnt/cryptofs/secretfs
And now you have access.
To remove the filesyste, perform the last few steps in reverse:
umount /mnt/cryptofs/secretfs
cryptsetup remove secretfs
losetup -d /dev/loop0
Whenever you want to remount the device, just follow all the steps above that don't use dd or create filesystems.
There you go, an easy way to have encrypted volumes on your CentOS/RHEL machines.
I’m a Top 25 Geek Blogger… for some value of “Top”
I’m not someone who wakes up every day and looks at how my blog is ranked by all of the various services. I check out my WordPress stats, but that’s really about it. However, someone went and did some of the work for me, and they’ve decided that, of the blogs that they read or that were suggested to them, this blog ranks #20 in a listing of 25.
I’m really flattered, but wonder if it’s an indicator that this is a quality blog, or that they should aim higher in their blog reading ;-P Either way, listing 25 bloggers in a flattering way is a fantastic marketing technique, because most of us are probably egomaniacal enough to say “Hey! Look!” and link back to the list on *your* blog, resulting in lots of traffic. Kudos, and thanks Mobile Maven!
addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2008%2F11%2F10%2Fim-a-top-25-geek-blogger-for-some-value-of-top%2F'; addthis_title = 'I%26%238217%3Bm+a+Top+25+Geek+Blogger%26%238230%3B+for+some+value+of+%26%238220%3BTop%26%238221%3B'; addthis_pub = 'jonesy';
