Posts tagged "web-development" - nolan caudill's internet house

Moderation Amplification

Fri, 20 Apr 2012 07:00:00 +0000

A few nights ago, I came across Tom Coates’ post titled Social whitelisting with OpenID… about how to handle moderating an online forum when the amount that needs to be moderated outweighs any one person or small group of administrators’ capabilities (due to time, sanity, etc). This post was written in early 2007 but this system which advocates building a web of trust between your friends was never built as far as I know.

It did remind me though of a moderation tool that me and couple other tech folks designed and built right around this exact same time for a TV station’s website overhaul.

One golden rule of the web is “don’t read the comments”, because, as nearly an absolute rule it brings out the worst in the worst people. On a news site, this is triple the case. In polite company, religion, politics, and money are things to tread carefully around even with close friends, but with the veil of anonymity that the Internet provides combined with a website that deals daily in stories of religion, politics, and money that often directly impact the reader, it’s a damned near-perfect recipe to attract explosive, hateful, and irrational comments.

As part of this TV site’s redesign, there was to be more of focus on contributions from the viewers, which were in the form of photo and video submissions, guest blogging, a simple “wall” for members to leave comments for one another, and, of course, comments on news articles.

As a hard rule, beside every piece of member-added content, we always put a link where anyone could report abusive content. This would go into a moderation queue in our administrative tools and our editors could act on it, either marking it as abusive or marking it as “seen but okay”. For a news site, with millions of visitors a day, this system wasn’t manageable, as there would be as many false positives and there would be valid abuse. For some, it was abusive if someone disagreed with them and they couldn’t find a way to logically defend their argument. This was overwhelming for a tiny editorial staff.

So, we had to devise something to at least bubble up the true offenders in the moderation queue. (Now, this was 5-6 long years ago and I’m sure it’s evolved since I left, but here’s how I vaguely remember it working.) The idea that we came up with was this: a member reporting abuse is right if we agree with their judgment, and people that report abuse “correctly” more often build up a certain amount of us trusting their judgment. If someone reported abuse 100 times and a 100 times we agree with them, there’s a really good chance that their next report will be correct as well.

So we assigned every user a starting trust score and for every time they reported abuse that we deemed valid, we’d bump up their trust score. On every abuse report, we’d look at the trust score of the person that reported it and if it met some threshold, we’d silently remove it from the site. Their abuse report would still exist in the system, but there was less of a time pressure to go through the abuse queue as after awhile, a small army of reliable reporters would be moderating the site for us.

On the flip side, if an abuse report was deemed wrong, their score would be drastically reduced, halved if I remember correctly. We were fine with the penalty being so severe as good users would build themselves back up and introducing a little chaos into this system was nice as different editors would have slightly different standards, and a lot of the time judging whether something was abuse or not was a judgment call. Chaos was inherent in the system from the start.

These scores were obviously secret, shown only to the editors, and I honestly can’t remember if we were actually doing the silent removals by the time I left, but I do think those reports at least got priority in the moderation queue and when going through thousands of reports, this was incredibly helpful.

I like to view this kind of system as sort of an ad-hoc Bayesian filter where your moderation efforts are amplified, rewarding and ultimately giving some weight to people that moderate like you do.

So, the social whitelist begins with allowing a subset of users to post, while the trust score model involves dynamically building a list of good moderators that agree with you on what is abusive content.

I still love the idea of social whitelisting or building up a set of trusted users to help you with moderating, as both are more organic approaches to moderating, meaning that it’s forcing you as a someone in charge of a community to actually make decisions about what kind of discourse you want on your site.

This is also why it saddens me a bit today as more and more blogs are just dropping in web-wide, generic commenting systems, like Facebook’s. While it is enabling almost everyone to be able to quickly log in and start adding comments, it’s horrible for the site owners that are trying to build an intimate community. Every decent community probably has a baseline standard of what’s acceptable: no hate speech, no physical threats, no illegal content, etc. This is what Facebook provides – a baseline – and nothing more.

Any community worth moderating is nuanced, has a voice and a direction. Facebook doesn’t offer this, so every blogger that drops in this commenting system is making that trade-off between ease of user engagement and being able to effectively manage a community. I’d like to see more sites go back to these more hands-on and thinking-hard approaches to how to moderate and direct their communities instead of relying on someone else’s standards of what constitutes a good contribution.

Thoughts on Pagination

Sat, 24 Mar 2012 07:00:00 +0000

A common navigation pattern on websites is what I call “chunked pagination”: each page has a predetermined number of pieces of content on them. Page 1 shows items 1-10, page 2 shows 11-20, and so on to the end of the stream. Easy. This pattern, though incredibly common, isn’t useful for navigation most of the time. The major issue is that it gives few hints about where those links will drop you in the stream. It’s an arbitrary chunking of how you actually display the content. Pagination should provide accurate navigation points that reflect the overall ordering of the stream, and pagination based around fixed-length pages provide nothing more than arbitrary access into this ordering, where we have to use estimation and instinct about the distribution of the content in order to make a guess of where a link will send us. Having a pagination scheme that closely models how a stream is sorted can give you both the casual browsing experience that the numbered pagination provides, as well as powerful navigation abilities that the numbered pagination can’t provide.

For example, take your average photo site that displays the content in a reverse chronological order: that is, newest to oldest. Let’s say your friend has posted 2000 photos to this site. The site shows the viewer 10 things per page. With our prolific user, this gives us 200 pages. Going to the middle of this content takes us to page 100. What does this mean, beyond we’re at the middle? Not much. Let’s say this user posted their first photo to the site years ago, but has just gotten back from a month-long trek through Europe where they took a thousand pictures. So our page numbers monotonically march back 10 by 10, but as we know this stream is sorted by date, and we want to go back in time on their photostream to a dinner you shared 6 months ago, we’ll just have to guess which page to start with. Since we know our friend’s usual posting velocity, we think that ten pages should take us a few months back in time, so we go to page 10. On page 10, our friend is in Europe, looking at the River Seine, just 1 week ago. Let’s go back 10 more pages. Hm, our friend is still in Europe, admiring the beach at the French Riviera. This is frustrating, so let’s try 40 more pages. Click. Damn, our friend is still in Europe (good for him, but bad for your navigation). After some clicking, we’ve got them figured out. We know that page 100 is right before their trip started, so our estimation applies again, disregarding the first 100 pages, we quickly find the dinner pictures. But, now that our friend is back from their trip, they resume their normal posting volume of 2 or 3 things per week. So, after our friend is back from his trip for a month or two, the first two pages cover a few weeks, while the next 100 pages cover four weeks. The concept of “page 100” no longer means anything, as this link is very much a moving target as things keep getting posted to the beginning of the stream. Lots of problems with the page numbers, it seems. First, we have to guess at our friend’s posting volume and frequency to even make a stab in the dark of how a particular page number relates to a point in time. Then, in an ideal world where URLs are meaningful, the page numbered link (to use Flickr as an example, http://www.flickr.com/photos/nolancaudill/page7/ is the seventh page of my photostream) doesn’t point to any specific resource, beyond that it points roughly to the 126th through the 142nd photos I’ve posted and will point to different photos on my next upload. This link is only “valid” as long as the dataset doesn’t change (which datasets tend to do).

Let’s look at the two main ways that people navigate a sorted list. The first, as we discussed, is by seeking. You know where something happened in a particular sorting (by date, in this example) and you want to get to it. Page numbers do let us narrow it down, but it’s usually a guessing game. Go too far back and you have to click back. Not far enough? Click further. Repeat these steps until you narrow down onto the correct page. The second way people navigate is by browsing, where there’s not a specific goal in mind beyond seeing some stuff. For this method, the page numbers are also not necessary. You either want to view the next page, or just jump to some point further in the list. For the former, a simple “next” link is adequate, and for the latter, you can provide this same action but in a way that makes sense for this case, but also for the “seekers”. The way to represent this that satisfies both the seekers and the browsers is to have pagination that actually chunks based around how the list is sorted. Some examples of this in real-life: dictionaries with the tabs on the page edges that show the alphabet, calendars (one page per month), and encyclopedia sets that have one (or more) books per letter. Dictionaries are actually a good representation of the problem of uneven distribution of the content, as the ‘T’ and ‘I’ sections are much thicker than the ‘X’ and ‘Z’ sections. Jumping halfway into a dictionary doesn’t mean much at all, but having those handy tabs on the edges give you a good head start. Also, even if lots of words are added or removed, jumping to the ‘J’ tab will always take you to the first ‘J’ word, regardless of changes in the vocabulary. On the web, blogs usually have both the seeking and browsing navigation controls. Wordpress, for example, has the ’next’ and ‘previous’ links at the bottom of the main list views, but usually provides a separate archive page that lists all the posts split out by month. Aaron Cope’s parallel-flickr has an interface that shows all photos uploaded by your friends in the last day. Instead of using pages, the list is divided up by the uploader (signified by their avatars), which is helpful as I have some friends that post one photo at a time, and others that empty their entire memory card in one fell swoop, but I can successfully navigate both cases. In all these cases, form and function work together nicely, with the pagination links reflecting how the underlying data is actually laid out, making both seekers and browsers happy. It also creates useful links such as linking to “January 2010” in a reverse-date ordered photostream will always be constant, regardless of how the data around it changes. Since it came up on the Twitters, I should mention the concept of infinite scrolling as a pagination scheme. (Since blech has a protected Twitter account, I don’t want to write out his tweet verbatim, but I can summarize it for sake of context by saying he took issue with an instance of infinite scrolling, which can be easily deduced from my reply). Infinite scrolling is basically a pretty representation of the ’next’ link that you ‘click’ by scrolling to the bottom of a page. I’ll leave whether or not it’s good user experience to others, but as a purely visual experience, I like it. If it’s the only source of pagination, that sucks, and another navigation scheme should be provided if having your users be able to look through the list or find something is important.

So to wrap this up, how would one create pagination links for our reverse-date order example, which is an incredibly common view? The obvious way is by actually chunking around dates. I believe people that are much better at designing useful things than myself could adapt this into the same form that our current paginate-by-arbitrary-chunk format occupies. I think adapting the links you usually see in an archive view could be represented in a succint form that makes both seeking and browsing easy operations.

IE6 will never go away (it seems)

Thu, 02 Jul 2009 07:00:00 +0000

The Yahoo! User Interface team updated their Graded Browser Support and IE6 is still alive as an A-grade browser, and that doesn’t appear to be changing anytime soon.

The Graded Brower Support, or GBS, is the categorization of browsers into different levels that Yahoo! has agreed to support. An A-grade ranking means that the browser has a large enough user base, and that those browsers will receive Yahoo!’s highest level of QA and testing support, while the other levels receive little to none. This doesn’t mean that Yahoo! sites (or sites that use the YUI tools) will be broken in non A-grade browsers, it’s just that they won’t be QAed as thoroughly. Usually these fringe browsers like Chrome and Opera are more compliant to standards, and work automatically with valid HTML, CSS, and JavaScript. They don’t get the A-grade yet, mainly due to their small user base.

IE6 is still an A-grade browser, while they have plans to move browsers like Firefox 3.0 to C-grade X-grade status by the end of the year. This is amazing. For comparison, Firefox 3.0 came out about a year ago (ie, June 17, 2008), while IE6 came out in August 2001. When IE6 was released, the WTC towers were still standing, no one had seen any of Lord of the Rings or Harry Potter movies, George Harrison was still alive, and it remained to be seen what kind of president George W. Bush was going to be. If IE6 was a human, it would be starting the 3rd grade soon. To put it in web terms, this was 4 years before the first video was uploaded to YouTube, 2.5 years before anyone Facebooked anyone, and even 3 years before Google went public. The browser’s been out awhile.

The fact that Yahoo! has still shown a commitment to ensure that IE6 users get the best experience that Yahoo! can give them means two things — there are a still a lot of IE6 users and the company still makes money off of them. This study done 5 years ago (though still 3 years after IE6 came out) reported that IE users were 4 times more likely to click an online ad than Firefox users. Since Yahoo! is still courting to IE6 users, this must mean that those users are still clicking those ads.

At some point, Yahoo! and the web as a whole will have to drop their A-grade support for IE6. HTML5 and CSS3 are now ready to be consumed with Firefox 3.5, Safari 4, and Chrome 2.0 all available for download. This doesn’t mean that developers and designers have to completely break the web for IE6 users, but maybe these users don’t get the full experience that the new browsers can deliver. We can do what we can for the IE6 users, but also allow the newer browsers to take advantage of the new, exciting things.

I like the analogy that YUI has on their browser support page comparing the different browsers to televisions. You have TVs that handle the incoming signal differently, starting with hand-cranked survival radios that only pick up the audio, to black-and-white TVs, all the way to the 1080p Hi-Defs with millions of colors and surround sound. The broadcast stations send out the signal and your TV handles it as best as it can. No one is surprised when a black-and-white TV doesn’t have color and IE6 users shouldn’t be surprised that their favorite website doesn’t look the same as when seen in Firefox. As stated on the GBS page, “[s]upport does not mean that everybody gets the same thing.” This is a hard fact to digest for companies with a web presence and designers that believe in one true version of their art.

So, why don’t IE6 users upgrade their browsers? Simple — they don’t know what a browser is. This video asks random people the simple question “what is a browser?”. The people they ask get the answer (sometimes hilariously) wrong, but these people are not stupid. It’s just that what browser they use is not important to them. All they care about are the sites they can access with it. And you know what, they’re 100% right. It is just about the web. The reason that web developers and designers are so passionate about browsers is that we are trying our damnedest to make the most forward-thinking, interactive, immersive sites for these people, but we can’t. IE6 – and its large market share – won’t let us.

Good news though is that I think this will change soon and IE6 will be nudged to the door by its own creator, Microsoft. The reason that people still use IE6 is because it was already installed on their Windows machines, so the browser changes when Microsoft says it does. Microsoft will soon see what Google saw when Google decided to build Chrome. Google owns the web, and Microsoft owns the browser. Google wanted a piece of that browser pie, as it meant that more people would be enjoying the web in its full capacity, so they created a browser. Microsoft saw this plan and realized if they wanted a slice of the web market they were going to need to put a real contender into the search market (sorry, MSN), and thus Bing was born.

If Bing takes off, and I honestly think it will, the incentive will be there for Microsoft to really encourage their users to take advantage of the newest version of IE, and finally put IE6 out to pasture.

Few small edits for clarity.

twitter-last-status

Thu, 23 Apr 2009 07:00:00 +0000

It was a night of firsts.

I wrote my first Twitter widget. It is the ‘Latest Tweet’ widget on the far right column of the page that uses Twitter’s public JSONP API to pull in my last Twitter update.

The only thing even mildly interesting is that it has a couple of regexes that finds any ‘@’ names and links them up and (naively) hooks up any hyperlinks as well. These weren’t difficult, but they always take a little bit of tinkering to get right.

More excitingly, I think, is that I decided to push it out to github under a BSD license. This is technically my first open source software, minor as it is.

Helping jQuery Out

Tue, 14 Oct 2008 07:00:00 +0000

I’ve made a commitment to myself to start helping out with some of my favorite open source projects. I’ve started helping in my own little way by hanging out in the jQuery IRC channel. It’s not much effort and I’ve already helped a few people, which feels good. I’m ‘mncaudill’ in there, so feel free to say hi.

I’d eventually like to start contributing code, but I’d like to get used to the community first and see how things are done. With such an intentionally lead codebase, contributing actual code might be a challenge, so helping people out in the channel might be the best way to help.

Next up, I’d also like to help out with django by doing the same. With a more extensive codebase where size is not as big of as concern as it is in jQuery, getting a patch in or two would not be as difficult I would imagine.

Setting up lighttpd/Apache for Django on Slicehost

Sun, 10 Feb 2008 08:00:00 +0000

I finally made the jump and moved the websites I was hosting on my home PC and moved them out to slicehost. Signing up for my slice could not have been easier and so far it has been a flawless experience. The PickledOnion articles were nice to double-check myself to make sure I didn’t miss anything.

I opted for the Ubuntu 7.10 256 MB slice since that was what the sites were priorly hosted on. I decided that I wanted to do things by the book so I set up a lighttpd server that served my media straight up and funneled all Django requests through Apache/mod_python. I couldn’t find an exact way to this previously published so I thought I would add a few code snippets to help others looking to do the same.

First off, I got the Apache/mod_python setup working. This was pretty much a cut and paste from the deployment docs found in the Django book. Nothing tricky there. By default, Apache runs on port 80, but that will be changed later on.

The next step was to get lighttpd working properly. The relevant snippet is below:


$HTTP["host"] =~ "^(www.)?example.com" {
    $HTTP["url"] !~ "^/(public|media)/" {
        proxy.server = ( "" =>
            ( (
                "host" => "127.0.0.1",
                "port" => 81
            ) )
        )
    }
}

Broken down, this says if we get a request for example.com and the URL does not contain one of our media directories (read: a Django request), proxy it to port 81. This way lighttpd will serve up all the static files and then redirect the Django stuff to our Apache instance that will be listening on port 81.

After that, change the ports.conf file for your Apache instance to listen on port 81, and the “server.port” variable in your lighttpd.conf to listen on port 80, and then restart both servers. If everything went correctly, you should now have a 2 servers doing what they do best and have a happy machine to boot.

I did add a couple of lines to my Apache conf to get better performance.

First off, I turned off KeepAlive as suggested by Jacob Kaplan-Moss. KeepAlive is useful if you are using Apache to serve up several files over one TCP connection, like multiple images on a page load. Since on every page load we are making just one request to Apache (for the HTML itself) and lighttpd is handling all of the static serving (which it is very good at it), KeepAlive helps us none and actually hurts us as Apache will quickly become RAM hungry holding on to your full Django code when there is no need for it.

Second tuning measure was to bump MaxRequestsPerChild up. I set mine at 500 and saw a big RAM usage drop. This way since these are fairly RAM-heavy Apache processes, they’ll get cleaned up once they reach a certain number.

Overall, I have been very pleased with slicehost and the Django book was very helpful in getting everything up and running.

jQuery: a new joy

Wed, 07 Nov 2007 08:00:00 +0000

At work, I have been working on somewhat complex site with a lot of Javascript going on in the form of some ajax calls and Google Maps. We noticed that the site had an almost painful load time, approximately 8 seconds for a fresh pull of the home page. This was too much.

The first thing I did was profile it using Firebug’s Net tab. Of all the tools in my web developer’s tool chest, Firebug is far-and-away my favorite, my oft-used, and my most recommended app. Not only does it give you a super nice DOM browser and a great Javascript debugger, but it also has a nice download profiler under its Net tab. It gives you every single element that the browser requests for your page with its HTTP headers and, even more importantly, the time to complete each transaction.

The profiling introduced a few things instantly. First, there were a lot of images and some were rather hefty. Since I was building the site from somebody else’s design files, and that is definitely not my domain, there was not much I could do about that.

The second thing was the Google Maps calls. I put the Google request calls at the end so the rest of the page could load before them. Doing just this took about a second and a half off a fresh homepage load. It was a marked improvement, but still nothing spectacular.

The final thing brings me to the topic of this post. Our standard JS library at work is Dojo. Dojo is great when you are trying to create a RIA interface with all the bells and whistles, such as tabs, number spinners, rich text editors, grids, and trees. Its widget library, dijit, is quite impressive. An early complaint of dojo was that it was a big download. If I remember correctly it was something like 100kb. The Dojo team fixed this 0.9 with a more modular dependency download structure. Say you needed to only use widget A. You’d make the Dojo call in your page for the widget and Dojo would then only download the bits of JS it needed to get that widget on the page. The problem with this is that if you are only using a small subset of the library, you had an exorbitant overhead, not in download weight, but in HTTP requests.

For example, I was only using Dojo to do some absolute positioning stuff for some help tip bubbles and a few simple Ajax calls. This was like going squirrel hunting with a Scud missile. Even with just using core Dojo (no widgets), I made about a dozen HTTP requests just to get the pieces of JS I needed. In total, it took a little over 2 seconds just to load the JS library, even though it only equated to about 60Kb of total file size.

These small tasks almost bordered on something that I could do myself, but since I can be a lazy programmer, I figured a small lightweight JS library that people much smarter than myself have written and tested would be the ticket. I noticed that jQuery was being mentioned more and more in a good way around the web, so I figured this would be the perfect trial to test it out. I can definitely say ‘mission accomplished’ with jQuery.

I used the non-gzipped packed version which had a weight of 47Kb. This took around 50ms to download and even better it all came in one HTTP request, with no just-in-time loading. The Ajax API between Dojo and jQuery was similar enough that it took very little modification to port it over and I even minimized some of my Javascript to use the awesome syntax of jQuery. By simply switching to jQuery, I shaved a little over 2 seconds from the page load time with no loss of functionality. Nice.

After all the profiling and modifying, I managed to cut the page load from a little over 8 seconds to a little over 4 seconds.

As a side benefit, it was actually fun to write Javascript. With jQuery’s well-document API and function list (something that Dojo definitely lacks last checking), I spent more time Getting Things Done than tinkering and experimenting. I was actually trying to do things to improve the visual effects of the site because jQuery made it simple to do so. It was almost completely painless.

This is not a slam on Dojo at all, because if you need some advanced UI features, I’d definitely go down the path towards Dojo. This was simply using the best tool for the job.

I am now looking for more excuses to use the tiny, tidy little jQuery library in other spots. jQuery brought joy to Javascript development, something I wouldn’t really have thought possible.

Django: My Framework of Choice

Wed, 10 Oct 2007 07:00:00 +0000

I am a web developer by trade, and like any skilled worker, I am always looking for the best tools for the job. With my current job, I write PHP on top of a custom in-house CMS. Off the clock though, I work with Python and Django.

First off, I like the concept of a framework. With web development, 95% of the stuff you do is stuff you’ve done before and unavoidably will do again. A well-written framework alleviates a lot of this. Most of the tasks can be broken down to 2 steps: 1) Grab objects from database. 2) Show them. The visual design of a website is always the fun part to do after you get all the heavy lifting done. There are also things like form handling, user authentication, writing XML feeds, and caching that gets done over and over that frameworks are nice enough to say, “hey, I’ll take care of this boring, repetitive stuff and let you get on with creating something cool.”

As far as specific frameworks go, I’ve read quite a bit about Ruby on Rails and wrote some simple apps to test it out. One reason I didn’t grasp on to this one was that I didn’t know Ruby. Another reason is that deploying it seemed tricky and you trade speed of deployment for runtime speed. This is true for any interpreted language, but Ruby on Rails takes it to the extreme in both directions. Also it seems too opinionated at times stepping in the way of what I really wanted to do.

Another one I’ve gave more than a passing glance to is CakePHP. While there are several good ideas in there, and it does increase coding efficiency, it has one big, glaring black mark against and that is it is writting in PHP. PHP is a good language in the regards that it fairly powerful, runs everywhere, runs relatively fast, and everyone knows it, but it is an unattractive and inconsistent language. This link sums it up quite nicely.

Now onto my favorite: Django. When I first stumbled on this one earlier this year, I amazed by the simplicity of the framework and how it just made sense, from organizational and logical standpoints. There is a very strict model/view/controller demarcation (except Django calls the controller a view, and the view a template) which leaves little debate of what piece of code goes where, which in my opinion is a great thing. It also has lots of super handy things that are built in or are easily added, such as an auto-admin interface, unit and regression testing, modular apps, an easy-to-use templating language, great form handling, and user authentication.

Far and away my favorite features are the generic views. In my opinion, most of web development is grab an item or a list of items from the database and show them on a page, possibly paginating if there are a lot of them or maybe dividing them up by their publication dates. With generic views, Django takes almost all of this repetitiveness and abstracts it out so you can get on with making it look pretty or adding some cool new feature. Django brings back a lot of the joy of web development that sometimes gets lost in the grind or when writing that same hunk of logic for the hundredth time.

Not to be one to follow hype, I didn’t fully buy into Django until I built a full site with it. Meghan and I built a soon to be launched site for our upcoming wedding. We built it from the ground up (with Django, of course) in the span of about 3 hours. This time included everything from registering the domain to designing the site. It even has a working blog with a commenting system, and a full-featured admin tailored to my exact needs. Django made the easy things really easy and the hard or repetitive things actually fun to work with.

Speaking of Django...

Wed, 10 Oct 2007 07:00:00 +0000

Within minutes of my last post about my infatuation with Django, I saw this link. It appears that Django is doing quite well out there in the real world.

Curses.com actually uses a lot of my favorite pieces of software. Nice.