Web Admin Blog

Real Web Admins. Real World Experience.

The Velocity 2008 Conference Experience – Part VII

We’ve reached the last couple sessions at Velocity 2008. Read me! Love me!

We hear about Capacity Planning with John Allspaw of Flickr. He says: No benchmarks! Use real production data. (How? We had to develop a program called WebReplay to do this because no one had anything. We’re open sourcing it soon, stay tuned.)

Use “safety factors” (from traditional engineering). Aka a reserve, overhead, etc.

They use squid a bunch. At NI we’ve been looking at Oracle’s WebCache – mainly because it supports ESIs and we’re thinking that may be a good way to go. There’s a half assed ESI plugin for squid but we hear it doesn’t work; apparently Zope paid for ESI support in squid 3.0 but no traction on that in 4 years best as we can tell. But I’d be happy not to spend the money.

[Read the rest of this entry…]

The Velocity 2008 Conference Experience – Part VI

After a tasty pseudo-Asian hotel lunch (though about anything would be tasty by now!), we move into the final stretch of afternoon sessions for Velocity. Everyone seems in a good mood after the interesting demos in the morning and the general success of the conference.

First, it’s the eagerly awaited Even Faster Web Sites. Steve Souders, previously Yahoo! performance guru and now Google performance guru, has another set of recommendations regarding Web performance. His previous book with its 14 rules and the Firebug plugin, YSlow, that supported it, are one of the things that really got us hooked deeply into the Web performance space.

First, he reviews why front end performance is so important. In the steady state, 80-90% of your average page’s load time the user sees is time after the server has spit it out. “Network time.” Optimizing your code speed is therefore a smaller area of improvement than optimizing the front end. And it can be improved, often in simple ways.

Man, there’s a wide variance in how people’s pages perform with a primed cache – from no benefit (most of the Alexa top 10) to incredible benefit (Google and MS live Search results pages). Anyway, Steve developed his original 14 best practices for optimizing front end performance, and then built YSlow to measure them.

[Read the rest of this entry…]

The Velocity 2008 Conference Experience – Part V

Welcome to the second (and final) day of the new Velocity Web performance and operations conference! I’m here to bring you the finest in big-hotel-ballroom-fueled info and drama from the day.

In the meantime, Peco had met our old friend Alistair Croll, once of Coradiant and now freelance, blogging on “Bitcurrent.” Oh, and also at the vendor expo yesterday we saw something exciting – an open source offering from a company called ControlTier, which is a control and deployment app. We have one in house largely written by Peco called “Monolith” – more for control (self healing) and app deploys, which is why we don’t use cfengine or puppet, which have very different use cases. His initial take is that ControlTier has all the features he’s implemented and all the ones on his list to implement for Monolith, so we’re very intrigued.

We kick off with a video of base jumpers, just to get the adrenaline going. Then, a “quirkily humorous” video about Faceball.

Steve and Jesse kick us off again today, and announce that the conference has more than 600 attendees, which is way above predictions! Sweet. And props to the program team, Artur Bergman (Wikia), Cal Henderson (Yahoo!), Jon Jenkins (Amazon), and Eric Shurman (Microsoft). Velocity 2009 is on! This makes us happy, we believe that this niche – web admin, web systems, web operations, whatever you call it – is getting quite large and needs/deserves some targeted attention.

[Read the rest of this entry…]

Quick Blogging Tip

All yesterday I was being annoyed by the need to write up my blog posts in another editor and paste them over into WordPress.  You have to do that because composing text longer than about 3 sentences in a browser window is taking your life in your hands.  But I discovered even in cutting and pasting from Wordpad you get bullshit formatting inserted that drives the TinyMCE editor crazy.  And Notepad was giving me line break problems.  (And it needs not be said that you should never ever paste from Word…)

But Robert cued me in to PureText, which is a little Windows addon that strips all formatting from text when you cut/paste it for you.  By default you Windows+V instead of Control+V and voila, no crap.  Yay!

The Velocity 2008 Conference Experience – Part IV

OK, now we’re to the final stretch of presentations for Day One.

Cadillac or Nascar: A Non-Religious Investigation of Modern Web Technologies,” by Akara and Shanti from Sun.

Web20kit is a new reference architecture from Sun to evaluate modern Web technologies. It’s implemented in PHP, JavaEE, and Ruby. It’ll be open sourced in the fall.

It uses a web/app server – apache, glassfish, and mongrel – with a cache (memcached), a db (mySQL), an object store (NFS/MogileFS), a driver, and a geocoder. The sample app is a social event calendar with a good bit of AJAX frontend.

I apologize for any lack of coherence in this writeup, but I was at the back of the hall, the mike wasn’t turned up enough, and there were accents to drill through.

[Read the rest of this entry…]

The Velocity 2008 Conference Experience – Part III

In the afternoon, we move into full session mode.  There’s two tracks, and I can only cover one, but that’s what I have Peco and Robert around for!  Well, that and to have someone to outdrink.  (Ooo burn!)  They’ll be posting their writeups at some point as well – you can go to the Velocity schedule page to see the other sessions and to the presentations page to get slides where they exist.

First afternoon session: My panel! I am on the Measuring Performance panel with Steve Souders, Ryan Breen of Gomez, Bill Scott of Netflix, and Scott Ruthfield from whitepages.com (a fellow Rice U/Lovetteer!) It went well. We talked about end user performance monitoring, all the other kinds of tools you can use and their drawbacks, and about “newfangled” monitoring of perf w/AJAX, SOA, RIAs, etc.  No questions; not sure if the audience liked it or not.  But I did get a number of people saying “good work” later so I’ll declare victory. 🙂

Actionable Logging for Smoother Operation and Faster Recovery,” by Mandi Walls of AOL. It’s a quick 30 minute session. Logging should be actionable – concise, express symptoms. Anything logged is something fixable. It should be giving you less downtime – shorter time to resolution. Logging takes resources, so make it worth it.

Filter down your logs to be concise and actionable. Production logging has different goals from dev/QA logging. You’re looking for problem diagnosis and recovery, and then statistics and monitoring. Insight into what the app’s doing.

You need a standard log file location. On our UNIX servers, the UNIX team gives us “/opt/apps” as the place where we can put stuff and gets cranky about any files outside of that. We make everyone log to one place – /opt/apps/logs/<appname> for this reason. Makes it easy to manage disk space, rotate logs, run “find”s, etc.

[Read the rest of this entry…]

The Velocity 2008 Conference Experience – Part II

Just two more keynotes till lunch, but these are larger ones (the previous speakers were 15 minutes apiece; these are 45).  I’ll try to take good notes; every conference always says they’re going to make all the slides available afterwards but at best they usually get a 50% success rate on that.

First, Luiz Barroso from Google speaks on energy efficient operations. Now, server usage is only about 1% of total electricity consumption, but it doubled between 2000 and 2005.  Measuring computing energy efficiency is harder than measuring a refrigerator or the like.  Efficiency is defined as work done/energy used in physics terms. Efficiency for IT can be broken down into computing efficiency (work done/chip energy), server efficiency (chip energy/server energy) and server room efficiency (server energy/server room energy). Surveys show an average PUE (1/server room efficiency) of 1.83, and power supplies dissipate 25% of the power going to servers uselessly, more in PCs. Servers have poor (computing) energy efficiency in their most common usage range.

How do we address this?  First, the power provisioning problem in the data center. Energy isn’t the largest cost – building the center itself takes $10-$22 per watt, but the 10 year power is $9/watt.  Efficiency saves  on both. According to the uptime institute, the average cost breakdown is datacenter – 28%, electricity – 22%, hardware – 50%. (Software dwarfs this in many shops, I’ll note.)

[Read the rest of this entry…]

The Velocity 2008 Conference Experience – Part I

I’m starting out the first year of Velocity, the new O’Reilly-sponsored Web Performance and Operations Conference, watching robots dance to Beck on a video screen. The conference’s tagline is “fast, scalable, resilient, available,” which is just about identical to our Web Systems’ team’s charter.  (And our reputation with the ladies!)

For a long time, we’ve had to bottom-feed off of developer conferences, general interest conferences, etc. to address Web site operational issues; it’s great to see a conference specifically targeted at this growing area. The conference staff noted that the demand was way above what was expected, and were scurrying about to ensure they had enough materials. By rough headcount in the first keynote I’d estimate 400 attendees, with more arriving over time as West Coast standard wakeup time (10 AM, for the record) comes along.

[Read the rest of this entry…]

Next Generation Firewalls

I went to a Lunch n Learn last week sponsored by PaloAlto Networks and Fishnet Security talking about what PaloAlto calls the “next generation firewalls”. PaloAlto boasts having Nir Zuk, principal engineer at Check Point and one of the developers of stateful inspection technology, as it’s founder and CTO. Their product, the PA-4000, Series Firewall, takes an application centric approach to traffic classification and they claim that this helps it to more accurately identify both traditional and emerging applications. This enables it to facilitate true application access control and broad threat prevention. They claim that it is:

  • The only firewall to classify traffic based on the accurate identification of the application, not just port/protocol information.
  • The only firewall to identify, control and inspect SSL encrypted traffic and applications.
  • The only firewall to provide graphical visualization of applications on the network with detailed user, group, and network-level categorization by sessions, bytes, ports, threats and time.
  • The only firewall with real-time (line-rate, low latency) protection against viruses, spyware and application vulnerabilities based on a stream-based threat prevention engine.
  • The only firewall with line-rate, low latency performance for all services, even under load.
  • The only firewall to offer a true in-line transparent deployment option for seamless integration into an existing network infrastructure.

While the presentation itself tended to focus more on analyzing internal user’s connections outbound toward the internet and it seems to do that fairly well, it didn’t cover external users connecting inbound to web applications and things like that so I started asking questions about the firewall’s ability to act as a WAF (Web Application Firewall). I was told that it will do some things like inspection for XSS and SQL Injection, it does not function as a true WAF. I wasn’t even expecting that much so kudos to them.

All-in-all, I tend to believe the hype that this is the next generation of firewalls and while PaloAlto is the first player in the field, I’m sure others will soon follow. The firewall is one of the oldest network security devices out there and PaloAlto has definitely put forth a product that changes the way people will look at them. We think about protecting our networks on an application level and not on a port level so why should our firewalls do things any differently? That said, with this being such a new technology, I’m skeptical of how it works in the real world and am quite certain that it won’t be long before hackers find creative ways in and users find even more creative ways out.

China Says It Lacks Skills To Hack US Systems

I was browsing Slashdot today and found an article on how a spokesman for China’s foreign ministry has said that China, being the “developing nation” that it is, lacks the sophistication to hack foreign systems.  This in response to recent statements from a couple of US Congressmen regarding Chinese probes of congressional systems for information about communication between US officials and Chinese activists.  Upon seeing this, I was instantly reminded of the infamous South Park “Small Penis” defense used by the Japanese in the “Chinpokomon” episode.  In case you haven’t seen this episode (for shame!), let me explain.  The Japanese use Chinpokomon toys to “train” American kids to be Japanese fighters.  When people in South Park start getting the idea that something may be up, the Chinpokomon executive tells the men of the town that they all have very large penises to distract them.  In the episode, President Clinton even addresses the American people by saying:

My fellow Americans, I wish to address the concerns many of us have over the growing number of Japanese military bases forming in the United States. The new Japanese emperor, Hirohito, has made our own children into fighter pilots who will soon fly to Hawaii and attack Pearl Harbor. I spoke with Mr. Hirohito this morning, and he assured me that I have a very large penis. He said it was mammoth, dinosauric, and absolutely dwarfed his penis, which, he assured me, was nearly microscopic in size. My penis, he said, was most likely one of the biggest on the planet. I applaud Mr. Hirohito in his honesty. Thank you.

Does China really think that we’re going to fall for this “developing nation” crap?  They might as well say this (quote from South Park) instead:

We cannot achieve so much with such small penis, but you American wow, penis so big, so big penis!

In all seriousness, you’re either very dumb or very naive if you believe that the Chinese government is not sponsoring hacking activities.  The same goes for the US, Russia, England, and just about every other country out there with an internet connection.  China just needs to man up and say “Yeah, we’re hacking you and it’s easy as pie.  What are you gonna do about it?”  It may not be the politically correct thing to do, but at least it doesn’t make me want to laugh out loud when I read it.  Maybe it’d be the wake up call that America needs to stop thinking about security and start actually being proactive about it.