4fite.com Blog

I’ve moved tf2.4fite.com off to a new host over the last few days. The transition should be complete, but if you see an ‘offline’ message that means you’re hitting the old web server - your DNS should update soon enough to see the new one.

The new host is basically a virtual server under my control so I’ve tweaked mysql a little to suit my needs, meaning a massive increase in rendering speed for some uncached pages.

At the old host with restricted mysql resources, the completely uncached homepage would sometimes take up to 20 seconds to render. Now it’s more like 200ms on average when uncached, or < 10ms otherwise.

Get what you pay for I guess. I’ll be using the server for a few other things so I can justify bit of a cost increase to myself.

During this process I pruned off data for players that haven’t been updated in 6 months or more. I may start shifting the site more towards updates for members only, period, instead of a mix. I haven’t decided yet.

All this adds up to a faster-responding site which should see updates for members happen on schedule for the forseeable future.

Wow, nearly 700 members and 86,000 player records - you guys rock. Sorry that I haven’t been around to make the site better.

Regulars to the site may have noticed not much changing recently. I took a bit of a break from stuff in general - ‘Two Months Off’, so to speak - and now I’m back into full time work. For now, I have to travel about 1.5 hours each way for this job, so it leaves me with much less time to do serious development work on my own stuff, but at least this time it’s a PHP job. So at some point it may motivate me into continuing to iron out this site and work on some of the new things I had planned.

Feel free to keep using the site, even if it’s in a bit of a feature freeze. I know it’s getting a bit slow again, I’ll look into the database resource usage and maybe go ahead with purchasing more from the host.

p.s. for anyone who is a spamusement fan, I occasionally make my own comics too - spam.whoah.net - hope you get a kick out of it

Update: The upgrade has been completed to a point where the site can be open again. There may be a few funny numbers here and there since the scale of ‘played’ time changed from fractional hours to whole seconds (ie. 1.1 hours becomes 3960 seconds or so) messing with a few calculations, but I’ll find them and fix them over the next few days if they’re there.

I’ve been working on changes to the update process to move it from HTML scraping to accessing the available XML data from Steam, which should make it faster and give it more accurate play time numbers. This requires simultaneous changes to the code in the background updater, the front-end presentation and some of the data structures.

To keep it simple, I’ll pull the site down completely sometime this week to make the transition. Probably a weekday afternoon (Sydney time) later in the week.

Update: Thank you achievement-grinders for highlighting the futility of taking stats seriously at all but at least with over-time figures the insanely high numbers will drop off soon enough.

Looking at the homepage, you may notice a slight layout change. This is to accomodate an extended players dashboard.

The new dashboard features the previous all-time listings, and the new listings for two-week figures. Since the two-week figures require regular updates, they only show members of the site (this is denoted by the little 4fite icon beside each header).

While the all-time figures are a bit of a novelty, hopefully the two-week figures are a little more relevant for people visiting the site regularly.

In the future I hope to take snapshots of this type of info, perhaps on a monthly basis, and add an award system; so it may display in your profile “Most points earned - April 2008″, for example.

Gold stars can be awarded to the “most improved” players each month

Update: NFS are willing to help with more resources, which is my preferred solution since I’d like to stick with them. I just need to make absolutely sure I’ve tuned all I can tune. I’ll do this over the next week or so, most likely there’ll be interruptions to the site and the update process as I restructure some of it.

My host for this site, and a few others - NearlyFreeSpeech - is pretty good. Paying for only what you use in both storage and bandwidth really keeps the costs down for hosting a bunch of micro sites. Apparently their distributed hosting setup can easily cater for much higher traffic sites, too, so long as the site’s structure itself can cope with it.

I seem to be hitting a bit of a performance brick-wall with them though. The nature of this TF2 site is that it stores and processes quite a chunk of statistical data, even though the resulting output in terms of traffic is not that large. Given a couple of servers of it’s own it would run without breaking a sweat, as the size of the database has yet to reach even 500 MB, with indexes accounting for about 20% of that space. But in the shared environment of NFS hosting, while their web nodes are distributed, there’s only one MySQL process per logical database, and their default configuration means it has fairly limited memory resources to use (for example, the key buffer size is only 16mb, and I’m not able to use InnoDB tables).

Some of the basic queries that compute numbers on-demand, although they only run once an hour or so and are cached thereafter, take anywhere from 1 to 10 seconds depending on how much memory paging the server is having to do - I assume. Whereas the same query on the same data will be more like 150ms when run on my desktop, which has an abundance of memory.

But, in all things, you get what you pay for, I just can’t afford to pay for much more at the moment.

The bandaid solution for this is what I’ll be working on next; to move some of those longer running queries, those that can’t be fixed simply through optimisation due to the server setup, into the update process so the numbers are generated in advance instead of on-demand. This doesn’t remove the problem of a long running query, it simply shifts it to a location where it’s effects will be reduced.

After I’m sure I’ve done all the optimising I can do, I’ll probably approach NFS and raise the issue with them, to see if there’s anything they can do. Failing that, I may have to put feelers out to some people I know, to see if I can get some higher-spec hosting in exchange for… positive karma? that’s a currency, right?

I’m trying out a new change - player avatars are now loaded from the Steam site once, and then cached on 4fite, as opposed to telling your browser to load them from Valve’s servers.

This should mean that old profiles that don’t get updated will continue to have valid avatars even when the source profile’s avatar is changed on the Steam site. Also, it paves the way for some future additions to the site. If this doesn’t put too much of a strain on the storage and bandwidth costs, the change will be optimised, kept in place and applied to group avatars, too.

There’s a handful of invalid profile avatars that cropped up during the transition, but they should be fixed up automatically by future updates.

In other news, I took a trip to Valve last week for a job interview… a major excursion considering I live in Australia. Unfortunately I didn’t get the job, for reasons of experience. Most of my experience so far is in tiny agencies working on small - medium websites. I’d like to thank Valve for the hospitality and opportunity anyway. We should be keeping in touch for future prospects, and working together to improve both of our web offerings.

Not much to say. I’ve been busy with work and personal stuff so I haven’t had a chance to do anything new. Apart from a few hosting glitches, the site is still operating as usual. Historical data seems to be working fine.

The site is slowing down a little, but I don’t have the time to properly test it. For all I know it’s the host’s server configuration. As stated in previous posts, their mysql server does act a little odd.

XML data for the player profile page is now available from the Steam site, but like the testing part above, I don’t have the time to make use of it yet.

That’s all for now. More important news to follow next month.

I made a widget that can be placed on your own site if, for some reason, you wish to display your profile(s) somewhere. This only works for 4fite ‘members’ (your profile is activated for regular updates).

Example:

Read the rest of this entry (link below) to see the HTML code.

Read the rest of this entry »

I’ve written a search that runs independently of the database. I can’t pin it down exactly, it’s either the hosting company’s servers or MySQL itself (or even PHP’s interface to MySQL), but a ” LIKE ‘%whatever%’ ” sometimes draws out to 60+ seconds when checking approximately 40,000 records.

All I know, for now, is that my flat-file index of player names is searched in under 1 second. Not fast by any means, or optimised at all, but this is opposed to a search for the same term direct on the database using phpMyAdmin resulting in a 112 second wait.

My indexes are fine. The same query on my Windows desktop’s MySQL server runs under 300ms, with a database exported from 4fite.com and imported untouched.

According to the MySQL documentation, any LIKE comparison where the argument starts with a wildcard character “does not use indexes”, so that could be the killer, but when I run the same query on my development server there’s no sign of any troubles.

As they are intended to index structured sentences with real words, using FULL TEXT solutions was out of the question. Player names certainly do not fit that profile, and a simply substring search is all that’s needed.

Unfortunately I’m fairly new to debugging a MySQL process in a remote / shared / high volume environment, so I’ll have to do some digging around to find some answers. For now though, my workaround will suffice.

Observation of the over-time figures I added in recently shows a discrepancy between the per-class and ‘past 2 weeks’ hours figures that are displayed on the Steam site. I tested this with my own profile.

I was at zero hours for the ‘past 2 weeks’ figure. I then played for a while, over a couple of days. My ‘past 2 weeks’ figure now says 3.6 hours, but there was only a 2 hour difference in my summed per-class figures.

There are two explanations I can think of for this.

First is that the ‘past 2 weeks’ figure takes into account time that isn’t spent in a real game (with both a team and class selected). I mention this because I would have spent at least a half-hour running around testing stuff out on my new PC on a local listen server. That, plus other misc times spectating or joining teams.

Second, and more likely, is simply due to rounding of numbers. If you take a fresh account, and you play each one of the nine classes for 0.09 hours (just over 5 minutes), then your time for each class will most likely be 0.0 (assuming Valve round the hours down) but your time played in the ‘past 2 weeks’ would show as 0.8, because Valve have access to the finer-precision numbers, compared to the ones I scrape from steamcommunity.com.

Neither theory accounts for the 1.6 hour difference on its own. So, without further pondering, I can only assume it’s a combination of both.

In conclusion - if you notice a difference between your ‘past 2 weeks’ figure on the steam site, and the two week figure on 4fite (when it’s available), it’s not a bug as far as I know. They may just be counting different things.