Helping checking users tracking a total of checking packages*! Join Them!  
Go Back   Touch Toolset > The Lounge > Repo Source
repo.sc Register Projects Helpdesk Members List Search Today's Posts Mark Forums Read


Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 05-04-2008, 02:47 AM
TTS's Avatar
TTS TTS is offline
Administrator
 
Join Date: Jan 2008
Posts: 12,475
Default Database "slightly" optimized

I've been spending the night working on optimizing some aspects of the database... While we don't have ridiculously high loads and what not yet (we get only around 100 users online during peak hours), you should see some small improvements in terms of page reloading and changing. You should, in fact, see a small improvement in most pages involving package listings (IE: search results, category browsing, own package list etc.), as the optimization there made things go much smoother than before.

Unfortunately, without revamping some major parts, and recoding a lot of things, it can't be optimized much further... this is due to a bad database layout I've planned for from way back. While I am working on revamping it slowly, the higher priority still goes to the previously promised feature which have been put back due to my flu which turned epidemic on me (I was kept in bed, and away from computer for about good week and half).

Aside from that, tonight, I intend to finish porting the view aspect of the promised feature for package details. If I get it done tonight, then I can start worry about inserting the data and mapping the layout to accept inputs etc. tomorrow. If all goes well, you should be able to see a majoe feature boost by Wednesday or Friday next week (Pacific time).

Cheers!
Reply With Quote
Advertisement
 
  #2 (permalink)  
Old 05-06-2008, 02:58 PM
TTS's Avatar
TTS TTS is offline
Administrator
 
Join Date: Jan 2008
Posts: 12,475
Default

Due to a performance snag, the feature set is being delayed yet again for an undetermined amount of time... The snag is related package generation takes too long to identify...

For those technical minded, here's the reason:
Currently, I've got a relations table which tracks two things: userid, and package id. There is an UNIQUE index happening between userid, and package id, so there can't be 1-1 twice in the table, and a general index happening on the package id, so the search can be quicker.

To pull a list of related packages, I using a self-join query, which looks for the packages that have most installs with the selected package based on everyone's recommendation, so the system can pull out the packages everyone feels most related. While the query and everything works, it works VERY SLOWLY.

A typical query takes anywhere from about 0.002 seconds to to 0.090 seconds to complete. With each page requiring around 8 to 10 queries, it ensures all pages are generated and shown to you in lesser than 1 second... In fact, most of the time it is generated in lesser than half a second.

The above mentioned relational query takes anywhere from 1 to 4 seconds to complete. This inflates the required page generation time by up to 800%... and that is only from me testing the query for values and testing to implement it into the current layout. As more users (such as each of you) start to use this feature, it will only get slower as system doesn't have infinite amout of cpu cycle waiting for everyone to request it.

As such, a cache system is required, and I am working on that right now... What will likely happen is something like this:
Upon request, system will check whether or not there is already a list of related packages, and when was it generated.
If there is NO related packages, it will generate a set and cache it.
If there is related packages, BUT they were generated more than 24 hours ago, it will generate a new set, and cache it.
If neither of the above conditions are true, then it will generate a new set of related packages.

Of course, the cache time will be configurable so we can play around with it to find a good balance between server load and maximum usability
Reply With Quote
Reply

Bookmarks
Digg del.icio.us Facebook StumbleUpon Google

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On




All times are GMT -8. The time now is 07:46 PM.
Powered by vBulletin® Copyright ©2000 - 2009, Jelsoft Enterprises Ltd. Content Relevant URLs by vBSEO 3.1.0
*Database houses over checking packages, number above indicates total amount of packages on everyone's repository. Numbers are updated hourly.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63