Skip to main content

October 25

Hi all, hope you're well! All ArtCentral stuff today ^^

Infinite scrolling for galleries

I had to refactor a lot of the Javascript for gallery displaying to make this work, and I'm still not entirely sure why it works... I think this is the first case of an eldritch abomination within the code that I'm not willing to question (yet) XD

Also fixed a minor problem with user profile displays being a bit off, as getting the username to be centered whilst keeping e.g. supporter buttons next to it (and not causing the username to move with it) was a bit challenging. Should be fine now though.

Logo / design

I've seen some draft ideas for the website's icon and logo, which all look very promising. I also got some helpful feedback on which fonts to use to match the website's theme (and the logo's theme), so I've implemented those so I can get used to them. It's a little difficult to evaluate right away since the natural feeling upon seeing a new site change is "ooh this looks weird", it needs a little time. The logo and name itself will stay under wraps until launch, though.

Artwork tags and search functionality

Mostly done. Tags are implemented, search is partially implemented; I've gotten artwork searches working, but I also need to implement user search, and the results pages to display them need some work. The real problem here has been deciding what to do with regards to Redis caching.

Redis

Figuring out exactly how to structure data for Redis has been pretty difficult.

Initially, I thought about trying to make a way of formally structuring Redis keys, which database queries they related to, what fields they queried, and then write functions that could automatically detect which Redis keys needed to be invalidated after a database change occurs, but that just hasn't turned out to be plausible. There's just too many complexities, and some queries need everything to be invalidated while some don't, etc etc... so ultimately the only solution is to manage it manually by hardcoding which keys should be invalidated by what conditions. 

Slightly annoying, because something more structured would be easier to maintain, but on the other hand I guess this makes one decision easier - reserving Redis use only for the heaviest and most common DB queries, which for now I think is just "searches" and "displaying user galleries". Even that's going to be a bit of hassle, but doable. 

Searches themselves can be stored in a Redis hash (basically a form of object, a group of key-value pairs). This is needed because searches don't return *every* result of a search, for example a very simplified search query in SQL would look something like:

SELECT a_bunch_of_fields FROM artwork_table WHERE 
description LIKE "%huge boobs%" ORDER BY some_field LIMIT 50 OFFSET 0

When the user wants the next page (i.e. they scroll to the bottom of the browser window, and an AJAX call loads the next set of results), the same query will be run but with OFFSET 50 (limit value + last offset). Since we don't want Redis to only cache the last set of results, we need it to cache one key-value pairing per "page" of results.

By grouping them all together in a hash, they are easier to keep track of. For search this isn't that important, but for user gallery caching, it's critical. Unlike the searching use case, the result of asking the database for a user's gallery never changes unless the user edits their gallery (e.g. updating/deleting/adding an artwork), and as such we don't want the Redis keys to have an expiry time. This means that we have to expire the keys manually, whenever a user modifies their gallery, so that users aren't presented with incorrect information. For the sake of example, imagine this query:

SELECT title,description FROM artwork_table WHERE user_id=1 ORDER BY date_submitted LIMIT 50 OFFSET 0

If a user has, let's say, 125 artworks, there will be three result pages, each of which will have queries like so:

SELECT title,description FROM artwork_table WHERE user_id=1 ORDER BY date_submitted LIMIT 50 OFFSET 0

SELECT title,description FROM artwork_table WHERE user_id=1 ORDER BY date_submitted LIMIT 50 OFFSET 50

SELECT title,description FROM artwork_table WHERE user_id=1 ORDER BY date_submitted LIMIT 50 OFFSET 100

Each of these queries needs a different key, but if the user updates their gallery, all of them need to invalidated immediately. If we store all of these keys for a given user's gallery in one Redis hash, we can simply invalidate the hash. If we stored them as individual keys, we wouldn't know how many keys exist to be invalidated - we'd have to either find the total number of artworks in a user's gallery and extrapolate from there, or some other messy solution.

Initially I didn't think of this, as I don't have much experience with Redis and it doesn't technically have a command for setting an expiry on a hash object. Redis has SETEX commands for making an ordinary key have an expiry time, but to expire a hash object means sending two commands: HSET or HMSET to set the hash initially, and then an EXPIRE command to give an expiry time to the hash, in the case of search caching.

Back to top