September 5
Hi everyone! First of all, thank you very much for your support, I do appreciate it! I know that with this kind of Patreon it can seem like not much is going on, coding tends to be like that, but it's busy behind the scenes :)
Domain name
As (I think) I've said before, ArtCentral isn't the final name of this project, but just the sort of prototype name. I've registered the domain for what will likely be the real website name, but that'll stay a secret for now. Suffice to say the name has nothing to do with art.
I'm also thinking about who to commission for a website logo. I have a few people in mind, it's a question of availability but also a question of me deciding how the logo will be placed (as most artists I know don't do vector graphics, which is common for logos). Nothing too exciting or difficult, just a consideration for down the road.
Security improvements: ReCAPTCHA
I've integrated Google's ReCAPTCHA tool into the Create Account page. Was relatively easy to implement, and the last thing I need is bots signing up en masse. There's actually two versions available for ReCAPTCHA: v2, which is the one users see where you have to tick a box or identify bits of images, and v3, which is entirely invisible and produces a "score" as an entity browses your website, which the webmaster has to act on if the score is too low.
For now at least, the v2 checkbox approach should be enough, but I might implement v3 as well later on if needed.
Security improvements: most files moved out of Apache DocumentRoot
As part of restructuring some of the code, it occurred to me that as the codebase grows, keeping individual sections of PHP code walled off from public access via .htaccess files is both bad for performance (the use of multiple htaccess files slows down Apache) and potentially dangerous if I forget to add one to a new folder that shouldn't be accessed.
I've therefore moved all non-public files out of the document root, which required a little refactoring for folder structure reasons, but now means I don't have to worry about that as the codebase gets larger (and it's pretty large already).
Email systems: Amazon SES (plus a bunch of other AWS services)
A big chunk of my time on ArtCentral in the last two weeks has been focused on email sending. There's two primary use cases for this:
- Sending users activation emails to verify their email addresses
- Sending notification emails if users opt into them, e.g. new artwork by an artist or similar
To do this in a scalable manner, I've been investigating Amazon SES (Simple Email Service). It works well enough on its own after setting everything up, but to make sure my web servers don't get bogged down doing a lot of email sending work at higher user levels required some additional work.
Using an AWS Lambda function, mass emails can be sent at scale without bogging down web servers. In practice, the SES API is a bit weird - you more or less need to send 1 email at a time (you can send to up to 50 people, but if one fails they all fail, so it's better not to). That means a *lot* of API calls, and that's on top of the fact that there are limits on how many emails you can send per day and per second, so the function is likely to take some time to complete on bigger calls. In test mode Amazon initially only let you send 200 emails per day to emails you own, but I've applied for production access and got it. The quotas change on your usage, but right now I can send 50k emails a day and 14 per second, which should easily be enough. I still need to implement a queuing system of some kind, as SES doesn't handle queuing emails for you (so you could lose some emails if you're not careful). Shouldn't be too hard, just needs to be figured out.
Doing this via Lambda comes with an additional complication: making sure we never send emails twice, firstly as a general case and secondly due to Lambda occasionally running events twice in some scenarios (I'm not sure exactly which scenarios, but I know I have to cover that case as the AWS documentation mentions it). This means making the Lambda function idempotent, i.e. making it aware of whether it has already processed a batch of emails. This is achieved using AWS DynamoDB, which is a NoSQL service - basically a schemaless database, where we can make a simple transactions table and record which transactions have been done and which not. I might transfer that to RDS (the main database) at some point, as I'm not sure how beneficial it is to use DynamoDB for this, but we'll see. There might be some benefits to it, as DynamoDB comes with a permanent free tier, so it could offload a bit of database demand.
Actually making the emails is another task. It sounds relatively simple, but designing nice-looking emails isn't that easy: you have to code like it's 1999 (because most email clients rely on ancient CSS standards), then create SES "templates" with HTML and plaintext versions for it to send to users. I've got the functional part of that done for activation emails, but the design still needs tweaking, and I need to define some general rules for the layout of those emails.
All of this will be done asynchronously, with the webservers simply calling Lambda with parameters of what to send and Lambda going off and doing the rest on its own, to minimise webserver load caused by email needs.
As an aside, there's a lot of extra administration overhead when implementing this. Amazon SES requires you to handle bounces and complaints from emails efficiently, which means using SNS (simple notification service) and SQS (simple queue service) to record those events. I then need to write cronjobs to periodically check the queue of bounce/complaint events, commit them to the ArtCentral database, and then both un-flag those users for emails and bring it to admin attention if needed. Some of this still needs to be done; I've gotten my servers to connect to Amazon SQS and read messages, but the actual handling of those messages still needs to be implemented (which shouldn't be too difficult, just another task to do).
Maintenance and optimisation
I've been doing little bits here and there for this. Firstly, adding commented lines to the server configuration in httpd.conf that can be un-commented to enable maintenance mode - where every request to the server goes to the maintenance page, instead of just taking the servers down during maintenance which would leave users confused and concerned.
I also looked into minifying the CSS and JS code for production; that should be easy to do with two npm modules for Node.js (https://www.npmjs.com/package/minify-all-js and https://www.npmjs.com/package/css-minify). I also need to refactor some of the PHP frontend-loading-code so that it can be adjusted to use either CDN-hosted CSS/JS or local versions, as the latter is a lot easier to deal with when testing and the former is much more efficient for production, but that will be quite simple to do.
Page progress and mobile-responsive layouts
This is the bit I can never seem to concentrate on... anyway, I've been reworking the settings pages to be clearer in design and to have mobile-responsive layouts. You can see an idea of what they look like here: https://imgur.com/a/PVYP0KE
The gallery page is mostly unchanged, as you can see here: https://i.imgur.com/y52cDOC.png. I'm focusing at the moment on implementing the other "sub-pages" (e.g. commission info, links) that appear in the bar above the gallery itself. I also had to do a bit of reworking to the Apache configuration to let those sub-areas load on their own via URL.