February 28

Hi all, hope you're doing well. I've had a very rough month, so I didn't feel up to writing two updates this month. Recovering a bit now I think, we'll see.

Lots of updates to share, details below. As a side note, I've been continually documenting new changes and tech stuff on the docs server, so you can take a look at that if you're interested in what's been going on as well. https://docs.desertedchateau.com/ 

Hotlinking Protection changes

I investigated some odd errors in setting custom error pages for CDN distributions, when a 403 is returned from S3 because a file either doesn't exist or isn't accessible by the user (it doesn't return 404 for security reasons).

Figured it out, basically it involves configuring an additional edge rule on each CDN distribution to direct users to a specific URL, which is implemented. Added to the BunnyCDN section of the docs.

Profile Albums

I've implemented albums for user profiles, which effectively function as separate galleries of their own. The cover image for albums is a separate upload, where the user uploads a cover image that can be marked NSFW or non-NSFW (rather than having lots of categories).

I also finished making separate on-hover buttons on each artwork when a user is viewing their own gallery, to make it easier to delete artworks, and remove/add artworks to albums without going to the edit artwork page each time.

Example of the inline artwork album editing and deletion menu is below, on an example artwork in a gallery. If clicked, a modal appears with the options for the artwork.

NSFW Profile Banners

User profiles can now have an NSFW profile banner, as an optional feature. Whether it's shown instead of the user's SFW banner depends on the following:

• If the viewer is logged in, and has opted into seeing NSFW banners automatically, the NSFW banner displays on load.
• If the viewer is logged in but hasn't opted in, the SFW banner is shown, with a "show NSFW banner" toggle button.
• Non-logged-in viewers see the SFW banner only and no toggle button. The NSFW banner image is not loaded onto the page.

Examples below. 

Font Awesome Pro license

I bought a license for Font Awesome Pro, as I figured the large amount of extra icons and other assets could be useful. Since Deserted Chateau is self-hosted, we don't need to have a constant subscription; the license lets us use the version we have indefinitely.

Display Artwork improvements: resizable

I did a bit of experimenting with the jQuery UI plugin, and figured out that making the details bar in the display artwork details page draggable to resize was a lot easier than I expected. It's now possible for users to resize the area to the desired size when viewing, though I haven't yet added this as a persistent setting which is on the todo list (though it might be a bit awkward to do, to account for users viewing on different devices; perhaps a session-only setting).

Financial Estimates Spreadsheet

Both to make it clearer exactly what's what, but also because of recent changes in both Deserted Chateau's implementation and AWS announcements about IPv4 addresses, I've updated the spreadsheets to be in separate sections so that each piece of the total cost is broken down in detail.

I also had to add a new cost: public IPv4 addresses. AWS has started charging $3.75/month for each public IPv4 being used on most servers, and from May they'll be charging extra on Lightsail instances as well. 

IPv6 migration, test environment infrastructure

To reduce total costs, the Lightsail web servers will need to run on IPv6 only from May onwards, so they don't have any cost associated with a public IPv4 address. This is fine in production, and is easy enough to migrate to in theory, though I encountered a hilarious problem...

My ISP doesn't support IPv6, so I can't SSH into servers that only have IPv6 connectivity. I'll probably have to buy a VPN service for my personal computer that will allow me to connect to IPv6-only servers from IPv4, which isn't a big deal, but it's crazy that some ISPs still don't support IPv6.

I also decided to scale down the test environment infrastructure a bit. For most test purposes, having a load balancer and WAF in operation isn't needed, and it was adding a lot of significant costs (an extra $35-40 a month or so). When testing some features prior to release they'll be useful, but for now it's easier to use the test environments in a more simple form.

Removed private message functionality

I decided to disable private messaging entirely. It's not consistent with Deserted Chateau's 'quiet' theme, it adds a lot of potential moderation burden, and it's an extra part of the codebase to maintain.

Initially I thought to remove the Node messaging servers that use WebSockets, as private messaging was the only function that needed them, but by chance I realised there was another use that could make use of them: long video uploads, where it takes a while to resize and reencode a video, or any other use case involving a long query.

Video Uploads: timeout issue

When uploading long videos (say a few minutes), it can take more than 1 minute to resize the video to 480p to show the user in the submit artwork form. This leads to a problem: it's easy enough to set PHP to not timeout, so that the client gets a response eventually, but when CloudFront is the entry point for the site, the maximum timeout is set at 60 seconds.

This means using a normal AJAX flow for uploading videos doesn't work: CloudFront returns a 504 timeout error before the video resizing is completed. I've implemented a new system that only uses AJAX to request the processing, which returns a JSON web token used to authenticate a request to the WebSockets servers; the WebSockets servers then invoke the AWS Lambda function that resizes the video, and when it's finished, emits the result back to the client. Since WebSockets are continual connections, there's no timeout issue.

In most cases, the user having to wait more than 60 seconds is generally a sign of bad design elsewhere, but here it is useful. That said, it might also be easier to e.g. show a user a very brief preview of their video (e.g. the first 30 seconds), or a thumbnail, to make the artwork submission form smoother.

Video Uploads: HD and long video processing

I've implemented dedicated video transcoding via AWS MediaConvert, for longer videos that Lambda functions can't handle (and for most transcoding, as it's much more practical). Making it work with user-uploaded videos is complex from an infrastructure standpoint, as you need to tie together a couple of services to make it work smoothly.

I detailed it in the documentation server here: https://docs.desertedchateau.com/books/image-video-transcoding/chapter/videos-dedicated-services . In short, MediaConvert is used to transcode the videos, EventBridge is used to get a notification when a job is completed, and a Lambda function is sent the notification from EventBridge. The Lambda function then updates the Deserted Chateau database, to add the finished video's different resolution URLs to the user's artwork entry.

The next step is improving the video player I've made to allow for switching between different resolutions, and choosing a resolution based on what device the user is viewing with (to avoid e.g. sending a 1080p video to someone using a tiny screen or similar cases, where it'll just be slow and waste bandwidth).

I'll also need to modify the code a bit, to allow for choices when deciding what quality of videos unsubbed users are allowed to display. MediaConvert has some slightly odd behaviour with this; I think I know the solution but haven't tested it yet. I'll write it up in the Video Transcoding chapter in the documentation once it's ready.

There's also a lot more to write in the documentation; I sat down and calculated some estimates for the costs of video transcoding, and using x265 is unfortunately prohibitively expensive. On a per-artist basis, it's not high: on an estimate of someone uploading 2 new videos per month, with an average length of 2 minutes, it'd cost $0.7 to transcode them to x265, or $1.15 if transcoding to 720p as well as 480p and 1080p. With 100 artists, that's $70 or $115/month - a major cost increase, and unless they were subscribed, there's no way to offset that. I will do some testing to get an idea of the file size differences on MediaConvert between x264 and x265 encoding so the CDN cost estimates can be adjusted, and finalise the job templates for x264 encoding jobs.

I'll do a more detailed write up of this later; it'll be in the Image and Video transcoding section of the documentation server when it's finished.

Subscription options and formatting

I decided to create an additional subscription option, so that there's two "tiers" of subscriptions.

The main subscription is the higher tier ($3 or $5/month, not sure yet), with a cheaper subscription being $1-2 per month paid annually or in 6 month intervals so it's not bogged down in payment fees. The purpose of this is to let people who don't need the higher tier subscription to use other features, or support the site, without paying the larger subscription fee.

Artwork browsing and display: SPA-hybrid

I've been working on a way to make galleries and search results more easily navigated, by implementing a kind of hybrid SPA architecture for some specific pages to make use of.

I'm not 100% certain on whether to go with this system; it does work, but it introduces a lot of potential error points and might be unintuitive, I need more time to test it. The main advantages are that it makes it easier to go back and forth between artworks and galleries, and makes next/previous buttons on artworks workable, as otherwise keeping track of that between pages doesn't really work.

Thumbnail JPG quality raised to 90 from 80

With videos being a much bigger CDN cost, it made sense to increase thumbnail quality for artworks from 80 to 90 for JPGs. It also helps avoid problems where a particular artwork can be displayed in large size in gallery areas, causing it to get displayed in a larger size than the 640x480 thumbnail format and thus reducing the quality, and should help in making the thumbnails more flexible in future display implementations.

I might experiment with using 85 instead of 90, as a middle ground, and fiddle with the gallery display code to try and reduce the number of scenarios where images display larger than they should, but we'll see.

Mobile Layout implementation

I'm certain Deserted Chateau won't have any mobile apps - they're an absolute nightmare to maintain - but it makes sense to make pages mobile friendly to some extent, so I've begun cleaning up the CSS code and making pages such as artwork display and galleries more friendly for mobile use.

I also need to improve the navigation bar for mobile layouts, so that it's easier to use the menu options; I probably need to redesign bits of it for that to work nicely, but first I'll need to decide on the basic interface, e.g. whether to use a modal as the mobile menu or something else.