📅 02 Apr 2024
Look at that, two blog posts? In the same week? Astounding.
TL;DR: I am releasing a new app Bunny which should hit F-Droid any day now! It is a media editor for Android, offering photo and video editing functionality. I’ll go over more details below regarding how it came to be, but if you just want to try it out and don’t care about my stories click here to go to the release page.
Evolution of media editing in PixelDroid
PixelDroid had quite some changes, some of which very under-the-hood for image and video editing:
1. Image editing
At first, PixelDroid only did photo editing. You could crop, rotate, add filters, change brightness/contrast/saturation. As discussed below, there were issues with the initial implementation: I’ll provide some context regarding those issues and how we resolved them.
Some history
When starting PixelDroid, we were surprised to find the offering of Android libraries for photo editing pretty lacking. We ended up cobbling together basic photo editing functionality by using an existing filter library (Zomato AndroidPhotoFilters) and uCrop, a library for cropping and rotating images.
uCrop has been mostly reliable: although falling a bit behind the rest of the (too?) fast moving Android ecosystem, it does it job and is easy to integrate. But over the last years Zomato’s AndroidPhotoFilters has proved less and less satisfying to work with and a major part of the tech debt in PixelDroid.
AndroidPhotoFilters does its image processing using native code: this is better than the JVM for performance, but the main part of the code is still a single threaded loop over the whole image, pixel per pixel. This library was made 6 years ago, so you could think that phones have gotten faster so performance of this library’s way of processing images should be faster, right? Well, it is true that phones’ CPUs have gotten faster. However, images taken by phone cameras have gotten bigger. And they have grown a lot faster than CPUs’ performance for single-threaded workloads like this. So, paradoxically, you could end up in situations where you’re using the fastest phone out there and are having a terrible experience applying filters and using the brightness/contrast/saturation sliders.
Now, of course PixelDroid wasn’t just naively using this library with full-size images for the preview of the editing UI, we put in a lot of work to reduce the work done in the preview, and only do work on the full image when saving. There were still other issues: for instance we needed to have multiple copies of (small versions of) the image in memory. Optimizing this code was messy and difficult, partly because it was layer upon layer of bad design decisions on our part, but also in part because of the way the AndroidPhotoFilters library was designed.
AndroidPhotoFilters is also unmaintained. One very in-your-face effect of this is that it is distributed using JCenter, which was discontinued and put into read-only mode. This makes development tooling very unhappy and further accentuates how this isn’t a stable base to build upon.
With this state of things, we were looking for a way to get away from this dependency, ideally to something extensible, not based on a single-threaded computational model, and maintained or maintainable by us.
A new hope
Many avenues were explored, but most of them had either compatibility issues (they would not work on all devices we would like PixelDroid to work on) or had similarly disapointing performance. Eventually we started researching hardware acceleration using graphics APIs: of those, only OpenGL was avaible on all supported devices and so the only one to seriously consider.
After a lot of trial and error and reading and adapting other projects’ source code, we settled upon an implementation consisting of multiple layers of shaders: this allows the user to do multiple edits and their changes are combined into the result image by the top-level shader. At first our implementation was broken on many devices and all emulators, and it took a lot of reading OpenGL documentation to remove the instructions these devices didn’t like and replace them with more verbose but more compatible ones.
With this OpenGL implementation, the performance is much better, brightness/contrast/saturation sliders don’t feel laggy anymore but are buttery smooth.
What’s really cool about this is that OpenGL shaders are a much more powerful tool to work with! For now, there is just feature parity with what we had before migrating to OpenGL, but in the next Bunny and PixelDroid releases, there will be a new feature to add custom filters. Users will be able to write their own or copy other people’s shaders and use them in app! I will also try to add some funky ideas I have, like a mock ‘pixel art’ filter, or maybe a filter that makes your photo look like a drawing? We’ll see.
The possibilities here are really endless, I can’t wait to see what people come up with.
2. Video editing
PixelDroid also added video editing using ffmpeg: you can crop, change speed, stabilize, remove audio, and change the start and end of the video.
Using ffmpeg is very powerful, especially now that they’ve added support for hardware acceleration on Android, and there is lots of room for easy wins by adding creative features that are just one ffmpeg call away! I mean, what can’t ffmpeg do?
However, it’s tricky to get started with and easy to get wrong: you shouldn’t have to do all the work we did to get everything working properly.
The birth of a new app
1. Creation of the Android Media Editor library
After creating this photo and video editing functionality, we thought it would be worth it to release a library so that others might use this in their apps: it is called, very creatively, Android Media Editor.
The library has a very simple API, basically you give a file to an Activity through an intent and you get another file back after the user is done.
Now that we have a solid base to work with, we are working on adding more features like drawing, text superposition, stickers,…
Check out the library project page for more info on how to add the library to your app and get image editing functionality in a few lines of code! Please share feedback with us if you have ideas of improvement, bugs to report, or any other thoughts :)
2. Bunny release and features
As a proof of concept and also to make our media editing functionality available outside of PixelDroid, I worked to also release a (for now, very simple) app that takes a photo or video file, lets the user edit it by calling the Android Media Editor library, and then saves the file. I called the app “Bunny” and made a nice little mascot for it (and a terrible terrible app icon 😆).
Bunny has already been built by F-Droid, and should be published by the time this blog post goes out. It is published using Reproducible Builds, so you can download the APK from the Gitlab releases and F-Droid has published the exact same APKs using my signing keys, which is very cool in my opinion.
What’s next?
Next up, as I alluded to before and in the last blog post, I will add a very cool custom filter feature to the Android Media Editor library and thus also PixelDroid and Bunny: you will be able to plug in your own OpenGL shaders and do all sorts of cool stuff with your pictures. I’m so curious to what people will come up with!