Since the content is indeed scarce, I searched the entire web, and there aren’t many people working on Deep Learning for Pixel Art either; there must really be no market for it, hhhh.
First, let’s talk about the ones on GitHub labeled with pixel-art-generator or image-to-pixel. The former are mostly randomly generated, meaningless pixel blocks, while the latter mostly just apply OpenCV or some other image processing library resize, which is basically it. At best, they might add a filter, which is already impressive.
I can only say that pixel art style is somewhat abstract yet requires very specific elements, which is likely something we still can’t fully handle right now.
What is Pixel Art
Regarding the definition of pixel art, I’ve heard many versions, but I feel this varies from person to person,since pixel art styles are evolving; pixel characters drawn years ago already differ significantly from today’s styles.
First, there is a consensus: no anti-aliasing, meaning every pixel is distinct, with no blurry boundaries. After all, pixel art is an art form based on pixels, requiring pixel-level modifications.
The rest, I feel, depends on personal preference. Some like borders, others prefer no borders, half-borders, or discontinuous/non-borders.
Secondly, regarding pixel art colors, there are actually no strict requirements now. Previously, computer limitations restricted us to 8-bit colors, but now you can use almost any color, though certainly not too many (this mainly applies to small-scale works).
Galleries & Datasets
Before we begin, let’s take a look at the galleries and datasets!
Galleries
eBoy: https://hello.eboy.com/pool/everything/1
Many of the works above are excellent!
PixelJoint: https://pixeljoint.com/
This is a gallery from a pixel art forum; the works are a mix of good and bad.
OpenGameArt: https://opengameart.org/art-search?keys=pixelart
Filtering for pixel art yields quite a few works.
spriters-resource: https://www.spriters-resource.com/nes/
Contains some sprites and game screenshots.
probertson: https://probertson.tumblr.com/
All include creation details.
Datasets
sprites: https://paperswithcode.com/dataset/sprites https://spritedatabase.net/download
Some datasets on GitHub
- https://github.com/AgaMiko/pixel_character_generator/blob/master/data.zip
- https://github.com/avidvid/OUP/tree/master/Downloads/Characters
Excellent Projects
Let’s first look at some excellent projects!
info
This section covers two topics: `Pixelate`, which is pixelization, and `Depixelate`, which is de-pixelization.
Pixelate
info
Below are some papers I haven't read yet, for reference only! Only some of the work uses deep learning, so I won't distinguish between them here. Also, I won't distinguish between projects and papers; I'll list them together. Papers will be enclosed in book title marks.
《Automatic portrait image pixelization》 1
This is a 2021 paper. Actually, some information is lost; it only uses image processing, implemented in MATLAB. Unfortunately, the code doesn’t seem to be public, but it doesn’t look too complex.
《Pixelated image abstraction》 2
The results look quite good. This is a 2012 paper, but it actually doesn’t use deep learning.
pixel_character_generator 3
Uses DCGAN, Conditional DCGAN, and DC AutoEncoder for character generation; I can only say the results are not ideal.
Make Pixel Art in Seconds with Machine Learning 4
This uses CycleGAN. Actually, the results are okay. It mentions that training with cartoon images yields better results than real-world images, which makes sense since the cartoon domain is closer to pixel art. Here I’ll also paste cartoonset
The results are indeed awesome, but it seems mainly targeted at faces; the effect on other domains is just average. Although there’s no paper or code, it likely involves background removal, generation with Pix2pix, and finally adding outlines.
《Deep Unsupervised Pixelization》 6 7 8
This is actually work published at SIGGRAPH Asia 2018, np. It achieves pixelization via an unsupervised method; I haven’t studied the specifics yet.
The author trained using StyleGAN with data from the aforementioned eBoy dataset. There’s a pre-trained model on Colab, but it seems it no longer runs.
Depixelate
There are actually many works on de-pixelization, but most are not done using deep learning, and there isn’t a very good compilation of them.
《MMPX Style-Preserving Pixel Art Magnification》 10 11
Also provides a Web tool with implementations and comparisons of various algorithms 12 Here is my own implementation13, but to be honest, I feel the results are just average; it might not even be as good as xBR2X14
《Geometric Total Variation for Image Vectorization, Zooming and Pixel Art Depixelizing》 15 demo 16
Also doesn’t use deep learning; it directly vectorizes the image, np. Maybe everyone feels that going from pixel art to normal graphics doesn’t require deep learning, hhh.
info
The algorithms provided in these two papers are worth referencing.
Other Works
《Towards Machine-Learning Assisted Asset Generation for Games: A Study on Pixel Art Sprite Sheets》 17 18
This work mainly uses Pix2pix to colorize pixel art, offering a good approach: first handle the light-dark relationships, then perform semantic segmentation on the characters, and finally generate characters in various colors (very suitable for making NFTs, hhhh). Unfortunately, the code is not open-sourced, and the dataset is not public.
Drawing Tools
There are countless tools for drawing pixel art. After all, this is a dimensionality reduction strike by other image editors. Besides traditional image editors that basically support pixel art, the most professional and widely used tool currently is aseprite.
aseprite: https://github.com/aseprite/aseprite
Although this is open source, it requires self-compilation or purchase on Steam.
LibreSprite: https://github.com/LibreSprite/LibreSprite
This is derived from the last GPLv2 commit of aseprite, has a release, and is also quite good.
There are a few other projects that are just too much; as I mentioned before, the basic implementation isn’t difficult—it’s just reinventing the wheel over and over.
Conclusion
With the current results based on GANs, one easily observable phenomenon is 太脏了: there’s simply too much fine-grained noise. This aligns with what I said at the beginning: although pixel art is abstract, every single pixel is very specific. There shouldn’t be transitions with only minor color differences. While subtle color variations can introduce color changes, if they’re too continuous, the result loses that authentic pixel art feel, and no one would really acknowledge it as pixel art. As for excellent projects like pixel-me, besides applying some GAN-specific techniques, they also performed image processing before and after the GAN stage. However, they haven’t open-sourced their work, which is genuinely disappointing, and they’ve even made a paid software version :(
If I have time later, I think I’ll try to do something similar.
