Conectar
Filestash v0.6 - Building a Better Dropbox, brick by brick
Hello everyone, Mickael from Filestash here.
Today marked the 18th birthday of the Dropbox initial launch on Hacker News, with the infamous top comment from the legendary "FTP guy". Fast forward to 2017, as I was frustrated with all the other Dropbox alternatives, I figured we should have a better path, instead of forcing parts you can't swap over to another, the better way integrates with an ecosystem of 3 different kind of interoperable packages: a storage, a web UI and a sync tool. There's literally more than 100 storage servers available, a couple great options for sync, but what we were really missing is the web UI that integrate everything together. That missing piece became my mission, and 8 years later, I'm very proud of the result even though there's still a very long way to go.
Milestone in v0.6
The frontend was entirely rewritten from React to vanilla JS with the idea to get every last bit of performance back so you have the best possible frontend. As of today, the new frontend which was published out of canary release last month is just better by every possible metric than the previous one.
A crazy amount of flexibility via plugins. You can change any aspect of the application both in the front and back by creating plugins. With this approach, you don't pay the cost of the features you don't need and don't have to maintain a complete fork just because you want to add or remove some features or customise some other aspects.
A new sidebar to navigate around your files - screenshot
A dark mode has been revamped to be much nicer - screenshot
Compatibility with other storage servers and vendors got greatly improved. You'd think SFTP is a standard that work everywhere? Well every vendor has interpreted the specs differently and they all come with their own quirks, same for S3, FTP, etc...
I've added support for a wide range of file type. The list is about to go up significantly this year since we can now make plugins targeting specific file types (eg: the latest one I've made is to handle swf file).
Documentation was entirely rewritten
The backend has become battled tested by millions of people including many attacks (I guess being used by Ukrainian military didn't help)
Thousands of small improvements + features requested by the community, like the video thumbnail plugin, new storages, new integrations with for example office document coming from microsoft, collabora / wopi, support for chunked upload via TUS, MCP server, authorization via signed URLs for QR code and many many more .... The whole list can be seen here
Fun
- 11k stars on github
- 10 millions downloads on docker hub
- 2 millions visitors last year
- 43 languages
- 6 months since I'm working on this full time (and taking a massive paycut)
- The project is used in crazy cool places, places like MIT, university of california Irvine, DHL, Bell and many I can't name publicly but the AWS people did: https://aws.amazon.com/blogs/storage/how-regeneron-built-a-secure-and-scalable-file-transfer-service-using-aws-transfer-family/
What's next?
The objective is to reach v1.0, not sure when this happen but when it does, Filestash will be 10x better than anything else. It's still missing many components, such as a mobile app, tag handling, improvements to make the setup simpler, a smaller size overall, make it easy to install it anywhere, better Chromecast support, enhanced video and image support, quota handling, automated workflows, and fixes for hundreds of issues. When we achieve the ultimate file manager, it will be time for v1.0.
In the coming months, I will be releasing a homecloud edition of Filestash which will be a Dropbox like experience outside the box with a set of premade parts that integrate well with each other and you can easily deploy on your server.
Also to achieve sustainability, the goal is to secure sponsorship from outside organisations. If you want access to some of the enterprise feature like SSO, drop me a private message.
What make Filestash different?
recognizing Dropbox is 3 parts that should be interoperable: storage, UI and sync. Since the very first day, the whole idea was about sitting on the shoulders of giants by integrating with the ecosystem. There's literally hundreds of storage server out there, from the simple openssh SFTP to proftpd, sftpgo, minio, nfs server, samba, ceph, open stack, Dell ECS, IBM GPFS... Reinventing that wheel is crazy, sitting on the shoulder of the whole ecosystem is a much saner approach.
separating storage / authentication and authorisation entirely so you can connect to say an SFTP server from a QR code or delegate authentication to an LDAP directory, a mysql database or anything some code could talk to. That kind of flexibility is unheard of in most selfhosted softwares, as you'd normally would have to fork the whole code base and maintain a fork over time when in Filestash you can just maintain your plugin.
going low level when necessary. The best example of this is thumbnail generation. There's a myth going on in this sub that generating thumbnails is slow, hence you have to generate them separatly and possibly cache them somewhere. While it's true genric tools like image magick are slow at generating thumbnails, they are only slow because they aren't 100% focus on that task. For a 768x1024 jpeg of my kid, Filestash generates a thumbnail in 15ms, the only tool we use is custom C code relying on many tricks exposed by libjpeg. If you take a GIF, Filestash can be 10x to 100x faster because of tricks used to parse things more efficiently than a generic tool like image magick. Why nobody does this? You would have to spend days reading C code made by other people and obsess over how to make it faster, but what I found out is if you constantly take the hard path, it potentially make things a lot faster and nicer.
obsessing over performance. Filestash is a proxy that open a pipe from your browser all the way to your storage and everything is being streamed on that pipe. The objective has been to ensure all the endpoints latency stay bellow 1ms. That kind of target would have been impossible to achieve with something like node, python, PHP, etc...
obsession over UX, nothing less than 60FPS. When you start browsing through a lot of data it would be normal to drop the refresh rate but not with Filestash. I've spent days obsessing of the dev tool performance tab to understand how you can create efficient virtualised list that don't waste CPU cycles. Same for making navigation instant on the folder you've already visited before, apply all the transcient state when you create a file/folder, move things around, delete things, etc... Despite the simple look, there's tons of non obvious things hapening to make things smooth no matter what you throw at it
no reliance on databases. Before I got started with Filestash, I wanted to contribute to Owncloud and Nextcloud to fix the speed issues I had with it but the core issue they had was too deep to be fixed, aka they were making dozens of call to a DB anytime you just list the content of a directory or upload something, and because of that db centric design you can't fix the sync issue that happpen if you touch the underlying filesystem.
a good architecture that allow crazy extensibility via plugins. Just to name an example, over the last week, I was able to provide support for MCP as a plugin so you can have an AI agent doing what you want in your storage. Because it's a plugin, it's totally optional and you can get rid of it entirely.
you shouldn't have to pay the cost for the features you don't need. That's the primary trap software fall onto, you start small and progressively add more and more features even if it does make things slower for everyone else, that's not good!
use the standard library as much as possible. I'll keep trimming on third party dependencies that aren't absolutly necessary. It get me sick everytime I use anything made in say node and see 10 critical security issue coming from dependencies of depencies from project build by high profile companies. If those guys can't get their shit together, it has to show something but nobody seem to care enough.
share links. There's 2 things I don't like with how everyone else does shared links:
- why can't I mount the share link as a network drive? Take the link and mount it natively in your favorite operating system, wouldn't that be awesome? Of course, that's the way Filestash does it since the very beginning
- why can't I share things externally with users who aren't part of the platform? Filestash allows for creating shared link for anyone working at "company.com" and will send a code via email if you set the user to "*@company.com"
From the very beginning I have been very mindfull of differentiating ground truth vs opinions so anyone with different opinions could override mine through plugins. It's a lot of small things like:
- I have a "no slow shit policy". That's why there's no video thumbnail enabled by default, as of today I don't know how to generate thumbnail efficiently for video but if you're fine with "just use ffmpeg" there's a plugin for that
- how should we handle html files? some people will want to edit them while some other will want to view them through say an iframe. Same for csv where some people will like the table view while some will prefer a simple editor. Filestash try to have sane default but if you don't agree with those default, you can always change those via a plugin.
- how search should be done? the default is a recursive search but some people might prefer either no search at all or full text search. Filestash ship with a fts plugin that will crawl and index everything if you want but there's no conscencius on that as not everyone will expect a software to keep downloading things on the background to build that index (especially if you use S3 as a storage which could be costly) and we could easily build extra plugin to support things like RAG in the future
- how should it start itself? a simple HTTP server is nice if you use a proxy to handle SSL termination but some other people might want to do SSL all the way either with their own certificates or self signed certificates or even generating those via letsencrypt directly. Filestash supports all those and more (eg: TOR and HTTP2)
- there's many more examples but the gist is about being able to customise things the way you want because not everybody will like the decision I took and you have a way to change all those
[link] [comments]
