This past week I’ve been reverse engineering the macOS IOMobileFramebuffer to find a way to disconnect displays in software for Lunar (https://lunar.fyi/).
And I kept stumbling upon these witness tables while debugging.
This led me to find this write-up and the much more interesting story behind it.
I feel your pain, reverse engineering Swift is such a nightmare. I don't know if Ida handles it better but both Ghidra and Binja produce entirely inscrutable disassembly.
Dynamic linking is in my opinion not that useful anymore in this day and age where the few MB of RAM and disk space you save is not worth the hassle. The amount of dynamic linking issues I encountered on GNU/Linux was insane (fuck libstdc++). Not even glibc manages to keep forward compatibility working (breaking memcpy, breaking DT_HASH, ...)!
It's much better to just statically link your binaries (unfortunately many programs on GNU/Linux do not support this).
Before the security guys come out the woods: No you don't need to rebuilt all programs linked against your dynamic library of choice. Just keep the object files around and relink them against the updated version of the static libraries. If now package manager just allowed differential binary updates (actually why is this trivial optimization not a common thing?) the overhead in terms of download size is not that of a big deal too.
The value you gain is immense: binaries just work (modulo linux ABI issues but they are doing quite a good job at keeping this interface stable)
Apple uses dynamic linking to enable its OSes to evolve. Apple's libraries talk to daemons using IPC protocols such as XPC or MIG; if those libraries were statically linked, the daemons would have to support old protocols forever. And when Apple changes the appearance of its UI, they want all apps to get the changes immediately, and not wait for the apps to be recompiled. For example, way back I implemented window resizing from all sides and every app just got it on day 1; no recompilation necessary!
It’s the main reason why dynamic linking is popular. It’s just that the people on Hacker News often write client software for Linux rather than having to deal with ABI compatibility :)
> Dynamic linking is in my opinion not that useful anymore in this day and age where the few MB of RAM and disk space you save is not worth the hassle
But space in fast cache IS worth the hassle. It's still extremely limited, and since we're hitting the limits of moore's law we're not getting much more of it.
The article touches on this. Statically linking can be a case of "tragedy of the commons" if you statically link a single program, you only see the benefits. But if you statically link all programs in a system, vs dynamic linking of all programs, you're giving up on the chance of having shared caching of commonly used pieces of code between programs, and you're giving up on minimizing code size so you don't have to prematurely evict useful code from the cache.
The article touches on this subject:
> It’s worth noting that the Swift devs [...] care much more about code sizes (as in the amount of executable code produced). More specifically, they care a lot more about making efficient usage of the cpu’s instruction cache, because they believe it’s better for system-wide power usage. Apple championing this concern makes a lot of sense, given their suite of battery-powered devices.
It's a common argument of computers getting powerful so we don't need to care that much for performance and/or efficiency regarding to cpu/memory/storage etc. In some limited cases the argument is valid but most of the time it's not.
First of all while maybe desktops and mobile phones are more powerful now, but we're getting more and more lower spec devices, like smart watches. Even when smart watches will be powerful enough one day, there will be eventually smaller computers, like smart contact lens, nano robots that run in blood vessels etc.
Secondly efficiency is still favourable for powerful computers. A small percentage of cost reduction can be big money save for large corps. A small percentage of energy save of all (or just a portion of) computers in the world can be a big win for the environment.
Lastly, we also run VMs and containers everywhere. Notice how we've already come up with all kind of ways to minimise VMs and containers size and footprint, in order to run more of them, to start them faster, and to transfer images quicker.
Yep. For hot call sites, these optimizations & inlining opportunities make a massive difference to performance. Static linking also allows for faster application startup time. (Though I don't have an intuition for exactly how slow dynamic linking is).
The only argument for dynamic linking being more efficient is that each dynamic library can be shared between all programs that use it. But not a net win in all cases. When you dynamically link a library, the entire library is loaded into RAM. When you static link, dead code elimination means that only the code you actually run in the library needs to be loaded.
But honestly, none of these arguments are strong. Dyld is fast enough on modern computers that we don't notice it. And RAM is cheap enough these days that sharing libraries between applications for efficiency feels a bit pointless.
The real arguments are these:
- Dynamic linking puts more power in the hands of the distribution (eg Debian) to change the library that a program depends on. This is used for security updates (eg OpenSSL) or for UI changes on Apple platforms. Dynamic linking is also faster than static linking, so compile times are faster for large programs.
- Static libraries put power in the hands of the application developer to control exactly how our software runs. VMs and Docker are essentially wildly complicated ways to force static linking, and the fact that they're so popular is evidence of how much this sort of control is important to software engineers. (And of course, statically linked binaries are usually simpler to deploy because static binaries make fewer assumptions about their runtime environment.)
> When you dynamically link a library, the entire library is loaded into RAM.
It doesn't. When you dynamic link a library, no part of it is loaded into RAM. It is page-faulted in, as it is used. In the end, only parts that were really used were loaded into RAM.
A page will be loaded in if any part of it is useful. Given that functions will be laid out more or less randomly throughout a shared library, and programs use a randomly scattered subset of the functions, I think its safe to say that you'll get a lot of bytes read in to ram that are never used.
Especially when we take the filesystem's read-ahead cache into account - which will optimistically load a lot of bytes near any executed function.
If your program makes use of some arbitrary 10% of the functions in a shared library, how much of the library will be read from disk? How much will end up in RAM? Its going to be much more than 10%. I'd guess that you'll end up with closer to 50% of the library loaded in memory, in one way or another. (Though I could be way off. I suspect most of the time the filesystem cache will end up loading the whole thing.)
If its 50% loaded, a shared library thats used once will waste 90% of its download size & disk space and 50% of its ram usage compared to the equivalent static library. And make the application slower to start because it needs to link at runtime. And make the program slower to run because of missed inlining opportunities.
> A page will be loaded in if any part of it is useful. Given that functions will be laid out more or less randomly throughout a shared library, and programs use a randomly scattered subset of the functions, I think its safe to say that you'll get a lot of bytes read in to ram that are never used.
Dynamic linking also allows for extensible applications that have to otherwise be implemented with slower OS IPC calls, and higher resource costs in process and CPU cores management.
Which also prevents good privilege separation/sandboxing. See PAM vs BSD Auth, where the former cannot be secured with anything like pledge/unveil or Capsicum, but the latter can.
So are you saying use Rust, more efficient at the expense of maybe using more memory? Or swift which may save some memory by being dynamically linked but perhaps is a little slower.
Honestly a ton of code is utility code, not run that often and nobody wants to take the time or expense to rewrite that Perl/python/bash script.
Some code efficiency is much more important than others.
> First of all while maybe desktops and mobile phones are more powerful now, but we're getting more and more lower spec devices, like smart watches. Even when smart watches will be powerful enough one day, there will be eventually smaller computers, like smart contact lens, nano robots that run in blood vessels etc.
My desktop/laptop distro dynamically link everything by default just so it can be slightly less work to port to a smartwatch? That sounds like a reasonable argument for having dynamically linking be possible, but it doesn't seem compelling for it being the default for the entire world.
You can't ship reasonable plugin for most of the popular content creation tools - be it audio, video effects, 3d modelling, etc. without support for dynamic/shared libs.
Okay, maybe you can - at the risk of performance cost - e.g. your out-of process plugin that need to move data in/out with a cost, and then it over complicates - because the plugin process might've died - how do you restart it, how do you get it back to state, etc. etc. You just don't have to solve this with .dll or .so
Apart from most perf critical cases, it's a good idea to run plugins in a separate process anyway. This way a bad plugin won't crash or corrupt your own process. You will be able to stop and unload it without risk of leaving locks locked or thread-locals leaking.
And also there's WASM now which has even better sandboxing and OS/architecture independence.
Note that this is how things were done before dynamic linking was invented, and dynamic linking was seen as a big step forward. Processes+shared memory is a really painful and slow paradigm compared to just invoking function calls on the same stack and in the same process.
"a bad plugin won't crash or corrupt your own process."
It still can. The plugin just has to return bad data you're not prepared to handle, or crash in a way you didn't anticipate and code around.
And note that with higher level languages the whole thing is moot. A plugin crashing just means it throws an exception which can be caught, reported and quite likely the host process will survive just fine even though it's all in-process and dynamically loaded.
Something existing and succeeding is not automatically a claim that it's the only one possible thing, or even a novel idea, so dismissal "it's just recycling old ideas" is non-sequitur.
I'm not even sure what you're trying to say with this comment. Is only 1958 LISP or maybe SmallTalk worthy and truly novel, and every other technology must come with a disclaimer that it's just a pale imitation and everything has been done before?
Maybe there was some underappreciated pioneer PL/BYTECODE-1968, but in 2023 there's WASM, CIL, Java bytecode (with marketing as large as the dotcom bubble), maybe Lua or eBPF. For practical application it absolutely doesn't matter which one was first or the most humble. What matters is that WASM is common popular standard now. It has easy to embed runtimes and decent language support. The marketing around it is a huge plus too, because I can tell people "use WASM" and it'll be cool and they'll use it, instead of e.g. instinctively reacting "ewwww, Java".
And good luck debugging / make it performant and still not clear how well the memory allocation is handled (in the dll-host situation, often the plugin may use the alloc/free functionality from the host, and give hints, in wasm case - this would be a bit of sandbox)
I'm not saying you need to never use dynamic libraries. Some tools (especially handling/supporting puligins) obviously should make use of dlopen and co.
It's just doesn't make sense in my head why e.g. nano dynamically links against 10 libraries. Lots of pain for little gain.
Latest dynamic linking bug I just encountered is that big picture mode of steam crashes somewhere in libstdc++ when said library is too new but works fine on older versions.
Cross process shared memory is a thing on Unix so there is no need for the performance cost of copying across a pipe. Sure you have to manage if it dies, but you have to manage it crashing you in the dl case.
But you don't have only Linux, you have (and probably first) OSX and/or Windows and then Linux, and you need an API that works well in that case. Being an in-process dll/so plugin, while fraught with perils gets you to avoid other issues (state, health, restart, identity, etc.)
Also sometimes you don't have a choice, but have to make a dll, for example:
(I wish most have used grpc/flatbuffers/whatever to communicate, but then every RPC call have to be checked/retried/handled, and or shared memory well handled, with (?) locks, etc. - not a trivial thing for someone who is deeply specialized in making a very good effect/renderer/etc instead of dealing with this extra complexity on top).
I use a Python script to read the shared memory and generate JSON with this information, then a web page using this JSON is overlaid using OBS, and this way we transmit races on YouTube.
I don't know C#, but it should work similarly. This also means you definitely don't need to use the same language to create and to consume the information.
Ah, so I (the programmer) build object files, send them to the distro maintainers, they link them with their specific versions of the libraries, ands send the resulting executable binary to me/other users.
...that's remote dynamic linking, only with (potentially) human intervention in the loop. I kinda see how it would have worked better if the distro maintainers cared about package quality and speedy updates but they don't (bacause they physically can't, it's way too much effort for way to few people).
Look, I don't argue that dynamic linking has zero problems — heavens know how much I've struggled with glibc but that's mostly because glibc is bloody insane. The whole slew of GNU dynamic-linking-related extensions to ELF is insane, now that I think of it.
The distro maintainers take your source code and package it. If they need to update the program due to an outdated dependency they don't need to recompile the whole program but rather just relink their cached object files with the newer version of the static library (if applicable).
In GNU/Linux, static libraries are precompile object files (.a are nearly indistinguishable from .o files)
Obviously it is not the same at all (LTO, inlining etc.
Indeed in some cases API and ABI breakage requires you to recompile your program from scratch. This is not differnt from dynamic libraries where ABI breakage is only discovered during runtime.
The use case Im talking about tough is your classical 3 bugfix in a library.
Now you have to convince all your software providers to ship those object files. I'm sure it won't be a problem, it's only a really huge change to the entire way in which software packaging and distribution works today...
It's fine as a thought exercise, and might work really well in a restrictive environment (e.g. Google) where you control all the source going into your system. For what most of us call the Real World, it's highly problematic.
Dynamic linking is the shit! It's so undersold. Static linking has been convenient & simple, but it gives up so much awesomeness.
On consumer systems, memory pressure is real, and being able to have quarter your program be effectively free is enormously helpful. Cache hits also go way up, which can be even more vital.
Server side, it depends. If you just run a db server and a monolith, it indeed doesnt matter. But if I look at the hosts we run at my work, if you look at many clusters, they're big nodes and they're running hundreds of processes. If we could get back decent double digit percent of memory that would be great. If caches would get more effective that'd be super. We're leaving a ton on the table because nothing is shared.
There's similar works to win back basic obvious performance happening in a number of places. Composefs has shown up on HN a couple times, trying to let containers effectively share files & page-cache.
Will I deny that dynamic linming has brought pain? No. But I think it's important to recognize that this is a fallback position. If we were better about not breaking conpatibility, not introducing changes, this would be less of an issue. If we had better software testing that could help us detect & see incompatibility faults as they arise, if we could canary test a new software Bill-of-Materials & have some early indicators, we might have, a decade ago, not collectively decided to give up & give in to "binaries just work".
> Just keep the object files around and relink them against the updated version of the static libraries.
This wouldn't work, as extra care needs to be taken to not break the ABI of a shared library. We have various conventions to aid with this including but not limited to "so versions", but these must be kept in mind by the library developer. Object file ABI stability isn't even guaranteed by the compiler.
Also, these days with LTO, the linking step takes a tremenduous amount of time anyway. Doing this wouldn't help much.
Dynamic linking is not the only way to deliver security fixes, just one particular, limited, C-oriented solution. Note that dynamic linking is also not sufficient to change C++ templates or C header-only libraries. It shifts all of the code replacement complexity from the OS (just replace a file) onto the language and library authors (ABI compat, fragile base classes, no inlining or generics).
Rust/Cargo gives you Cargo.lock (and more precise build plans if you want) that can be used to build a database of every binary that needs to be rebuilt to update a certain dependency. Distros could use it identify and update all affected packages. There are also caching solutions to avoid rebuilding everything. It's just something they didn't have to implement before, so naturally distros prefer all new languages to fit the old unbundled-C approach instead.
You ask the vendor, since they need to look after their software anyway. Don't count on closed-source abandonware to conveniently have all vulnerabilities placed in external libraries.
I am talking about apps on the iPhone where Apple does not have access to developers' source code, right? We might be talking about different situations.
Updates of system libraries there are definitely a plus.
However, besides the OS itself Apple doesn't do anything about app security. Not only they don't require unbundling, they don't even support it. Even if you use dynamic libraries in your app, they will be bundled with your app, and never updated separately.
From App Store perspective there's absolutely no difference between a Swift app diligently split into frameworks and a Rust monolith library. They treat both as an opaque bundle.
That's what you would expect from the App Store as well, since why change a dynamic library when the code-controlling developer can just build a new version with his own code when the developer wants to update the app.
Injecting new (the nature of not being available on app-submission) framework versions on the developers' behalf would be scary, since these versions weren't tested with the app because the versions did not exist before. As you said, ABI compat is impossible when no one cares about that much.
I like SDL2's approach to enable both: https://old.reddit.com/r/linux_gaming/comments/1upn39/sdl2_a... A game can statically link the library, but the user can with an environmental variable override the functions to point at their dynamic library instead. Very useful especially with games as you can then keep them running on new platforms or fix certain bugs even though the developers have long since disappeared or moved on.
But then if it works with shared libs, you kind of lose the advantage of the static linking which was mostly (if I understand correctly) that dynamic library was not working well...
It's up to the user if they want to see whether the dynamic library is "working well". They work well for many people! But the statically linked version will be used unless their custom env variable is declared, everyone can be happy.
But they do patch the known security exploits that are likely to be actively used. I'm happier with a security exploit (almost) nobody knows than with a published one that appears in hacking tutorials from 10 years ago.
There are two degrees of separation here though: The software vendors and then the linux distros.
If you sell software that requires your clients to upgrade their system-wide security stack, so they might not. If it is statically linked, no need for them to.
Not convinced at all. Say a big exploit gets published for a very popular library: you probably want to patch it as fast as possible. Whereas Rust will help you have fewer security issues in your own code, which is likely not as popular as some of your critical deps.
I too wonder if it's worth the effort to ship around a trillion versions of different libs (with their trillions of different versions of dependency libs)^n.
One neat thing about it is that we get to override functionality easily, either by replacing entire binaries (such as we do with glide-wrappers to get our 3Dfx fix on modern windows machines) or by using LD_PRELOAD to load hooking SOs to fixup data in old binaries as they are used.
With a "single vendor" providing your stdlib (Apple is a good case in point) dynamic linking is actually better. It is also absolutely no fun whatsoever to recompile your entire hard drive when doing development (which is what Rust work is mostly like, but also Crystal and other "static link everything" approaches). Static linking for a "fast deployable binary for minimum common denominator OS" like the one Go supports is great, but it is great for a usecase (downloadable binaries with narrow functionality) and when your dependency stack is small (no UI, no accessibility, no graphics, no native code).
IMO dynamic linking is rad if you have a known system compatibility table and known vendor (distro) selection for the OS components/key deps. The issue rather is that it is a pain in the butt to configure and understand.
That pretty much sums up what everyone does on Windows. Lots of things are linked dynamically, but apart from the C/C++ runtime library and the OS libraries you just ship all those DLLs with your software.
But this works because in Windows each software is installed in it's own folder, and the search path for dynamic linking starts in the binary's folder. That way you can just dump everything in your installation folder without worrying about comparability with other software. In a Unix or Linux this is much harder to achieve. Sure, you can install into your own folder in /opt and add a wrapper script to load libraries from there, but it's hardly idiomatic.
I did it for 10+ years at my last job, you need a build system that hammers on everything really hard to set the rpath on everything, you shouldn't need wrappers. It definitely isn't idiomatic though.
You assume there’s a philosophy or coherent reasoning behind it, rather than “This is the way we did it with static libraries, so when we adopted shared/dynamic libraries we didn’t change anything else.” Because near as I can tell that’s exactly what happened when BSD and Linux implemented Sun-style .so support in the early 1990s, and there hasn’t been any attempt to rethink anything since then.
Probably because the purpose of the dynamic linker serves the typical O/S layout where there's only one copy of different dynamic libs and everything is linked against those, and packages installed by package managers are authoritative for the things they ship. Distro maintainers want this and lots of system admins expected packages to behave like this.
There's an alternative universe somewhere in which containerization took a different path and Unix distros supported installing blobbier things into /opt, but without (or optionally) the hard container around it. Then fat apps could ship their own deps.
The problem is that there's a lot of pushback from people who want e.g. only one openssl package on the system to manage and it legitimately opens up a security tracking issue where the fat apps have their own security vulns and updates need to get pushed through those channels. It was more important to us though to be able to push a modern ruby language out to e.g. CentOS5, so that work was more than an acceptable tradeoff.
Containerization of course has exactly the same problem, and static compilation probably just hides the problem unless security scanners these days can identify statically compiled vulnerable versions of libraries.
I need to look at NixOS and see if it supports stuff like multiple different versions of interpreted languages like ruby/python/etc linking aginst multiple different installed versions of e.g. openssl 1.x/3.x properly. That would be even better than just fat apps shoved in /opt, but requires a complete rethink of package management to properly support N different versions of packages being installed into properly versioned paths (where `alternatives` is a hugely insufficient hack).
> and static compilation probably just hides the problem unless security scanners these days can identify statically compiled vulnerable versions of libraries
Some scanners like trivy [1] can scan statically compiled binaries, provided they include dependency version information (I think go does this on its own, for rust there's [2], not sure about other languages).
It also looks into your containers.
The problem is what to do when it finds a vulnerability. In a fat app with dynamic linking you could exchange the offending library, check that this doesn't break anything for your use case, and be on your way. But with static linking you need to compile a new version, or get whoever can build it to compile a new version. Which seems to be a major drawback of discouraging fat apps.
Indeed, containers and static linking are just hiding the problem.
> each software is installed in it's own folder, and the search path for dynamic linking starts in the binary's folder.
I think the benefit of this is that an app can be fat but doesn't have to. And an app can be made fat afterwards if need be. The app folder is just the starting point for searching. If the library is not there it is probably shared.
Wrappers and a build system that tweaks everything feel like a hack, not a system wide solution.
There is also Gobo Linux. I wonder if they solved this.
It is done on Linux, for most large popular software. Blender, Firefox, VSCode all are distributed this way. The reason it's not done more is probably some combination of culture and tooling.
Interesting suggestion - basically treating a statically linked binary as a cache which is invalidated when any of the dependencies is updated, triggering either proactive or reactive relinking.
The disadvantage vs. dynamic linking is that you can't easily replace libraries on the fly. It is less flexible but it might actually be better for security. Dynamic linking could be emulated, albeit slowly, by relinking each time a program is launched. This would be slower but potentially useful for certain purposes such as debugging.
The vast majority of libraries on your system are used by only one program. I'd imagine dynamic linking also freezes progress and improvements on libraries because it's extremely difficult to roll out changes without breaking packages that haven't been tested and updated for them.
That post keeps being linked in these discussions but it's highly misleading.
> Do your installed programs share dynamic libraries?
> Findings: not really
The posted findings actually show that common libraries ranging from libc to libX11 are used by high-hundreds to thousands of binaries. The fact that there are also a lot of not-widely-shared dynamic libraries doesn't seem particularly significant.
> Wouldn't statically linked executables be huge?
> Findings: not really
> On average, dynamically linked executables use only 4.6% of the symbols on offer from their dependencies. A good linker will remove unused symbols.
This analysis is completely wrong because it ignores the effect of transitive dependencies. If all exported symbols are independent and don't depend on anything else, then using 4.6% of the symbols means the linker can strip the library down to 4.6% of the size. Simple libraries like libc or libm are pretty close to that ideal. But for more complex libraries, the exported symbols are just the entry points for a deep nest of interlinked functions, and often a single symbol will transitively depend on a large fraction of the library.
> This analysis is completely wrong because it ignores the effect of transitive dependencies.
I hear your criticism, and I'd love to see the corrected results for this. My instinct is that most fully statically linked binaries wouldn't be that much bigger than their dynamically linked counterparts even with transitive inclusion. But I don't know for sure.
Honestly I'm surprised its not easier to run these experiments. If dyld can traverse the call graph through dynamic libraries, why can't it just output the result as a single statically linked binary? I don't know how useful this would be - and it would miss out on inlining & optimization opportunities. But it'd be great for these sort of questions.
At least with ELF, shared libraries are organized into segments and you really only have a handful of them. The information about where individual function boundaries are has been thrown away and in principle this can matter, e.g. if cold code was moved away from hot code. Therefore, the linker can't just split the segments apart.
As for your instinct, it'd be wildly wrong depending on the use case. Maybe if you only link with libc, sure. But now consider linking against something like a GUI framework...
We run hundreds of processes on big nodes at work & we absolutely would save hella hella memory & have way better cache utilization if we could share the massive massive massive reams of code each process duplicates for itself.
Static linking is basically like running Electron. You take a wonderful multi-machine capable environment then use it to run a single thing, exacerbating the size of what should be minor overhead.
As a Linux user, there's like 30 dependencies used by basically half of what I run. If you click in to the article & look at the first chart, it shows this clearly: it tapers down quickly yes but so what? What actually matters? There are a good solid chunk of core libraries used en volume. You dont save much disk space, yes. But many libraries are used by many-hundreds of consumers. Thats where the pain is, that's the loss, that's where you are burning memory space again and again and again with every process you load, and worse, is the cache you are contesting for no reason. You could share that. You should. Giving it up is only because someone has decided it's too hard & too difficult to try,.. and sometimes they are right. But most Linux distros have been working through & managing it & doing ok for years.
>We run hundreds of processes on big nodes at work & we absolutely would save hella hella memory & have way better cache utilization if we could share the massive massive massive reams of code each process duplicates for itself.
Assuming those hundreds are mostly duplicates why can't they all share their entire code section?
There's ~60 different things we run. It would be tempting to try to make some kind of a group scheduler. But on the other hand, we get some nice load levelling/load-averaging from having mixed processes. Ultimately there's not really any reason for us to have static linking, but it's just "how it's done" now and it's unquestioned and there aren't as many well established alternatives as there ought to be.
> The vast majority of libraries on your system are used by only one program.
I find this is a Linux-ism rather than the common case on Windows or macOS. With Windows and macOS a given version of the OS is a fixed set of libraries and services available for third party software. Even third party libraries tend to call into the system libraries for some things. On Linux the only fixed facility of the OS is going to be kernel syscalls since even the libc can differ between distros.
I think dynamic libraries which are only used by a single program are even more common on windows and macos.
Look at how many DLLs ship with applications, and get installed inside the Program Files/MyApp directories! (Seriously - do a search for .dll files there. They're everywhere). Practically all of those DLLs are only be used by a single program. In each case, the price is being paid for dynamic linking (in startup time and the memory cost from not having dead code elimination). But with none of the benefits.
MacOS and iOS apps have the same problem - but I think the modern apple security policies make it even worse. As far as I know there's no way for a mac application bundle to install a shared library on macos or ios which is made available to other applications. The "Drag the .app to the Applications folder" requires that every application ships with its own set of "shared libraries". Outside of the shared libraries provided by the operating system, why would applications ever use shared libraries when they won't ever be shared anyway?
(If its not clear, I'm using the terms "shared library" and "dynamic library" interchangeably here because they mean the same thing.)
On Apple’s platforms the primary motivator for sharing code within an application is that an application isn’t just one component. You may have an application that has any number of app extensions in it that provide access to its features in various settings; if you build that functionality into a framework and use it from the app itself and any app extensions, it’ll still only be loaded into memory once even though there might be several processes mapping it.
Creating frameworks and libraries also helps the developer break down software into more testable components, which doesn’t necessarily require them to be dynamic but if you do go that route you remove an axis of variance that can result in different behavior.
For example, a static library on UNIX is just an indexed archive of object files, so it doesn’t have a single overall set of imports and exports or a single set of initializers to run at load, whereas a dynamic library is a fully-linked module with its own set of dependencies, imports, exports, and initializers. If you assume equivalence between them you’re eventually going to be wrong, often in hard-to-diagnose ways.
iOS developers have the choice of bundling shared code into a static library, or a dynamic framework, and the tradeoffs between them has historically been a little hard to determine, and have also been a moving target.
Some reasons to choose a dynamic library, even though it’s only ever consumed by a single app bundle:
* Shared code between apps embedded in the same bundle (see sibling comment)
* Some apps may want to load portions of code on-demand with dlopen()
* Dependencies / duplicate symbols. Common example is an analytics static library that contains sqlite, and either the main app or another library also linking sqlite, and getting link warnings about “duplicate symbols, which one is used is undefined”. I think this isn’t necessarily a static / dynamic library issue, but 3rd party static libs were often a “single file with all dependencies” and didn’t always understand why that was a problem.
* for a long time, it was the cleanest way to provide & consume headers & object file when distributing closed-source 3rd party libraries.
There may be more, I’m fuzzy on some of the details. If I remember correctly, some of the reasons could be considered tooling issues: creating a Framework was easier for some cases, even if you didn’t want/need the dynamic linking inherent in that choice.
This is a somewhat myopic take on the value of the dynamic linking.
One important aspect is that a dynamic library provides a host of implementations that have been tailored or optimised for specific subsets of CPU's or specific generational CPU features. Let's take «memcpy» as an example that is taken for granted. «memcpy» is defined as weak symbol in the libc.so or similar, and the dynamic linker will replace it with a version that has been optimised for the CPU flavour and CPU flavour's features that have been detected at the runtime or replace it with a less performant, default generic implementation if the dynamic linker does not recognise or support a particular CPU variety. Remember how «memcpy» got an instant manyfold speedup when the AVX-512 optimised version appeared? Yep, all dynamically linked binaries got an instant speedup without even noticing the change. The same will happen (likely, it has already happened) for aarch64 SVE2 vector instructions, and it will happen for RISC-V 64 bit once the vector extension is ratified etc. Doing away with the dynamic linking will slow the adoption of new CPU features down substantially.
If you do away with the dynamic linking, you will not be able to troubleshoot or debug a failing app by substituting the default «malloc» with a debug (or a more performant) implementation via LD_PRELOAD. The Boehm garbage collector was widely used this way (whilst it was being maintained) to overload «malloc» and «free» to successfully run leaky apps that would run out of memory without the GC.
Then there are linking times. Linking Chrome and KDE binaries has been reported to take over an hour and require vast amounts of disk space. We now have «sold» and «mold» so it might be less of an issue but they are not a mainstream linkers yet. Updating a single dynamic library is undisputably faster than relinking 100k binaries that use it. 99.999% of end users will never bother with the relinking anyway.
Naturally, you could claim that the same can be accomplished by fiddling with link maps supplied alongside of object files, but… the link maps are hostile to developers and require a large body of the low-level fringe knowledge that libc authors have taken the burden of. The link maps would have to be very detailed thus very large and also require every developer to have an advanced degree in witchcraft. That would hamper software development efforts for nearly everyone.
Which is not to mention that distributing object files has been tried before, and it has never caught on especially amongst commercial vendors for a number of mostly non-technical reasons. The amount of IP that can be extracted from an object file and the ease of extraction has made the vendors unwilling to distribute anything that is not a final binary product and balk at the idea of it.
You could make a stronger case with suggestion with the OS/400 style of the static binary translation (or AOT) which is functionally combines benefits of object files and some dynamic library features (with the exception of LD_PRELOAD that would have to be replaced with a separate AOT run to «re-link» the final binary product with whatever version of «malloc» / «memcpy» / etc is required), and LLVM has tried that with Bitcode, but for a reason unclear to me, Bitcode seems to have been fading away.
In (open)SUSE we also use delta RPMs. They work in a slightly odd way: The CPIO archive of the installed RPM is is rebuilt based on the installed files, then a binary diff is applied to create the new RPM. That is then installed. (ref: https://github.com/rpm-software-management/deltarpm)
It saves a lot of bandwidth, but takes a huge amount of local resources to install them so I just disable deltas everywhere.
In my mind I see the problem of dynamic linking in rust to have a bunch of overlap with the "I want this rust library to be exposed in my higher level GC'd language with minimal safety issues/tedious handmaintained bindings" problem.
My hunch is that the lack of expressiveness of the C ABI is holding back both. the thing I'd love to see some sort of "higher level than the C ABI" come out. And something like `wasm-bindgen`[0] to exist for more languages.
Here's a link to the rust "interopable_api" proposal! I don't understand all the implications, but it seems to be in the right direction https://github.com/rust-lang/rust/pull/105586
The result of Microsoft developers in the 90s drunkenly making up an object model on top of C with the goal of it being cross-language and cross-platform (in theory). The idea being that you can define an interface and implement it in any language and consume it in any language. A good idea in theory. In practice, DLL hell often made the experience unpleasant. A big part of Windows's APIs are COM-based. For historical reasons it's also used at the core of a few macOS system components, hence the grandparent comment
COM is the main way Windows APIs are exposed since Windows Vista, as the .NET ideas for Longhorn were rebooted as COM.
While plain old bare bones Win32 APIs are still coming up in every release, they aren't the main ones.
Also COM as concept wasn't something designed by Microsoft, it rather build up on the ideas of what Sun, IBM and co were doing with distributed objects protocols.
It is no coincidence that COM IDL language is based on DCE RPC.
At the most basic level (IUnknown), COM doesn't care about DLL loading at all, so DLL hell is not in the picture. Many Windows libraries are actually like that - you call a single global exported function that gives you an "entrypoint" COM object, and from there you call methods on that and/or on other things that it returns. This is the extent of COM that you would usually see in cross-platform code (e.g. Mozilla's XPCOM, or COM Interop in Mono when running on Linux or Mac).
Beyond that, if we're talking about coclasses etc, one of the things about Win32 COM was kinda sorta solving DLL hell by using UUIDs as primary identifiers for those classes - CLSIDs - and mapping them to actual files on disk via the registry. Thus, you could have things installed wherever, including DLLs with identical names in different folders. A new version of the class would get the new CLSID, so multiple versions could be installed and correctly resolved at runtime.
Of course, that just made it a registry hell instead, where messing up those (global!) entries would break apps that depended on the affected classes. So Windows XP (IIRC) added the ability to register CLSIDs and map them to DLLs directly in the app manifest, allowing for a fully self-contained solution.
Yes, lower-level modern APIs are written in it, but you can write a modern windows GUI app without really needing to touch COM directly because there are other abstractions built on those APIs.
This was not as true of Windows programming 20 years ago.
Not sure how this disagrees with my statement that "you can write a modern windows GUI app without really needing to touch COM directly because there are other abstractions built on those APIs"
COM is several things. Although COM on its own always means Microsoft's version and they invented it, there have been quite a few reinventions over time. Mozilla uses/used their own version called XPCOM inside Firefox for example. It can be thought of as a standardized way to expose objects across languages, compilers, incompatible ABIs, processes and machines. It provides:
1. A standardization of vtables and how to obtain/query them from objects.
2. A standardized memory management protocol based on reference counting.
3. A standardized type description/reflection IDL and binary data format, so dynamic languages can reflect objects to learn what methods/properties/events they support.
4. A standardized protocol for dynamically invoking methods and properties by string and using variant types, for scripting languages again.
5. A whole lot of machinery for auto-generating implementations of dynamic method dispatch for objects written in C++/Pascal/C#/etc.
6. A way to load/dynamically link against objects by UUID instead of library name. In COM the actual location and name of a shared library is (in theory) unimportant.
7. A way to "activate" objects either in-process, which is basically a wrapper around dynamic library loading, or out-of-process, in which case Windows starts an EXE that then exposes objects as a server using RPC.
8. A way to do OOP RPC on objects over the network or a process boundary (DCOM).
9. A protocol for negotiating UI embeddings (OLE) so one component could contribute menus and toolbars that would get merged with the host program's UI.
And a lot more stuff that I've probably forgotten.
It's really a very feature complete and powerful framework for solving common problems you face when writing software made up on components written in different languages and by different teams. For example, an advanced use of COM is to run some objects at one privilege level and others at lower privilege levels. Installers/updaters often do this so the GUI runs as the user, and the engine runs as an administrator.
Most platforms (except the web) have something vaguely COM-like or COM inspired. Apple's stack uses MIG, Mach ports and Objective-C for similar purposes. Linux never really did but the closest equivalent is DBUS. Android has the Binder.
COM doesn't get much use these days outside a few stock use cases because other platforms never implemented it, so modern devs (web+mobile) aren't familiar with it and so we are now seeing a new generation 'rediscover' parts of it in the context of WASM, thinking it's new. However in its heydey COM was ubiquitous. There were large markets of components for sale, distributed in binary form and written in/for many different languages. COM stood in the middle making it all work. It was one reason that Windows was relatively language neutral and friendly to language innovation. In comparison other platforms are quite language and binding unfriendly.
Aria is so good at putting things into accessible (and entertaining) words. One of the best at it I've ever seen (they're the same person who wrote the (in)famous "Learn Rust by writing Entirely Too Many Linked Lists", which is one of my favorite pieces of technical writing of all time)
this post makes me realize why swift is starting to look more and more bloated from an end-user pov. It was designed to be right from the beginning. instead of carefully backing off complex problems and slashing features to keep the design elegant, they just "went for it" at full speed.
If I’m reading it right, reabstraction is about allowing you to make a protocol implementation more generic in future. If it’s implemented for String today, then if you change it to be implemented for all T in future then the dylib is still usable. The reabstraction thunks are checking that the type you’ve called it with is actually covered. So they’re not free.
The idea of calling polymorphically compiled functions with pointers to witness tables in extra arguments seems like something we could throw a simple JIT compiler at. A poly-function might have a list of available concrete implementations it could forward you to. You could, as a consumer of a dylib, for each call site on a poly function where you’re passing in a type known at compile time, initially write out a stub function, that the first time it runs, caches the result of this lookup and replaces the stub with something better. The stub would have a signature shaped optimally, the initial body of the stub would pad that out with the witness tables and code to initiate JIT using the poly function, and the JITed replacement would in the ideal case just be a jump instruction to a concrete implementation with a signature identical to the stub’s. Would that help?
> Swift reserves a callee-preserved register for a method’s self argument (pointer) to make repeated calls faster. Cool?
Years ago (2000 I think), I remember Eliot Miranda explaining that the biggest optimization—-over and above all the jit stuff—in the VisualWorks Smalltalk VM basically came down to something very similar: very specific management of registers for the commonest of calling patterns.
They are exactly the same except for when they're not.
(On 64-bit) Rust very naively has two 64-bit integers for the strong and weak count, Swift packs them into only one. Swift also packs in several extra flags for various things [0].
These flags mean that retain/release (increment/decrement) is actually an atomic compare-and-swap instead of a fetch-add. Allegedly performance issues with this were fixed by the hardware team, just, optimizing CASes better.
Swift also has to interop with ObjC "weak" pointers which have move constructors because their address is registered with a global map which is used to null them out when all strong counts go away, but I don't think this changes the design much when not using them.
Swift ARC is built into the language and a huge amount of the compiler's energy is dedicated to optimizing it. This is why it's part of the calling convention (+1/+0), why there are special getter/setter modes with different ARC semantics, why many stdlib functions are annotated with "this has such-and-such semantics" and so on.
Swift ARC is also very pervasive, as basic collections are all ARC-based CoW, all classes are ARC, and I think existentials and implicit boxes also go through ARC for uniformity? You can in principle avoid ARC completely by restricting yourself to value types (structs/primitives) but this is complicated by polymorphic generics and resilient compilation necessitating some dynamic allocations.
ARC is also why historically Swift gave itself fairly extreme leniency on running destructors "early" based on actual use [1]. Eliminating a useless +1 can be the difference between O(n) and O(n^2) once CoW gets involved!
By contrast in Rust it's "just" a library type which you have to clone/drop (increment/decrement) manually. It doesn't do anything particularly special, but it's very predictable. The existence of borrows in Rust lets you manually do +0 semantics without having to rely on the compiler noticing the optimization opportunity, although you do need to convince the borrow checker it's correct.
It's automatic for Swift because Objective-C (where ARC was introduced) already had reference counting but you had to manually indicate whether an object should be retained or released at any point. ARC added in the retain/release and such automatically during compilation.
The designers and implementors are just using standard terminology in the field. In both cases where the concept applies, the underlying objects are being reference counted, with counts being atomically incremented/decremented to allow for thread safety.
Swift builds the behaviour into the language runtime for objects where the behaviour applies, while Rust eschews language runtimes and provides it as an explicit wrapper structure in its standard library.
With one very important difference. Unless you use a thread_safe interior mutability implementation like Mutex Arc won't allow you modify or change the wrapped value. C++'s shared_ptr does not enforce the same. All of which means that while the shared_ptr control block is thread safe what the shared_ptr contains may or may not be. This has caused, and will continue to cause, no end of confusion for folks doing C++ especially for less experienced devs.
There are many differences when it comes to safety:
- Like you said, Rust doesn't let you data race on the contents of an Arc.
- Rust also doesn't let you data race on the Arc itself (i.e. repointing a global Arc without synchronization). This is Bug #5 in this great talk: https://youtu.be/lkgszkPnV8g
But I'm not sure I'd call these Arc vs shared_ptr differences per se, because almost any comparison between a Rust container and its C++ equivalent looks similar.
DLL Hell comes when you have multiple versions of dynamically linked libraries from third parties. In practice, Swift's dynamic linking exists to support Apple's own first-party libraries which are shipped with the system, so there is no DLL Hell. There is only one UIKit at a time.
In 2020 I wanted to leverage the new apple silicon in my Rust app, so I needed to do some work in Swift, and maybe i was looking in the wrong spots - but the Swift community felt incredibly dead. Swift documentation is horrible, I asked for help in a few places online and got none. I eventually solved the problem by guessing to fill in the gaps of documentation. A very different experience than the rust community, which was extremely eager to help with every problem I had.
The TL;DR is: Swift strives for ABI stability, Rust does not.
There's also a driveby comment that C++ doesn't really support dynamic linking because of all the templates in its standard library, which is a weird statement to make.
Yeah - the big, complicated parts of platform libraries for which post-release bugfixes make sense are usually not templated. The vector container class is a template, but Korean input method support is not.
Templates are not a problem if you're defining the complete set of valid specializations in advance, as would be the case here - that's what "extern template" is for.
(Actually, Korean was not a great choice of example. The Korean writing system isn't very complicated. Japanese would have been better - they even have three writing systems!)
Indeed, on Windows C++ dynamic linking has always been available, sure there is the ABI issue, but that doesn't mean there aren't C++ frameworks being shipped as dynamic libraries, e.g. MFC, VCL, Qt,...
A lot of people who are invested in the Apple ecosystem have been trying to present Swift as a general competitor to Rust, but no matter how many bullet points you list, as far as I can tell, it hasn't made any progress at all on that front, and I haven't been convinced that it will change to become closer to that any time soon, either.
As far as I can tell, there's really nothing wrong with Swift, and it probably has a lot to offer, but at some point someone's going to need to point out the obvious: basically nobody outside of the Apple ecosystem is considering using Swift for anything. That doesn't mean it's bad, but it's not a competitor to Rust today, and it's not trending in that direction.
It's a bummer because I feel like I'd really like Swift if I felt it were a real, durable cross-platform language to spend time learning and using. Some of the trade-offs it makes seem really nice
But I don't feel like Swift on non-Apple platforms will ever be more than a hobby for the Apple org
My understanding is that it's got second-class Linux support, for now. What happens when Apple loses interest? What about Windows? I don't get a sense of commitment from them outside of their own platforms, and that doesn't make me want to commit to learning or starting major projects in the language (outside of their platforms). If I can't have faith my project's foundation will continue to be solid, then I don't want to build it there; if I know I'll never have an employer who wants to let me use Swift (again, outside of Apple apps) then I don't really want to invest the time to deeply learn it (unless I'm going all-in on Apple apps)
I could do a toy project or an iOS app with it, and maybe I will some day, but it's hard for me to feel like it's worth going past that while this is the feeling I'm getting from Apple
(And I'm not really even blaming/judging them here; it makes total sense they would want to have a modern in-house language that's perfectly integrated with their systems and now their chipsets, and lets them ship more effectively. They don't owe anyone more than that. It's just a shame, for me personally, to think about what else could have been)
Foundation (Swift's stdlib so to speak) is being rewritten as OSS [0] which will unify it between Linux and macOS. Right now, they use a different implementation on each platform
And that's well and good, it's clear there are some engineers at Apple who would really like to see Swift become something more than just The Apple Language, but at the end of the day they aren't the ones getting to set the priorities. And my (very subjective/anecdotal/external) impression is that the business is just sort of humoring them - maybe Linux support helps a little with internal CI or something - without actually prioritizing multi-platform Swift. If a company with Apple's resources cared about multi-platform Swift, at the business level, we would've seen sterling support yesterday.
For comparison, look at what Microsoft's done with .NET since they started to prioritize multi-platform support.
Speaking as someone with similar feelings… “Well enough” and “decent” are not how I would describe something I’m thinking of investing significant mental time and effort learning and familiarisation building on. I’ve done the basic tutorials and generally like the language ergonomics, it’s not bad, and I kinda like it for the little bit of Apple platform dev I’ve done with it. But it’s basically a different language on any non Apple platform.
This is all very hand-wavey. Are you just describing a chicken-and-egg scenario?
Also, Swift probably can compete with Rust, especially when they finish adding Rust-style lifetimes. But the real advantage of Swift is it has a much more "high-level" feel than Rust – it's much more familiar to use than Rust by virtue of having classes and things. Swift also doesn't instantly force you into weeds of dealing with memory management complexity. I think I'm going to enjoy the "drop into manual memory management for hot paths" approach Swift is taking.
Not really no. It's not a matter of convincing other people to do support, it's a matter of Apple making a long-term on-going large investment into other platforms. They've definitely made some investments and are still making some more, but they're not enough if their goal is to promote Swift as a general programming language for any platform.
I can make some concrete examples: If Apple wants Swift to be successful in the open source world, it needs to at least get basic "passes test" level of support for the BSDs, and some other less common architectures. Look at the architectures supported by the "golang" package in Debian:
Now compare to the list of architectures supported by the Swift package in Debian: ... Oh. There's not one!
If this isn't getting the point across, I don't know what will.
(Side note: There are "official" Swift packages for Ubuntu, even if they're not upstreamed, but they only exist for amd64 and arm64. I know there is "community" support for riscv, which is pretty cool. Still... This is definitely a problem for Swift use in the open source community. Without Swift packages in Debian upstream, for example, I don't even think software based on Swift can be packaged for upstream inclusion.)
I'm particularly interested in this line of thinking, and I'd probably lean the same way. Would love to hear more, why do you think it's a better approach to default to more high-level, with the ability to drop down into manual memory management for hot paths?
Simply because hot paths are less common for a "typical" workload. So you'd want it to be more ergonomic by default. Where possible, jam all optimizations into the compiler so nobody has to think about it. If you can't do that, then allow developers to give the compiler more information even if it's more complex.
Relatedly, it allows the language to help you "first make it work, then make it good".
From an aestethic pov, swift is taking the exact opposite direction than go: it keeps piling on more language features, with much less concerns on keeping the language simple.
As a developer currently working in both, it feels like jumping to another civilisation every time i switch.
I'd say that Swift is more of a general competitor to the likes of C# and Java. The lack of JIT-compiling VM is the obvious difference, but if you set that aside and look at the language itself, I'd say that it's on a similar level of abstraction - a fair bit higher than either Rust or C++.
Definitely! There's only one enormous issue with this outlook: the rest of the world is bigger. Much bigger. Exactly how much bigger depends on who you ask, but it's a lot bigger. Just on the consumer side...
What about servers? Network services and SaaS are a huge industry, of course. Admittedly, I can't even find a good source for what marketshare macOS has: most "server marketshare" counts seem to exclude it or just merge it in with "UNIX-likes". The number appears to be extremely small. There's definitely some exceptions; I think Imgix has been deploying Macs to the datacenter. However, in order to do so, they essentially had to build custom rack mount equipment for desktop machines... I think it's very safe to say that macOS deployments for services are extremely in the minority, and the majority of macOS data center deployments aren't even for general network services specifically but mainly just for CI.
OTOH, macOS Server and any "server" SKUs of Macs have been discontinued as far as I know.
"Rust isn't a competitor to Swift" is not the zinger in Apple's favor that you think it to be. In fact, Apple would be best to support making Rust work better on Apple platforms if it can't support making Swift work better on other platforms in a much more significant way than it has in the past few years.
And this all makes sense. Apple developers had been extolling the virtues of Objective C for a long time, including how it is superior to C++. I'm not even arguing for or against that PoV, but Objective C had at least pretty good compiler support on platforms outside of macOS and it did nothing for its adoption. The main place you'd see Objective C on other platforms were in software originally written for macOS, like `unar` on Linux. That's mainly it...
Unless you want Swift to fail in the long term, I think this PoV is a very badly losing position.
Your argument seems to contradict itself. On the one hand you say “objective c failed because it was available on multiple platforms, but no one cared” but you also infer “Apple is failing because it hasn’t made Swift available on multiple platforms”.
Is it possible that it just doesn’t matter (for Apple) whether Swift is Apple ecosystem only?
It makes me sad, because I really like Swift. Not enough to become an Apple dev though.
I'm saying Objective C was further along in some regards and it still wasn't enough; the investment that Apple would need to make to make Swift popular across platforms is very non-trivial, and just simply nowhere near where they're at today. That's basically it. They're doing some things with Swift that they never did for Objective C, like making a cross-platform version of Foundation. That helps... but it's not good to start out behind on other aspects.
I'm sure they invested tons into devrel on macOS and that family of operating systems. It's weird to think that they could get away with doing less on other platforms where they're already on a much more competitive and less favorable landscape and still have it work out.
Maybe accounting-wise, it's difficult to justify putting all of this effort into something that will ultimately make it easier to leave your ecosystem. That's fair enough but that makes Swift's cross-platform strategy broken.
Ultimately, they're competing with programming languages that have the level of support they do not, but as far as I can tell they're treating it as more of a tool for developers who are primarily Apple-oriented but wish to deploy software across platforms.
What you are saying makes a good deal of sense. It’s a shame though, as Swift seems a really great language with an interesting value proposition. If it were properly cross platform, you could see it being a real competitor to Rust in luring away C++ devs as opposed to traditional GC (deterministic destruction! Can do RAII! Better handling of real-time constraints! Easy learning curve!). Ahh well.
I know this might be opening a can of worms, but technically doesn’t Swift do reference counting? Which is faster than “mark and sweep” GC but will never compare to manual management (like C/C++ and rust ownership)
Reference counting is not faster than tracing GC for general purpose workloads, at least not without a lot of extensive tuning that includes stuff like using tracing and moving GC for the young generation (prior to the first collection), deferring reference count increment/decrements using write barriers, limiting reference count bits (and reclaiming stuck objects with some variant of full heap tracing), and so on. Many of these benefits cannot be effectively realized if you insist on prompt execution of finalizers, either. Languages like Rust and C++ gain their performance benefits not from using reference counting or immediate reclamation, but from avoiding creating garbage in the first place by stack allocating most stuff, creating large objects by-value and within vectors to reduce pointer chasing, using direct interior references into data structures in ways that it would be difficult to trace efficiently, and using data structures like arenas to emulate the benefits of generational GC in controlled contexts where a maximum lifetime can be statically determined.
My personal feeling is that it's very hard to directly compare manual memory management and GC from a performance perspective because so many of the techniques that make tracing fast involve moving pointers, which is very costly to combine with many of the techniques that make manual memory management fast which rely on values not moving in memory. I think any such solution that is efficient for both kinds of memory will probably require the programmer to have very tight control over when the garbage collector can execute, whether pointers can move at particular times, etc., and the complexity of maintaining that kind of control (and getting it right!) seems so high that I'm not sure it would ever be usable for general purpose programming.
The Swift compiler does some of the optimizations/techniques you mention Rust does. It won't reference count, when the setup is possible to analyze statically and elide ref count. Swift also has widely used complex value types, which are stored on the stack. It will also even put some reference types on the stack after doing escape analysis etc.
I don't know which benchmark to trust anymore, but for some reason iOS software tends to seemingly overperform comparable ART (GC) code.
If they are very large and implement copy-on-write, sure, parts of them may be on the heap. But they don't need to be on the heap if they escape if they're just returned by copy.
I think you’re missing the context that people are choosing Rust over Swift even in domains like backend web development which should theoretically be a good fit for Swift. And this is largely because the Swift ecosystem outside of Apple is pretty minimal.
And frankly, I have zero confidence that Apple will sufficiently invest in the language outside of their ecosystem. That's just not how Apple operates.
And the fact that it does not have clean and easy C integration.
If you could do direct memory stuff and control the ARC stuff, and easily integrate with C ...
... and the docs were clean, and you could easily use something other than XCode ...
Then Swift would be huge.
It's a neat language but it's just not designed to go beyond Mac.
The entire C++ world would move away in an instant to something clean and flexible. Rust is just too specific for most things, and Swift is just about outside. So sad.
What do you mean about rust being too specific for most things? It seems like a fairly obvious choic for c++ replacement in a new project, the issue is large c++ codebases already exist, and that migration is painful in my experience. Autocxx type support is still largely missing or in work, diplomat seems reasonable, but I haven't had an opportunity to use it for something real yet.
Rust will not replace C++. Everything is thrown out the window, including developer productivity in order to provide the 'zero overhead runtime guarantee' feature. The code ends up quite laborious, and it's just not suitable for many things, and it doesn't play so easily with C.
Most projects do not require what Rust provides at the cost it provides it at.
C++ people really wants a C++ that is clean, modern, parsable, without all the legacy cruft.
Also I think that Rust is a 'V1' version of a borrow checker and I can't wait for newer iterations which I think will be better.
> Most projects do not require what Rust provides at the cost it provides it at.
If this is true, it's because most projects should use a GC'd, memory safe language, not because most projects should use memory unsafe languages. There is little to no place for memory unsafe languages such as C++ for new projects in 2023.
> Also I think that Rust is a 'V1' version of a borrow checker and I can't wait for newer iterations which I think will be better.
This is the literal definition of vaporware, without a specific plan for how "newer iterations" will do it better. People have been searching for silver bullets for years in this space and nobody has really found a way to improve on the balance Rust has found without introducing garbage collection of some kind. I personally don't believe a "better borrow checker" can exist without GC (which is not such a bad thing, by the way--in my experience, C++ and Rust developers tend to be unfairly dismissive of garbage collection).
Most C++ apps are mostly memory safe and that's just fine. It's ridiculous to suggest that this secondary artifact concern is raised to primacy for all projects.
Rust V1 is a vastly complicated and verbose language, slow to make, slow to write, difficult to integrate, which offers really only one glorious feature that's not important to such a degree for most projects. To suggest wanting something better has nothing to do with vapourware.
Hypersensitivity over a compiler might be a sure sign in a cult!
> Most C++ apps are mostly memory safe and that's just fine. It's ridiculous to suggest that this secondary artifact concern is raised to primacy for all projects.
It's demonstrably not "just fine", as the steady stream of security issues that hurt real people will attest to. It's also important to remember that C++ is an extreme outlier here: the majority of programs written in 2023 are in memory-safe languages. Memory safety is not some weird new thing; it's the norm everywhere but in C and C++, usage of which declines every year.
I think Jasmer has a good point about c++ users wanting a safe enforced subset. I jave suggested a cpp23{} to denote code where the compiler is free to break anything older and enforce best modern practices. Its like a version of safe{} vs unsafe{}
I don't agree that rust is particularly difficult to use, and certainly not verbose. My productivity level is not as high as pytorch or matlab, but it is already on par with c++17 and I've been using that for ~5yrs (c++ for 30yrs) and I've only been using rust for work projects for a year.
Lastly I agree that memory safety isn't a small thing. 70% of security vulnerabilities MS and google find they say are memory safety related. 70% of the financial losses due to security vulnerabilities is... A really big number. And rust, unlike a GC language like java, go, swift, or c#, isn't just memory safe, the borrow checker adds thread safety and more generally safe access to any resource, like file or network io. That is a huge benefit in any domain, and the cognitive overhead just isn't that high.
Swift is memory safe. It’s the better C++ I want, just out of reach because Apple. There is at least one memory safe alternative to Rust that doesn’t require a GC.
I’ve made this argument before but here we go again. GC is great and works better and faster than reference counting in most cases, but there are degenerate cases and they matter on phones. One degenerate case is memory pressure. A GC running out of memory will perform terribly. Another case is bounded latency. Most obvious with hard real-time, but also a problem for any system where a long GC pause will lead to failure modes or user dissatisfaction. RC allows for predictable (albeit slower) performance.
So if we can agree that “no GC” is preferable in a constrained environment like a phone, but you still want memory safety, then you are left with “is Swift or Rust better for productivity?” It’s pretty obvious that the answer is resoundingly “Swift”.
Ubiquitous ARC has even worse performance than tracing GC. Only Rust gives you complete memory safety with fine-grained control over memory management strategy.
Swift has some tricks to make things faster than naive ARC, but the point is that you trade a little speed for deterministic behavior. The post I was replying to was making the argument I see frequently. That if not Rust, you should go GC. This is simply wrong. The degenerate GC cases can be avoided with Swift. It’s better for a phone application in nearly every case. If it wasn’t bound to Apple, I’d say it’d be a better choice than Rust in 90% of the places Rust is used.
Rust has its place, but the idea that it’s the obvious and only replacement for C++ is overblown. A language like Swift seems like a better path forward.
None of the other languages you're thinking of are particularly usable as industrial strength languages for general purpose programming, for reasons that go way beyond mindshare.
I don't need to be a mind reader because I know which languages other than Rust support sound affine types. They broadly fall into three categories:
* Languages with comprehensive support for affine types, but that are otherwise too limited to be really useful for general purpose programming. Usually this is due to a lack of industry support.
* Languages with bolted on support for affine types where you can't get consistent performance benefits out of code written in that style and it's very difficult to write and compose code written in that style.
* Complex languages with advanced type systems that can emulate affine types, and are extremely powerful, but are borderline unusable for any purpose.
Like I said, whichever of those three language categories you're thinking of, my reasoning applies. No mind reading necessary! I understand though that you believe safe concurrency has been solved since Modula 3 because you can just lock every object while it's in use (why didn't anyone else think of that?), so maybe you don't really appreciate the vast gulf in usability between affine types in these kinds of languages and in Rust.
I never said such thing about Modula-3, but I do believe indeed that Rust doesn't solve the concurrency problem when accessing external resources in distributed computing.
It is nice it has an answer for data concurrency for internal in-memory data structures, that is a tiny slice of solving distributed computing access patterns.
Like those languages that you "know", Rust's place is being specialized on deployments where no form of automatic memory management is allowed.
Nice goal post moving. So which imaginary language with affine types are you referring to? Being specialized doesn't automatically mean being industrial strength or usable.
I think for Rust to truly replace C++, it needs a better dynamic linking story. I wish they would make their ABI stable within a given edition. That would make it much easier for people to rely on dynamic linking, similar to how C++ does it. Without it, Rust binaries are quite huge.
The stable ABI in Rust is called the C ABI. Which means you can write C compatible dylibs/.so in Rust that can be used by any language with a C FFI. This is used in the real world for seamless rewrites of existing dynamic libraries in Rust - librsvg being a well-known case. You don't need all the hacks OP is talking about.
Sure, dropping down to a C interface is easy enough, but far from ideal. So much of Rust's expressive power is lost by doing so. What would be the downside of providing a stable ABI for an edition? It seems to me it resolves both sides of the argument on stable ABIs. By the way, I'm mostly thinking about Rust -> Rust dynamic linking.
I think that is a good idea, but dynamic linking seems to be reserved for system level stuff like glibc. When even embedded arm7 targets are shipping as comtainers, what difference does it make if we use dynamic or static linking within that container? The dynamic libraries aren't getting used by multiple executables anyway. The main case I see is to protect yourself from a copy-left license impacting all the bsd and mit stuff from crates.
I've found Swift->C to interop very well with Swift. Yes, you get absolutely buried in UnsafeXXX wrapped types but it does do exactly what you expected. Going the other way is somewhat "herebe dragons" since @_cdecl is technically non-standardized (though there is a movement to standardize it [1]) but it definitely works fine.
After using Rust for my own personal tryouts, I liked it more than expected, and though I see the value in the borrow-checker don't work in areas where it's warranted.
In most of my uses, I would be fine always using Rc/Arc which would make Rust equivalent to using Swift.
As a language though, I much prefer Rust to Swift.
If reference counting was faster, Java would do it. It’s better for UIs and lower latency, but a modern, generational GC can beat C in throughput (because allocations get a lot faster).
That is the whole point of Valhala, which is a big engineering effort to achieve keeping running those old world JARs without doing a Python 2/3, we already have enough with Java 8 stagnation.
The two biggest languages I'm aware of in that space are Java and C++. I don't think I've heard of any shop using C. Java seems to be keeping up with C++ there as far as I know. Do you have evidence to the contrary?
I don’t think your article makes the claim Java keeps up with C++. That article seems to be about getting latency low and consistent so Java can compete with C++.
There is a long tail of other languages. JaneStreet uses Ocaml. I wouldn't be surprised if there was some usage of Go out there. I'm familiar with at least one trading house that uses Elixir. The bulk of it is Java or C++ though.
GC has worse p99 latency, even if it may have better throughout.
Also, with manual memory management, you can do all your allocs and frees on other threads that are not executing your latency sensitive logic.
With GC, you have to at least execute a stack scan on every thread, and possibly a few extra stw passes. The highest performance gcs also have a lot of peanut butter costs from read barriers.
It’s not faster in the normal case, it’s generally quite a bit slower. It is more predictable however (you can actually use it with real-time constraints!), and performs better under memory pressure. Whether this is a net benefit depends on what hardware you’re running on and what the app is supposed to be doing.
That's certainly a reason, but Go doesn't have the adoption issues that Swift has despite Google's reputation. Go's GC may be fast, and on some workloads maybe even competitive against ARC - But, I really don't think that has anything to do with why Swift isn't seeing widespread adoption outside of the Apple ecosystem.
Here's my personal opinion for why. Bear with me for a second.
I just searched "Swift programming language" on my search engine and the first result looks promising:
OK. It's a language by Apple, so it's not surprising that it's on Apple's website, even though I think if you want people to get the idea that it's a serious independent project you probably should give it its own top-level site personally. (edit: Yes, I know it does. This is my honest first impression. Please keep reading.)
> Swift is a powerful and intuitive programming language for iOS, iPadOS, macOS, tvOS, and watchOS.
Alright, we're only talking about Apple platforms. Technically it doesn't say that it can't work outside of these OSes, but it's clear that Apple is not marketing Swift as a general purpose operating system. From Apple's marketing PoV, it's an operating system that is made for Apple platforms and anything else is incidental.
> Download Xcode and learn how to build apps using Swift with documentation and sample code.
No download links for users of other operating systems. But if you go to that link, all the way at the bottom, you get:
> Swift is developed in the open. To learn more about the open source Swift project and community, visit Swift.org.
Finally, you might notice that Swift is actually available outside of macOS. It's available for Windows and Linux, but not BSD as far as I can ascertain. The platform support of Swift is excellent when it comes to Apple devices, but it lags far behind other options when it comes to other platforms and architectures.
My perception is that Apple hasn't really made a serious enough investment into making Swift a general purpose programming language outside of its own ecosystem. That's honestly fair, as doing so is no doubt extremely expensive and time-consuming, and it's an on-going investment.
What they're doing now appears to kick it to the community, who has done a lot, including as far as I can tell, make Swift work on RISC-V. But Apple is literally the world's biggest technology company, so it's not exactly desirable to put a bunch of community effort into giving it the on-going support that it needs to feel "first-class" outside of Apple platforms.
Even just the optics here are quite bad, and I think the fact that it's still like this is going to haunt the long-term prospects of Swift outside of Apple platforms.
All of this is aside from its technical merits. Normally, I'd like to stick to purely just technical merits, but unfortunately this stuff does matter. It's part of the reason why Go is still a solid choice today even though Rust has a lot of technical advantages on paper.
Apple's goal may be is to create a SW ecosystem that mostly only works on Macs. If a Swift application could be easily compiled to run on Windows, it would be much easier for Apple users to switch to Windows or Linux.
And if Swift apps would run happily on all platforms Apple would still have an incentive to make them run fastest on Macs. Therefore it is not a good idea to rely on a computer manufacturer to provide you with a programming language.
Microsoft has plenty of conflicts of interest to contend with, but that hasn't stopped them from being able to make some inroads towards appealing to other platforms. Perhaps it would be unwise to trust that they've "changed" and invest in Microsoft tools or ecosystems, but it's beside the point: I think it's fair to say that Visual Studio Code and TypeScript have achieved widespread success across the board.
I think that Apple could still make Swift a success outside of their own ecosystem if they wanted to. It would probably have a lot of benefits for them to do so too, but they're all pretty long-game gambles (not to mention quite large investments, too.)
Yeah, but if you want top tooling for .NET, the answer is still pretty much Visual Studio on Windows.
The cross platform version of .NET and VS4Mac/VSCode only support a subset of the whole development experience.
Many libraries on the ecosystem are still stuck in .NET Framework, wrappers for COM libraries, or plain Windows APIs, given its Windows focus between 2001 - 2016.
If you want a Visual Studio like experience without everything that is still missing from VS4Mac/VSCode then you need to shell out for a JetBrains Rider license.
Mostly because of the Windows/Visual Studio interests versus Azure folks interests.
Exactly! Microsoft has basically been stifling .NET adoption across platforms. The community reception was really poor when they took away live reload from the CLI, and it was equally bad when they restricted debugging functionality to only the Visual Studio of family products. These things have contributed to fairly limited adoption of .NET outside of the Microsoft ecosystem.
What's crazy is that in .NET's case, Microsoft has it a lot more "free" because the community is willing to go great lengths to just do it themselves. Like for example, the entire Mono ecosystem: MonoDevelop, Xamarin, Unity, etc. has proven quite successful. Now that Roslyn is open source, it should easily overtake Mono... but it hasn't really.
.NET is basically a cautionary tale. It's pretty amusing that the same company that made TypeScript and VSCode popular across platforms is having strategic trouble doing the same for their wildly sought after CLR/.NET ecosystem.
> OK. It's a language by Apple, so it's not surprising that it's on Apple's website, even though I think if you want people to get the idea that it's a serious independent project you probably should give it its own top-level site personally.
I did. But why would you include a false claim only to contradict yourself? And when, at least for me in Google, swift.org was the top link. It wasn't like it was hard to find. People don't need to go to the bottom of Apple's page on Swift to discover it.
Just to be sure, I Google'd "Swift programming language" again in incognito mode and the first result is still https://developer.apple.com/swift/. Swift.org is listed, but it's so far down the page that I'd have to scroll.
It wasn't really a contradiction, I was giving you my honest first impressions as a not-Swift user. I had no reason to scroll down the result page when the first result is clearly the same Swift I'm looking for. Why wouldn't the first SEO'd result/marketing page be the right thing? I don't have this problem searching for "Go programming language" or "Rust programming language", those both point to their official top-level sites on both Duck Duck Go and Google.
Swift will always have to contend with this problem.
No doubt people are busy using Swift, but Apple technology advocates are about as "religious advocates" as they come. And frankly, I'm not trying to say that as some kind of negative. It can be annoying, but clearly Apple users are genuine enthusiasts, even if I can't really see eye-to-eye with them. I do think that this genuine enthusiasm creates some distorted images of the reality of the world, though. The thing is, Apple users are likely so used to having more enthusiasm than competing communities of higher marketshare products and services that it's not surprising at all that the perception is that Rust is all hype. I'm more a Go coder than Rust by SLoC, and Rust has to be one of the most enthusiastically supported languages I've ever seen.
Still, it'd be silly to discredit Rust on the basis of its enthusiastic userbase: it's successful by any measure. It's invading the Linux kernel, the campuses of Amazon, Google, Microsoft, it's going to be in both Firefox and Chromium, and that's really just the beginning. Even if Swift had more collective SLoC than Rust, Rust is clearly on its way to solidifying its place as one of the most impactful programming languages of the past 20 years. That impact is even going to be felt in Swift: when talking about Swift, it is very common to advertise it in terms of Rust. I mean hell, that's basically what this blog post title does: "Look, Swift did something even Rust couldn't!" -- If Rust wasn't so influential, I do not think anyone would bother to compare the two in this way.
All that says is that people decided to downvote the comment. I honestly think that's mainly because people feel it is an unfair or flippant characterization; obviously, plenty of people DO write Rust. Even despite the position Apple is able to hold Swift in with iOS, that has not stopped Rust from being similarly popular among professional developers; in Stack Overflow's last survey, double the professional respondents claimed to use Rust than Swift:
Even if you consider some of the respondents to be lying on purpose to skew the stats, I'm of a mind to believe that failing any real incentive, a majority of the correspondents are likely telling the truth. (The numbers don't show anything too wild, after all: JS on top, and C/C++ having a commanding lead on "modern" alternatives.)
Believe me though, some of my comments in this thread also got downvoted before recovering. The sad thing is, I don't really harbor any ill intent, I'm being blunt because I genuinely want things to be better; after all, if Swift is so good for developers, then I definitely would like to be able to consider it an option, especially considering that Rust is, frankly, a pain in the ass to code in sometimes. Today I do not, and what I need is a louder and clearer signal from Apple, not assurance from a community.
All in all, I am not really disagreeing that Rust has an extremely enthusiastic userbase. But what I am definitely disagreeing with is the idea that it's just hype. From my PoV, I 100% understand the value proposition of Rust. As an alternative to C and C++ it's extremely compelling, and in practice it's being integrated into large and complex applications like web browsers today in core components like the rendering engine, video codecs, etc.
As a closing note, to see why Rust and Swift are different ball games entirely, I feel you only need to ask just one simple question: "Why can't Swift serve the purpose Rust is serving in the Linux kernel, Firefox, Chromium, etc?"—each one has its own answers: insufficient control over memory allocations, interoperability issues, missing support for platforms or CPU architectures, lack of alternative toolchains (believe it or not, both Go and Rust have multiple complete toolchains; see gccgo, rust-gcc, mrustc, and even more if you count less complete implementations), and probably more things that I lack knowledge of. Finally, Rust just simply has more outreach: I can install it from my distribution's own packaging in virtually any Linux distro, it supports a lot of architectures and platforms even if they're not all tier-1, the rustup and cargo tools make it extremely fast and easy to get started, and more. Apple could totally close the gap here, but until they do, I don't feel like investing time into this language is a good idea for me personally. Like I said, I genuinely consider this to be a shame.
I have heard great things about Swift but haven't spent a second looking into it because I assumed it's an Apple exclusive language - making it unsuitable for my use case of services deployed on Linux, developed from any OS.
To what extent, if any, is Swift exclusive to Apple devices?
It's Apple-exclusive in the same way that C# is Windows-exclusive or Go is Linux-exclusive.
You'll have a much better time programming C# on Windows or Swift on macOS, because that's where all the best tooling is. The tools and libraries are open-source, cross-platform, and mostly maintained by one company.
That's really not the case with C# or Go. C# has first class Linux support nowadays (minus GUI support) and using Go on windows is quite easy (I daily drive go on windows at work).
With Maui, you can now have first class Linux gui support, same with Blazor. You can also even do some of the more cutting edge stuff on linux like compiling straight to ASM.
Wow, you're right. I stand corrected. I've been leveraging it for Windows and Macos, and incorrectly assumed it would work on Linux as well. And when I did target linux it was using the Blazor(Webview stuff). Thanks for the correction, such a weird decision for Microsoft to market it as cross platform so heavily and omit Linux. I would definitely put it on Lie tier. Even more strange since some other features have first class linux support like compiling to native code, but others like Blazor apparently omit only linux.
No Maui doesn't support Linux desktop and Microsoft engineers have specifically acknowledged this and so far there are no plan to support Linux desktop.
Not having GUI support is definitly not first class.
Secondly many libraries are missing as a large portion of the ecosystem is still focused on Windows, and VSCode is a tiny portion of Visual Studio features.
One needs to buy a Java based IDE (Rider) for good support doing C# development on Linux.
Swift is backed by LLVM and Linux solid is pretty solid. The cross platform problem has always been the core libraries. Fortunately a pure Swift core foundation rewrite is underway. https://www.swift.org/blog/future-of-foundation/
Looks like I may eventually give Swift a try once the rewrite is done! I've been watching Swift for a while now but I don't have a Mac so I haven't had a reason to try it.
Do you happen to have any idea about UI library support? Will UI still be Apple only?
That’s my understanding of why it’s on Linux (besides the community).
Apple uses tons of Linux servers, cloud and DC. Swift means they can write high performance networking stuff with memory safety and modern features in a language/library they already use internally instead of using yet another new language (or losing safety with C/C++/Obj-C).
Most of the Linux machines inside Apple are tucked away in data centers, they're not sitting on engineers' desks (there's always exceptions). The people writing stuff that will run on Linux are usually writing in languages that are already cross platform. The stuff running on Linux inside Apple has a pretty good chance of being Java or Python (or Ruby for some teams). These languages are usually chosen for their ecosystem.
There's not much of a Swift ecosystem except the proprietary one on macOS and iOS. Without all the frameworks available on macOS there's not a huge Swift ecosystem to leverage.
As far as I know all of the UI stuff is dependant on iOS and macOS specific libraries. Apples UIKit and AppKit libraries are still proprietary here (and even have completely different capabilities depending on your platform and OS version).
I wish fast.ai would have succeeded with Swift. The world desperately needs the ML community to move on from Python and towards a fast, statically typed language with truly concurrent multi threading.
I am honestly curious what kind of content you must be consuming to come to that sort of conclusion. Like, is there a secret “anti-dynamic linking league” somewhere who pushes people to think it’s the literal spawn of Satan?
I never got the anti-dynamic linking crowd, being old enough that static linking was the only option (minus hacks like overlays), I don't miss those days.
My very vague understanding of Swift’s dynamic linking is that it works but comes with a huge performance penalty. To the point of it becoming unusable for high-performance apps.
The article doesn’t have any performance metrics. It mentions surprising performance cliffs but I couldn’t find details. It says there is a perf cost you can opt out of but doesn’t detail the cost or what you lose by opting out.
It sure seems from the article that there are a lot of negative performance implications. Which totally be a price worth paying! But I would not agree that the article explains what that price actually is.
The biggest problem is that if you have a big """zero-cost-abstraction""" blob like iterator adaptors -- `Map<Filter<Fold<ArrayIter<MyType>>>>` -- and a single drop of Resilient Type is in there (i.e. MyType is resilient) then the whole thing gets polymorphically compiled and the compiler won't boil away any of the things that are supposed to be "zero cost".
How much you get burned by this kind of thing really depends on how you design APIs and where the hotspots are. Like if the big iterator blob is only ever for like 10 items, whatever. If the iterator blob is iterated inside the dylib where it's not resilient and can be inlined away, whatever.
And I kept stumbling upon these witness tables while debugging.
This led me to find this write-up and the much more interesting story behind it.
The end note was priceless ^_^
collapses