Blog

What makes uv slow?

2026-03-12T00:00:00+00:00

A lot has been said about uv being fast, but not about how and when uv is slow. Let's talk about both.

The paragraphs are in no particular order, except the first two (overview and benchmarks) and the last one (tooling).

Overview</a></h2>
We get a lot of questions if there's some sort of hidden magic, if there are shortcuts we're taking, but in practice it's trying optimizations that worked well for others as well as a loop of profile</a>, poke at the code</a>, benchmark</a>, PR, repeat.
If you're mainly interested in the resolver algorithm, head over to the uv internals docs</a>, the pubgrub introduction</a> and the pubgrub-rs guide</a>. They cover the subject in-depth, i'll only touch briefly on them here.
In general, what's slow is always roughly the same, from high-level to low-level:

Invoking external (build) tools</li>
Network requests</li>
cpu heavy code, such as (de)compression, hashing, parsing and constraint solving</li>
Disk reads and writes</li>
Singlethreaded and synchronous code</li>
Allocating, memory copies and syscalls</li> </ul>
Optimizations happen roughly in that order: For example, transition from source distributions to wheels</a> removed the need to invoke build tools each time, then caching wheels reduced network IO massively, an unzipped cache built reduced the decompression burden, reflinking/hardlinking turns file read/writes into link read/writes. Each of those is sped up proportionally by multithreading and IO concurrency, while avoiding allocations, memoization, faster hashmaps and similar local optimizations remove remaining bottlenecks throughout. uv uses existing heavily optimized implementations of staple cpu-heavy algorithms ((de)compression, hashing, http parsing, etc.)1</a>; classic "Big O" optimizations are rare. The one exception is pubgrub, which exists as a library outside uv.
Benchmarks</a></h2>
Throughout this article, i'm using three main benchmarks: airflow, boto3 and homeassistant.
Apache airflow</a> is a massive workspace with 126 `apache-*</code> packages and 892 total packages in a resolved workspace. Resolving its dependencies requires some backtracking and includes some source distributions. It's useful as the biggest real example that's freely available.`
boto3</a> is the official interface for AWS. It's one of the most downloaded python packages, and creates a new release every time AWS changes. boto3 depends on botocore, which depends on a recent urllib3 version. When resolving with an old urllib3 version, such as urllib3<1.25.4 boto3</code>, the resolver needs to do a lot of backtracking through boto3 and botocore versions, to find a boto3 version that depends on a botocore version that is compatible with urllib3<1.25.4</code>. This is something that happens to users when they depend on one package that can use data stored on S3 and another one that requires a specific urllib3 range, so we adopted it as backtracking and parallel networking benchmark2</a>.
`Homeassistant</a> is a home automation platform that can connect to about every smart home there is. Supporting all those diverse protocols requires a large number of dependencies</a>, and install performance matters a lot</a>. Despite the large number of dependencies, it's not particularly interesting as a resolution, the dependency tree is rather flat and heavily constrained</a>.`
`If you have cases that are particularly slow except for source distributions, please let me know!`
`To give you a sense of the numbers, let's compare a linux desktop (fiber), a raspberry pi 4B (fiber) and a windows laptop (wifi).`
`A cold cache uv sync</code> in airflow takes 54s on my desktop and 10.5min on the raspberry pi. It's dominated by building two large packages, art of that are two parallel builds (krb5 and gssapi) that take 44s and 22s respectively, the bottleneck for the whole process. The same operation on the raspberry pi takes 10.5 min. After the installation, uv run python -V</code> takes 171ms on my desktop and 1.1s on the raspberry pi.`
Plotly is ideal as small installation speed benchmark: It's a popular package with a lot of files, but only two dependencies, all three with platform-independent builds. Installing plotly with a cold cache takes 310ms on the desktop, 2.3s on the raspberry pi and 4.8s on the windows laptop. With a warm cache, we get to skip the first phase with network requests and the download, and it goes down to 40ms for the dektop, 120ms on the raspberry pi and 900ms on the windows laptop. Windows is slow due NTFS, on a ReFS dev drive it takes only 450ms. 3</a>
Toml parsing</a></h2> Toml is a file format that's easy to write, doesn't have a lot of complexity and is compatible with a lot of json tooling such as jsonschema. uv chose toml for uv.lock</code> (just like poetry.lock</code> and pylock.toml</code>) for being user readable, and more importantly, generating good diffs when updating dependencies. Those are important for auditing dependency changes, such as through dependabot or renovate. 4</a> While json parsers can do 1GB/s parsing on a 10 year old laptop</a>, rust's toml parser unfortunately is slow</a> 5</a>. For a large project such as apache airflow6</a>, it takes uv ~30ms just to read the lockfile. For comparison, print("hello world :3")</code> is 10-15ms7</a>. </a> Figure: uv run python -V</code> in a airflow fully installed airflow checkout. The x-axis is time, each blue bar represents a span of the kind on left running. In this case, the top bar is parsing uv.lock</code>, while the many small bars are parsing the workspace member pyproject.toml</code> files to decide whether the lock needs to be updates (satisfies). Click the image for a (large) svg with interactive tooltips showing the path for each parsed file. Obviously, you should also avoid parsing pyproject.toml</code> 131 times</a>. But even if you parse every pyproject.toml</code> only once, large workspaces such as airflow have an overhead: uv run</code> needs to check whether the lockfile is fresh or whether we need to resolve and install change dependencies, and that needs to look at each local package and parse its pyproject.toml</code>. Non-local dependencies either have static metadata (registry dependencies), are only updated on demand (git dependencies) or follow refreshing rules (http dependencies). For airflow, it takes ~20ms just to check whether the lockfile is satisfied, almost entirely for workspace member parsing (the "satisfies" bar in the plot above). For registry, git and http (link to a file) dependencies, uv stores the metadata for registry dependencies in a zero-copy format</a>. With rkyv it only needs to do the minimal validations against undefined behavior and pointer rewriting, the data on disk is in the same format as the in-memory representation, no parsing involved. Loading those is fast (though not entirely memory efficient at the moment) in cases in which uv does need to resolve, and not only to validate. Version representation</a></h2> Surprisingly, a big bottleneck is the type used for representing package versions. As the index page for a package gives us all versions of that package, and all clauses in the resolver use versions as literals, we create and copy a lot of versions, so much that with a naive version type, benchmarks are dominated by version operations. This isn't only the case for uv, but also a bottleneck for pip and packaging</a>. uv uses an optimized version parser and small version representation</a>. The version type is internally an enum (tagged union)8</a> of a small type for a subset of versions that fits into 8 plus 8 bytes 9</a> and a large, heap allocated type that can represent any version. The small type can represent notably represent three part versions with components <255 and an optional prerelease. Its bytes are ordered in the same way that versions are and copying it is a 16 byte copy. The large variant supports the full feature set of PEP 440</a>, some of which require separate heap allocations, such as the strings in a local version. In the version enum, the large version is wrapped in an Arc</code>, rust's basic reference counting type, which itself is only a point to the heap allocation of reference count and payload. Copying Arc</code>s is generally fast, it's an atomic number increase, but can become notable at resolver-scale. Resolutions that include packages with a huge number of releases that don't fit into the small version, such as many too large dev releases, make uv slow. For example, concluding and reporting that tensorflow-io-nightly</code> can't be used with python 3.14 takes 500ms, as this package has a huge number of releases that all have too large dev versions. (Finding a valid solution on python 3.10 takes 40ms.) Removing the small version optimization makes resolving boto3 2.1x slower (490ms -> 1030ms). Additionally replacing the Arc</code> in the full version with a Box</code>, this becomes 4.7x (490ms -> 2270ms). Notably, removing the Arc</code> optimization without removing the small version has almost no effect: Almost all versions fit into the small variant. </object> </object> </object> Figure: A performance ablation of uv resolving boto3. The first flamegraph is the optimized version. In the second, the small version type is disabled and the version comparisons become notable. In the third once, the large version doesn't use an Arc</code> anymore on top of the small version being, and version cloning and dropping starts to dominate. Note that they while they have the same width, they represent increasing amounts of time. TODO: Cut to 1% frames Figure: The samply inverted stack trace of resolving boto3 without the small version optimization in uv's version implementation, uv_pep440</code>. It shows the functions that take the most time on their own (flamegraph leaves), rather than the total time of each. TODO: EXAMPLES and DATA for the outlier from the ecosystem test TODO: samply plot from the ecosystem test Another important packaging type are platform markers, whose arbitrary boolean expressions uv represents with algebraic decision diagrams</a>. Those are technically an SMT solver, parallel to pubgrub being a SAT solver. This is one of the few locations where with a classic algorithms and data structures optimization. The marker representation is canonical, deterministic and allows queries such as whether one marker expression is a subset of another. Missing wheels and dynamic metadata</a></h2> Python packages come in two forms, source distributions and (built) wheels. If any wheel exists, uv can use the dependency information from the wheel during resolution, which is always statically encoded 10</a>^,11</a>. When there's no wheel, modern source distributions (PEP 643</a>, core metadata v2.2) can specify static dependency information. If they don't, or they use a too old core metadata version, getting this information from a source distribution is a complex process that requires downloading the entire source distribution, setting up a separate venv with an independently resolved set of build dependencies and invoking python hooks. This alone can be slower than an entire resolution with static metadata. If the source distribution requires building native code, it's even slower. During resolution, uv may have to do this process for several versions of a package as it backtracks through finding a suitable version. During installation, it can use cached builds, but installing on a cold cache system needs to build all missing wheels, which is several times to magnitudes slower than the wheel installation process. TODO: An example is resolving airflow, ... Missing index protocols</a></h2> Another potential reason for it being slow is that the registry12</a> doesn't implement PEP 658</a> (Static Distribution Metadata in the Simple Repository API) nor supports http range requests</a>, an old trick</a> that allows reading the metadata file from a wheel without downloading the whole wheel. When both are missing, uv has to download the whole file for each package, and in the bad case that we're backtracking on torch, we download hundreds of megabytes repeatedly. Compression</a></h2> Wheels are zip-compressed with default DEFLATE algorithm, while source distributions are gzip compressed tar-archives. DEFLATE (1990) and gzip (1992) both produce larger files and with slower decompression than modern compressions such as zstandard (2015). While uv uses a streaming zip</a> and tar</a> decoder to combine downloading and unzipping into the cache, at least the larger download sizes are notable. The slower decompression may not be noticeable as it's usually still faster than the download itself, but leads to unnecessarily high cpu load. Hashing</a></h2> uv needs to check the hashes of all artifacts it downloads, to ensure they haven't been manipulated13</a>. Using SHA-256, this is notably slow. While hashing is by design slow, a different hashing algorithms such as blake3 would be faster. uv currently follows SHA-256 being the ecosystem standard and widely available from registries. Virtual environments</a></h2> Installing python packages involves copying all the packages into a directory for each installation. Even if you reflink them (also known as copy-on-write, supported on the default mac filesystem, some linux filesystems and rarely on windows) or hardlink them, and even if you do this with a thread per package, you still have to walk the whole source tree and copy it. There is no fundamental reason why python can't use packages from a centralized store, as other ecosystems do. </a> Figure: uv pip install transformers[torch]</code> into a fresh venv from a warm cache on linux (hardlinking). Each blue bar in the link step is a package, where the long 70ms bar is installing torch. "solve" is the dependency resolution step, each "link_wheel_files" us installing a package from the cache. The source svg with more details is linked. TODO: use a uv sync</code> plot, prune it and display the svg All those packages in the cache are already unpacked and in the correct general shape for python to import, the installation process largely duplicates the existing tree. uv recently switched to trying reflinking by default on linux</a>, which is faster on the filesystems that support it (many servers and CI machines use ZFS or XFS), unless it's a filesystem where somehow reflinking is slower than hardlinking</a>. There isn't a known way to detect which methods are supported without trying them, nor to know which one is the fastest. Accepting venvs, uv is comparatively slower to bun building node_modules</code> inside of a package. Backtracking</a></h2> uv uses pubgrub for dependency resolution. Checkout this page</a> and the linked pubgrub docs on how uv's solver works - it's too complex to repeat here. The talk uv: an extremely fast package manager</a> also has a great explanation, including some of the algorithmic and implementation optimizations. Pubgrub is responsible when uv is fast during complex backtracking scenarios and also when it gives helpful error messages about failed resolutions14</a>. In a basic pubgrub implementation, once you choose a package, you try versions from most preferred to least preferred (newest to oldest), only discarding are version if no solution with it is possible. A bad decision early on can lead to trying a lot of versions, only to fail at the same conflict over and over again. uv extends the basic pubgrub algorithm to record which packages are involved in a conflict, and when two packages cause too much backtracking, switch their priorities and manually backtrack before both of them, allowing pubgrub to explore a different part of the solution space. There is a detailed write-up of the problem</a> and the solution</a>. This however doesn't work</a> if the conflict is through an indirection. uv has to be somewhat conservative when it comes package priorities and heuristics, to ensure the robustness of the solution and to keep churn minimal, e.g. from updating a single package. A faster resolver could always pick packages and versions in the order in which it gets metadata, but that would mean running uv pip compile</code> or uv lock</code> twice with no new version could give different results. Instead, uv prioritizes based constraint wideness and the order of discovery, which is deterministic giving the registry doesn't change (or an exclude newer cutoff is set). In the case from above, where there's a missing optimization</a>, we ended up not shipping</a> it. This would have been a behavior change in the resolver, and the library that initially motivated the change had solved the problem with better dependency bounds in the meantime. When resolving the top 15k pypi packages by downloads with uv pip compile</code>, the slowest ones take ~160ms on my machine, without particularly strong outliers. How good or bad dependency structures worked in old tools, most notably pip, has shaped what subset of the theoretically SAT-hard space packages actually occupcy. A resolver can be fast by handling those patterns, and while it's easy to craft stress test examples, i'm not aware of any real-world examples where a warm cache uv resolve is notably slow (If you got any, let me know!). Simple api</a></h2> The repository api for python packaging evolved from user-facing html pages of code archives to download, which started to get scraped by installers. While there's now a proper json interface</a>, it's still fundamentally a list of filenames. That's kinda slow to parse (TODO: numbers). More fundamentally, it means that for each version the resolver tries, it needs to do another request to fetch the dependencies of that version. The information a package manager actually needs is small, in most cases it's only the concatenated contents of project.dependencies</code> and project.optional-dependencies</code>, yet we have to pay a separate http roundtrip each time. On top of that, each response is separate in uv's cache, so even with a warm cache we pay extra for loading the metadata for each version. TODO: perf of parsing a large index page TODO: In uv's cache we save the responses raw, which is much larger than they'd need to be, which gives uv such a big memory footprint, even through rkyv. TODO: boto3 pic Multithreading is hard</a></h2> Scheduling diverse network, IO and CPU loads is hard. It's not intuitive that you need to put the resolver into its own thread</a> even when it's largely IO bound, the resolver's CPU boundness still makes network IO slower. There's also convenience vs. performance: uv likely isn't faster by using async/await than by using threaded IO, but there's a large async ecosystem and async abstractions combine well. There's a hard to quantify tokio scheduling overhead visible in uv. Still, uv is using a mixture of regular futures on tokio, spawn_blocking</code> for CPU intensive work to use the tokio threadpool, a subprocess pool of python subprocesses for bytecode compilation and rayon for parallel unzipping and installation. There's also a lot of file locking and atomic moves involved to ensure that uv processes running in parallel can't break another. On the upside, concurrency is key to uv's performance. We can simulate a single-threaded uv with mostly serial network requests by setting the concurrency limits for network, builds and installs to 1 each15</a>. TODO: Benchmarks of concurrency 1 Prefetching</a></h2> Unreliable network</a></h2> Some environments, such as github actions have (as of writing) unreliable networking. If a download fails mid-request, which it does a noticeable number, uv has to start the download anew as it can't resume downloads (another slowness), or because the request wasn't streaming (only for small requests though, where this matters less). Similarly, if a server replies if intermittent 4xx or 5xx codes, we have to retry and/or wait for the server to recover. Lack of batch refresh</a></h2> If you uv pip install …</code>, uv caches (for pypi) all responses for 10min, following pypi's caching headers, then it has to revalidate. But there's no endpoint to ask whether all packages are fresh, uv needs to send n revalidation requests for n packages. Those ideally all return 304 not modified, but if n is larger than the network parallelism (50 by default), it takes a multiple of the roundtrip time. This applies similarly to uv lock</code>/uv lock</code> with --upgrade</code> or uv add</code>. Forking</a></h2> Platform markers in Python allow TODO: Forking is kinda wasteful, because we clone and rebuild structures, only helped by preferences. It's also not the most efficient inside uv, while most forks reuse the solution of the previous fork through version preferences, each fork clones the entire resolver state. Fat wheels</a></h2> Some wheels need different compiled artifacts for platform differences that can't be expressed in wheel tags. Numpy functions get compiled for different sets of CPU extensions, introduced in different cpu generations, which are merged into one big binary and with runtime dispatch (NEP 38</a>, living docs</a>). Torch similarly compiles its nvidia gpu code for a number of gpu architectures (streaming multiprocessor architecture, not cuda) and merges all of them into a binary that's somewhere between 100MB and 2GB. This is hugely wasteful, as there's no support for selecting only specific wheel variants</a> yet. Projects without the resources of numpy or torch generally build against the oldest supported hardware only (at least for pypi), so uv installs code that's slower than necessary at runtime (or worse, fails on old machines). System and abi dependencies</a></h2> Some packages, such as psycopg2, mysqlclient, flash-attn, a number of scientific packages and a number of hardware integrations, don't ship wheels because they depend on system headers or on the specific version of an already installed library. The most popular packages now ~all ship wheels</a>, but the remaining non-wheel packages are a noted pain for many users, both for performance and for setting up the build environment outside uv. Allocating memory</a></h2> uv puts a lot of effort in uv to avoid unnecessary allocations, such as passing around references as default, using Box</code> to constrain type sizes, using Arc</code> for shared references, using Cow</code> to only allocate if an object was changed and using small string optimizations 16</a>^,17</a>. Contrary to its reputation, rust doesn't default to being good about memory usage and allocations. A modern GC does a lot to offset allocations, and bakes optimizations into the language, and avoiding clones requires ceremony throughout the code. In python as counterexample, all objects are reference counted, while Rust requires changing the type to Arc</code> throughout. Python has a free list of objects that gets reused, and java features a generational garbage collector to deal with short-lived and long-lived objects, while in rust you need to build or bring your own arena. Some types, such as Dist</code>, could also be smaller. In rust, all objects include their children directly by default (not references), and enums have the size of their largest variant recursively. This requires well-placed Box</code> types to keep type sizes in check, including for error types (which determine the size of the entire return type) and for futures. What rust does provide is precise control over allocations and memory layout, allowing to pick the optimal bytes in critical paths. uv for example has assertions about type sizes in critical locations. Rust uses the system allocator by default, which slow, on musl doubly so than on glibc, so uv has to switch to jemalloc and mimalloc</a>. I bet an allocator optimized for Rust instead of C/C++ usage patterns would be a speedup. A location where uv isn't optimized much yet is memory. For example, uv takes 110MB just to say that the lockfile is fresh, with a total of 10.7 GB and 8.8 million allocations for a warm cache uv run</code> in airflow: </a> Figure: A local codspeed memory profiler run for warm cache uv run</code> with the full results linked. On the flipside, the reports about OOM errors we could confirm were caused by source distribution builds, not by uv itself. Freeing memory</a></h2> Rust uses the Drop</code> trait to recursively free datastructures and to run pre-free</code> hooks, such as refcount decreases. To deallocate an object or a collection, drop()</code> on the object or collection first needs to call drop()</code> on all of its members. This can be noticeable, for example when having a large number of Arc</code>'d versions where each drop is an atomic refcount decrease and check if the count is zero to deallocate the object it wraps, especially combined with large datastructures such as the distributions and incompatibilities built up during a resolution. Figure: The samply profile for resolving urllib3<1.25.4 boto3</code> with uv pip compile</code> (10x for better data). boto3 is a backtracking heavy benchmark, that builds up a large database of incompatibilities in the resolver. The highlighted block is a recursive drop after a successful resolution. Command: samply record --rate 5000 --iteration-count 10 --reuse-threads ./uv-main pip compile boto3.in</code>. Tooling</a></h2> This isn't a performance problem, this section is about all the great tools that make these investigations possible. citerion</a> is our benchmark harness and CodSpeed</a> provides our continuous benchmarking. I want to shout them out for being great with feedback an implementing a lot of features we need and often fixing problem same-day. Their web interface even has a flamegraph diff tool18</a>. Below, you can see a small regression from a correctness fix</a> we merged: For local profiling, i use samply</a>, which uses the firefox profiler UI, and for benchmarking, i use hyperfine</a>, which very helpfully includes standard errors, so you can tell what's below the noise threshold and what's real. </a> Figure: A criterion report from a noisy machine, causing the two outliers. Criterion features a t-test, which helps with telling if something is actually faster even for benchmarks with high variance. hyperfine can do t-tests through a separate script</a>. A mature library ecosystem is a requirement for adding a cpu-intensive operation to a packaging standard. For example, a (previous) blocker for zstandard was for example a missing std implementation. ↩</a> </li> While uv is technically a SAT solver, the cases that exist are very specifically shaped, often driven by what earlier tools would support or struggle. It's important to pick realistic examples, rather than artificial or stress test benchmarks that do not represent real workloads. See also the talk Gotta Go Fast</a>. ↩</a> </li> NTFS is mysteriously slow for a lot of things, including uv's own test suit, but i've never found an explanation why. ↩</a> </li> Bun was using a binary lockfile for maximum performance, but migrated to a text-based format</a> as a binary file is tricky to review, merge conflicts are hard to resolve and tooling can't easily read it. ↩</a> </li> The toml and toml_edit crates are optimized on features, such as preserving spans on parsing for excellent error messages and style-and comment-preserving rewrites on files. It's just not a performance audience, most people aren't bound by parsing configuration files. We use both its great error messages for configuration errors and its rewriting features editing commands such as uv add</code>, uv remove</code> and uv version</code>. ↩</a> </li> Benchmarked as of airflow 7fa400745ac7aebc7cc4ec21d3a047e9fb258310</code> on Ubuntu 24.04. You can get some reproducibility with --exclude-newer 2026-03-15</code>, but as new distributions are published, the size of the server responses increases, which slightly changes the numbers. For scale, the warm cache boto3 benchmark that takes 420ms on my desktop machine and 3s on a raspberry pi 4b. ↩</a> </li> As a general note, the numbers in this post aren't overly exact, and sometimes even show some divergence between different runs of the same benchmark. Due to the diverse kinds of resources uv uses (network (pypi, cloudflare CDN, github, other internal and external registries), disk, CPU (singlethreaded), CPU (multithreaded), CPU-accelerated code such as SHA-256, different concurrency frameworks), bottlenecks vary. It doesn't make too much sense to optimize for getting exact low noise numbers on a specific machine. When developing speedups, we focus on determining if something is speedup beyond the noise threshold (there's enough of them that we usually that we don't need more sensitive benchmarking) or if there's another indicator for optimization, for example a flamegraph. The numbers here are for giving a sense of scale. For example, the profiling build i use for benchmarking (cargo build --profile profiling --features tracing-durations-export</code> is 12% slower than a regular release build (cargo build --release</code>). There's some tricks to reduce noise for low-percentage performance optimizations, such as ensuring a quiet machine, setting the CPU governor to performance and using taskset</code> to pin the benchmark to performance cores, but they may not even be realistic when users aren't setting these options in practice. ↩</a> </li> In rust, enum variants can have different members. In python terms, this would be a version protocol with the small and the large representation as classes implementing it. What Are Sum, Product, and Pi Types?</a> is a great introduction. ↩</a> </li> It should really just be 8 bytes and leaving a bit for the enum discriminant, another thing that makes uv slow: https://github.com/astral-sh/uv/blob/7cd6738def40106b1d0ac9eaefc233fcffedc9de/crates/uv-pep440/src/version.rs#L1144-L1154</a> ↩</a> </li> If wheels could have dynamic metadata that changes depending on the host, it wouldn't be possible to build lockfiles, and it would require code to determine this on installation, which is a security and stability risk. Dynamic metadata here specifically means that you need to run code to get the real values, it does not refer to e.g. platform-specific dependencies, that the package manager can evaluate with its own (or vendored) code, since the description of the platform markers is a well-defined expression. ↩</a> </li> There's some debate over whether all wheels of a release need to have the same metadata, and whether source distributions need to build into wheels of that same metadata. While it's not technically required in any spec yet, it is explictly required by both poetry and uv - both fail when this assumption is violated - and it is respected in the ecosystem. I'm only aware of two high-profile cases with old releases with tensorflow and torch which accidently shipped different metadata due to the complexity of their build systems. It's technically possible to work around wheels having different metadata, though it's unclear why this should be done, as on one hand platform markers exist precisely to have a single dependency specification for all platforms, and on the other, it would be a huge overhead in terms of network requests and resolver complexity. The other property is that source distributions are required to build into the same metadata as wheels (if any) or alternatively into the same metadata each time. This is required by every tools that uses lockfile: You can't install from a lockfile if a package has a new or mismatching dependency at install time that was never considered in the lock. Even pip behaves inconsistent when this property is violated. ↩</a> </li> Index and registry are synonyms ↩</a> </li> This prevents against attacks where the server that serves the file has been corrupted since the file was audited. It does not replace the part where you need to trust the author of the binary, but it makes this a one time trust, not a continuous trust that the infra is never taken over. ↩</a> </li> Shout-out to Natalie Weizenbaum</a> for developing pubgrub and writing about it, and to Matthieu Pizenberg</a> and Jacob Finkelnman</a> for writing and maintaining the pubgrub crate</a>. ↩</a> </li> This technically still allows concurrency between the resolver and a single network request as well as a single build, but it shows the effect sufficiently. ↩</a> </li> See e.g. https://swatinem.de/blog/smallstring-opt/</a>, https://mcyoung.xyz/2023/08/09/yarns/</a> and many more under the category "small string optimizations". ↩</a> </li> Arguably, we lose a bunch of free performance and memory because String</code> and PathBuf</code> are mutable by default, when we really need Box<str></code> and Box<Path></code> instead. String</code> and PathBuf</code> are three words (pointers) wide, and may over-allocate. We almost never modify string or paths in place, and definitely not in hot loops. We would likely also profit from defaulting to small string optimizations, most strings in uv are small-ish and i don't think we require the array-ness often if at all (OS APIs require conversions to C string or Windows datatypes anyway). ↩</a> </li> They support instruction counting next to walltime benchmarks, which should give more accurate results especially in CI. For uv there was a too high number of spurious regressions, so we had to deactivate it. Writing this article, i think i figured out why: https://github.com/astral-sh/uv/pull/18487</a>. Fingers crossed we get the instruction counting benchmark back as most reliable. ↩</a> </li> </ol> </section>

Reimplementing PEP 440 2022-12-01T00:00:00+00:00 I've reimplemented PEP 440</a>, the python version standard, for monotrail</a>: pep440-rs</a>. Did you now that 1a1.dev3.post1+deadbeef</code> is a valid python version, there's not only ==</code> but also ===</code> and that version specifiers are context sensitive? Let's start with the normal stuff: There are basic version numbers with dots in between (like 2.3.1</code>) and optionally alpha/beta/release candidate suffixes (canonically 2.3.1b1</code>, but conveniently lenient so 2.3.1-beta.1</code> also works). For dependencies, there operators for minimums and maximums separated by comma such as >=2.5.1,<3</code>. You can of course also select a specific prerelease (e.g. 1.1a1</code> being matched by ==1.1a1</code>) and maybe you've also seen constraints like 1.2.*</code>. But below the clear semver-y surface lie many demons of the old. Photography by Norbert Buduczki</a> It all starts with the part of the version that's hidden in the default: The epoch. By default it's zero, but if you want to switch versioning system, you can add the new epoch with an exclamation mark like 1!4.2.0</code>. Since version ordering is defined as a total order, 2020.1</code> < 1!0.1.0</code>, but also for some reason <1!0.1.0</code> matches 2020.1</code> and >2020.1</code> match 1!0.1.0</code> (the specifiers are not a total order normally, I don't know why it doesn't specify to never match across epochs). Being a mere mortal, I have never witnessed the turning of epoch myself, but the feature remains part of The Old Code. You can also add .dev</code> and .post</code> with some number to all versions, e.g. 1.0.0.dev1</code> or 1.0.0.post1</code>. Or just combine them and do 1.0.0.post1.dev1</code>, which is a developmental release of a post-release. That of course doesn't stop at final releases, you can now do 1.0.0a1.post1.dev1</code> to have a developmental release of a post release of a prerelease (in canonical form alpha/beta/rc don't have a dot, but dev and post do, while also in PEP 440 dev release are sometime included with the prereleases). If you sort them, obviously 1.0.0.dev1</code> < 1.0.0</code> < 1.0.0.post1</code>, and 1.0.0a1.dev1</code> < 1.0.0a1</code> < 1.0.0a1.post1</code> < 1.0.0</code>. But dev releases of the final version are sorted lower than any prerelease version, so suddenly we have 1.0.0.dev1</code>< 1.0.0a1.dev1</code> < 1.0.0a1</code> < 1.0.0</code>, while also having 1.0b2</code> < 1.0b2.post345.dev456</code>. That is, the try-out release for 1.0 proper is considered older than the try-out release for the 1.0 alpha. The only sensible way to implement this sorting is making a five-tuple where you map (pre-releasity, pre-number, post-number or None as smallest, dev-number or int max as largest, local version) and let tuple-sorting sort out the rest. Even pypa/packaging uses tuple logic feat. ±infinity</a>. Matching version with specifiers such as >=1.2.0</code> or <2.0.0</code> is tricky because PEP 440 says "Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, unless they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release". That's really fuzzy, and also it means whether a single version matches a specifier depends on the environment, something I confused myself with</a>. pypa effectively says that you must add to specifier whether you want to match prereleases (which here again include dev releases) or not when using the library. The consequence is that when you say ~=2.2</code> but there's only a 2.2.1a1</code> it will pick that alpha version (but not 2.2a1</code>, which never matches). There are also local versions which can added with a +</code> after the regular version, such as 3.4.0+my.local.123.version</code>. The 123</code> is going to get ordered as a number, everything that can't be parsed as a number will get ordered as a string. Information about usage is sparse, apparently linux distributions use it to tag their python packaging. Those are also in semver</a>, but more reasonable as "build metadata": "Build metadata MUST be ignored when determining version precedence. Thus two versions that differ only in the build metadata, have the same precedence". Finally, there's also ===</code>, "Arbitrary equality", which is advertised as "simple string equality operations" that "do not take into account any of the semantic information". pypa/packaging has a test that ===lolwat</code> parses with the comment "=== is an escape hatch in PEP 440". For those wondering why python1</a> didn't pick a sane standard like semver to begin with, the basic syntax format for writing things like >1.0, !=1.3.4, <2.0</code> was written down in PEP 314</a> in 2003 (!) 2</a>. PEP 386</a>, the first python version standard, was written in 2009, "codifying existing practices" and its successor and current standard PEP 440</a> in 2013. In comparison, the first commit to npm was made in 2010, semver v1.0.0 was published in 2011, v2.0.0 in 2013, npm inc. was founded in 2014 and cargo had its first commit also in 2014. So python has a hard time doing modern packaging because they were trying to do modern packaging before it was being invented. I still believe that (a) bringing in features from semver and tools such as poetry, cargo and npm would greatly benefit the python ecosystem and (b) python packaging isn't doomed to stay in its current state. While e.g. pypi's backend will have to handle everything that ever used to be legal, i believe that the ecosystem at large can and must migrate to better tools and standards. This is largely informed by having to deal with a lot of the breakages of the current state of python packaging and trying to support friends and colleagues. The easiest is probably to deprecate ===</code>, even PEP 440 soft-deprecates it with "Use of this operator is heavily discouraged and tooling MAY display a warning when it is used". For epochs, i haven't seen them used even a single time. To try to make this at least a bit empirical i ran two queries on the pypi bigquery data3</a> 4</a>, with the result that in one month there were 19,699,031,713 downloads, 40,281 of which for versions specifying an epoch, that's 0.0002%. Post releases can be replaced with publishing a new patch release or a one-higher pre-release. Historically, it was a good idea that you could specify 1.2.3</code> and if the author messed up 1.2.3</code> and has to publish fixup wheels you'd be directly moved to the fixup, but nowadays you want lock files where this doesn't work anymore, and it also interacts weirdly with yanking. This applies especially to post releases of prereleases, which PEP 440 acknowledges: "Creating post-releases of pre-releases is strongly discouraged, as it makes the version identifier difficult to parse for human readers. In general, it is substantially clearer to simply create a new pre-release by incrementing the numeric component". Dev version on prereleases seem also strange to me (just publish a higher prerelease instead, a test-release of a test-release is kinda redundant). For dev versions of final releases there are certainly workflows that benefit from them5</a>, even though other ecosystems do fine without special casing .dev</code>. The main problem are the strange semantics, and while PEP 440 defends this as "far more logical sort order", i strongly disagree, this was and is super confusing, and also the implementation is a mess. When removing dev (and ideally also post) releases at least for alpha/beta/rc versions, the semantics would become intuitive again, with dev release simply being a prerelease one level below alpha releases6</a>. For local version the semver style "purely informative and no semantics" definition would be imho more reasonable; i unfortunately can't tell if de-semanticizing local version would break anything (as in, is anybody currently depending on the fact that 1.0+foo.10</code> has precedence over 1.0+foo.9</code>). Given that pip now has a backtracking dependency resolver</a>, i think we can simplify the spec a lot by separating it into three parts: One part that defines the version number schema and precedence (a total order as it currently is), one part that translates operators such as ~=</code> into normal ></code>/=</code>/></code> sets that directly translate to the version order, and one part that specifies the rules for resolvers, that is when are they allowed to pick which prerelease. The latter isn't well-defined as of PEP 440, but imho we should agree about this across the ecosystem7</a>. See e.g. node on prereleases</a> and cargo on prereleases</a>8</a>. I particularly like the node/npm "If a version has a prerelease tag (for example, 1.2.3-alpha.3</code>) then it will only be allowed to satisfy comparator sets if at least one comparator with the same [major, minor, patch]</code> tuple also has a prerelease tag". For comparison, firefox estimates 16–20 minutes for the semver spec, but 57–73 minutes for PEP 440. For all change there would need to be long announcement and deprecation periods with a specific focus on helping people migrate their workflows. For the deprecation period, tools should print big red warnings whenever they encounter something broken. Speaking of announcements, there's really a lack of an official pypa communication channel! An official blog for announcements on deprecations, changes, release, and (proposed) PEP status changes together with a community aggregator like This Week in Rust</a> would be extremely helpful over the current word-of-mouth-in-twitter-replies-and-buried-github-issues system. Two features that would be great to add are the caret operator (^</code>) and the tilde operator (~</code>) from semver. Nowadays semver is arguably the most popular version scheme even in python, and for most packages you want ^1.2.3</code> and for the remainder (including calver projects that treat the last digit as semver-like patch version) ~1.8</code> will do the right thing. I'd like to add them to pep440-rs eventually but i'm neither sure about the exact semantic yet nor how to let users switch between PEP 440-only specifiers and the modern superset. Next Up: PEP 508 Well, technically not python as the python interpreter but pypa as the vague group of people who make the packaging PEPs. Python itself didn't even have a concept of package versions at all until importlib.metadata</code> introduced optionally reading a version as a string to the standard library, and the language itself still doesn't have a concept of packages but merely one of modules. When you import foo</code> it effectively just asks sys.meta_path</code> if anyone can import foo, which will check if any location in sys.path</code> has a foo</code> module, but this has no relation to packaging. If you ask stdlib's importlib.metadata</code> for an installed package version, it really just asks sys.meta_path</code> with a different method if anyone optionally wants to tell it about the package version</a>, which by default will just look for .dist-info</code> folders in your sys.path</code>. ↩</a> </li> If you ever wondered why wheel metadata is in some archaic e-mail-headers RFC 822 STANDARD FOR THE FORMAT OF ARPA INTERNET TEXT MESSAGES</pre></div> that's because it was picked in 2001</a>. Even XML 1.0 was published just 3 years prior</a>. I'm still very much in favor of migrating to a JSON or TOML format</a> such as pkg-info.json</code> or editing pyproject.toml</code> similar to what cargo does, but that's for another time. ↩</a></li> I ran this on 2022-11-29 and the queries were SELECT COUNT(*) FROM bigquery-public-data.pypi.file_downloads WHERE timestamp BETWEEN TIMESTAMP(DATETIME_SUB(CURRENT_DATETIME(), INTERVAL 1 MONTH)) AND TIMESTAMP(CURRENT_DATETIME())</code></pre> and SELECT COUNT(*) FROM bigquery-public-data.pypi.file_downloads WHERE timestamp BETWEEN TIMESTAMP(DATETIME_SUB(CURRENT_DATETIME(), INTERVAL 1 MONTH)) AND TIMESTAMP(CURRENT_DATETIME()) AND CONTAINS_SUBSTR(file.version, '!')</code></pre> ↩</a></li> Blessed be whoever came up with the bigquery datasets for pypi</a> ↩</a> </li> E.g. some people want to build {Major}.{Minor}.{Patch}.dev{YYYY}{MM}{DD}{MonotonicallyIncreasingDailyBuildNumber}</code> in their CI workflows. Local versions are used to indicate when linux distributions did some downstream packing, so you can directly tell when you're looking at a distro patched install. ↩</a> </li> I'm still not sure if they provide any benefit over just using alpha versions, but once they behave like normal prereleases their implementation and cognitive overhead is near zero so backwards compatibility is way more significant. Note that semver relies on "alpha", "beta" and "rc" being alphabetically ordered, while we need to make "dev" lowest manually, otoh semver also allows any random stuff for prereleases and uses the same duck-typed logic for comparing them as PEP 440 uses for local versions. ↩</a> </li> Consider the case where a user adds a library A</code> from pypi that has multiple transitive dependencies on B</code>, some specifier with preleases in their specifiers and some without. It would be bad for the authors of A</code> to work if they couldn't clearly reason which prereleases of B</code> might or might not be picked independent of which tool the user uses. ↩</a> </li> According to the python survey results, those are the two most popular other package managers in use RubyGems states</a> "The RubyGems team urges gem developers to follow the Semantic Versioning standard for their gem's versions. The RubyGems library itself does not enforce a strict versioning policy, but using an "irrational" policy will only be a disservice to those in the community who use your gems", but i couldn't find any details on what versions and operators are allowed. Composer on the other hand is very much like python (docs</a>): "This must follow the format of X.Y.Z or vX.Y.Z with an optional suffix of -dev, -patch (-p), -alpha (-a), -beta (-b) or -RC.", where dev is below alpha. It also seems to allow 1.2.*</code> but i couldn't find any more documentation on what's allowed and what the semantics are except that they apparently transform prereleases to a version digit</a> ↩</a> </li> </ol> </section> A dive into packaging native python extensions 2018-07-21T00:00:00+00:00 The complete guide to building you own native wheel from scratch There are cases where you want to extend python with native code, e.g. for scientific computing (numpy, scipy), database connectors (mysqlclient, psycopg2) or UI (pygobject, pyqt). For cpython this is traditionally done in C/C++, but you can also use the C api from D (pyd</a>), go (cffi</a>) or rust (cffi</a> or pyo3</a>). Distributing those extensions is a big problem. Until recently, the only viable option was to write special plugins for setuptools, e.g. milksnake</a> for cffi or setuptools-rust</a> for pyo3. Inspired by the new pyproject.toml</a>, I wanted to get rid of the flaws</a> and resulting pain</a> of setuptools. So I went to write pyo3-pack</a>, which aims at making packaging and publishing native python modules in rust as easy as wasm-pack</a> makes it for javascript. It turns out that writing such a tool is relatively easy (less than a thousand lines of rust to get from source to wheel). The hard part is to find out what you need to do in the first place. The documentation is scattered across different and partially outdated tutorials, PEPs, stack overflow answers, references, examples and source code; I sometimes even had to resort to reverse engineering. So I decided to write down everything I learned about native wheels, which eventually became this blog post. The good parts</a></h2> The official tutorial on native modules, Extending Python with C or C++</a>, is a good introduction to the core concepts of native modules: The header files, the calling conventions, GC, the object protocol and error handling. It only shows building for C/C++ with distutils (the predecessor to setuptools) though, omits the officially blessed manylinux, and lacks an explanation of the abi and linking options (more on those later). For the daily work, the Python/C API Reference Manual</a> is often much better. It also has some explanation for the ABI. For the rest of the post I'll assume that you have built your native python module as shared library (e.g PyInit_<modname></code> function for python 3) with your technology of choice. Metadata</a></h2> Each python package, whether it is an egg or a wheel</a> or a source archive, is described by structured metadata, which contains fields required for pip to work and informational fields used e.g. for pypi. There are five versions for the metadata of python packages: 1.0 (PEP 241</a>), 1.1 (PEP 314</a>), 1.2 (PEP 345</a>), 2.0 (PEP 426</a>) and 2.1 (PEP 566</a>). 2.0 was an attempt to replace the key-value structure of the metadata with a json like structure. This could have been a big improvement, but was withdrawn (and is not accepted by pypi or pip) since it would have been a to big breakage. This is why the current version is called 2.1, even though it is backwards compatible to 1.0. The current specification can be found at PyPA's Core metadata specifications page</a>, which is pretty self-explaining and worth reading. N.B.: https://www.pypa.io/en/latest/roadmap/ is completely outdated as it still features as Metadata 2.0 as part of the roadmap. https://packaging.python.org/specifications/core-metadata/#description is misleading since you must not use the RFC 822 in the metadata for the pypi upload (see the section on uploading) and for the METADATA file inside the wheel you can just put the description in the body, i.e. after all the keys. Tags and naming</a></h2> Native modules need to specify with which platforms and python interpreters they are compatible. Python has two coexisting standards with slightly different syntax: PEP 425</a> for packages and PEP 3149</a> for shared libraries. Both are based on abi tags, so let's discuss them first. The cpython ABI</a></h3> cpython abi is composed of the major and minor version of cpython and a set of abiflags, which are determined by compiler flags. According PEP 3149</a> there are three such compile time options we need to consider (at least for linux and mac): d</code>: --with-pydebug</code></li> m</code>: --with-pymalloc</code></li> u</code>: --with-wide-unicode</code></li> </ul> For practical purposes, d</code> is irrelevant, m</code> is always set and u</code> may or may not be set - more on u</code> below. The tag for this abi is cp{major}{minor}{abiflags}</code> or cpython-{major}{minor}{abiflags}</code>. My python 3.6 installation is for example cp36m</code> and cpython-36m</code>. The u</code> or wide-unicode flag is about the representation of unicode characters (introductory article</a>). Initially, python unicode characters were fixed to two bytes (UCS-2), meaning that any 3 or 4 byte characters were not representable. This changed with PEP 261</a>, which added optional support for wide unicode characters (UCS-4) to python2. The choice between UCS-2 and UCS-4 was made a compile time option, creating the abi without "u" for UCS-2 and one with "u" for UCS-4 (ignoring the option to completly disable unicode). In python 3.3</a> this was replaced by a system that determines the representation at runtime described in PEP 393</a>, removing the "u" flag from the abi. This means that the wide-unicode option is only relevant for backwards compatibility with python 2. The stable abi</a></h3> There are obviously some big drawbacks from having tons of different abis which you all need to support and build and test, so PEP 384</a> introduced the "stable abi" in python 3.2. This abi with the tag abi3</code> contains a subset of the full abi and that is forward compatible with all future 3.x releases of cpython. In the header files, everything that is not part of the stable abi is gated with #if !defined(Py_LIMITED_API)</code>. The stable abi is extended from time to time, meaning that you can require the stable abi and a minimum version. In the header files this is done by setting Py_LIMITED_API</code> to the minimum support python in the PY_VERSION_HEX</code> format as described in the documentation</a>. In the headers this is checked e.g. with #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03030000</code> for a function that was added to the stable abi in python 3.3. Sysconfig</a></h3> In the initial version of this post, I wrote about getting the required information about the interpreter through sysconfig. But it turned out that sysconfig behaves inconsistently across python versions and operating systems. E.g. the VERSION</code> field on linux is in the format {major}.{minor}</code>, while it is {major}{minor}</code> on windows (both with python 3.7). There's also EXT_SUFFIX</code>, which tells you the complete extension of the library filename on linux (e.g. ".cpython-35m-x86_64-linux-gnu.so"</code>), but on windows it's just .pyd</code>. I've collected a few samples in a folder in the pyo3-pack repo</a>. You'll find more of those weird cases in there. I'm currently using the following snippet with python -c</code> and do the logic and sanity checks in rust. import sysconfig import sys import json print(json.dumps({ "major": sys.version_info.major, "minor": sys.version_info.minor, "abiflags": sysconfig.get_config_var("ABIFLAGS"), "m": sysconfig.get_config_var("WITH_PYMALLOC") == 1, "u": sysconfig.get_config_var("Py_UNICODE_SIZE") == 4, "d": sysconfig.get_config_var("Py_DEBUG") == 1, # This one isn't technically necessary, but still very useful for sanity checks "platform": sys.platform, }))</code></pre> This is than deserialized into the equivalent of the following python 3.7 code: @dataclass class Interpreter: major: int minor: int abiflags: Optional[str]</code></pre> If you still want to use sysconfig, the easiest way is through python -m sysconfig</code>. As seen above, you can use WITH_PYMALLOC</code> (1 means m</code>), Py_UNICODE_SIZE</code> (4 means u</code>) and Py_DEBUG</code> (1 would mean d</code>) for the python 2 abiflags. To get the flags in machine readable as Dict[str, Union[str, int]]</code>, use: python -c "import json, sysconfig; print(json.dumps(sysconfig.get_config_vars()))"</code></pre> For a Dict[str, str]</code>, use: python -c "import json, sysconfig; print(json.dumps({k:str(v) for k, v in sysconfig.get_config_vars().items()}))"</code></pre>Naming shared libraries</a></h2> PEP 3149</a> defines that shared libraries will get a tag between the file name and the extension, separated by dot. It tells you that this tag needs to include at least the implementation (i.e. cpython) with its major and minor version. It also shows .cpython-32mu.so</code> as an example for such file extension, from which we can derive .cpython-{major}{minor}{abiflags}.so</code> as template. This sounds nice, but is extremely misleading if not plainly wrong in reality. From picking apart other native libraries and trial and error with filenames I figured the following: Python 2.7 - 3.2 doesn't have any abitags.</li> Python 3.2 - 3.4 actually use the scheme .cpython-{major}{minor}{abiflags}.so</code> for POSIX (i.e. linux and mac), but accepts files without tag. Windows still doesn't use tags.</li> Python 3.5+ uses the a new scheme with the platform included, which is now also used for windows. 3.5.+ also accepts files without any tag, but not those with a 3.2 - 3.4 style tag.</li> </ul> The only place the new, 3.5+ schema has ever been announced were the python 3.5 release notes</a>. But rejoice, even those are wrong. (I tried googling both the wrong</a> and the correct</a> version, but it really seems to be only in those release notes) For 3.5+, I found that the following is what's actually working (and also what setuptools produce): Linux Template: .cpython-{major}{minor}{abiflags}-{architecture}-{os}.so</code> architecture</code> is either i386</code> or x86_64</code>, and os</code> is linux-gnu</code>. The release notes state that the file extension is .pyd</code>, which is wrong and doesn't work in practice. Also note that os has an internal minus, breaking the general rule of separating parts of the tag with a minus. Example: steinlaus.cpython-35m-x86_64-linux-gnu.so</code> Mac OS Template: .cpython-{major}{minor}{abiflags}-darwin.so</code> Example: steinlaus.cpython-35m-darwin.so</code> Windows Template: {name}.cp{major}{minor}-{platform}.pyd</code> The platform is either win_amd64</code> or win32</code>. .pyd files are just renamed .dll files, which is confirmed in the official windows FAQ</a> (which is otherwise extremely outdated) Example: steinlaus.cp35-win_amd64.pyd</code> Naming wheels</a></h2> The documentation for defining wheels is much better than the one for naming so files with most parts being specified in PEP 425</a>. The official schema from that PEP is {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl</code>, which is used for all python versions. The distribution is you package's name escaped with re.sub("[^\w\d.]+", "_", distribution, re.UNICODE)</code>, we can ignore and skip the build tag, the python tag for our case is cp{major}{minor}{abiflags}</code>, the abi tag is either the python tag, abi3</code> or none</code>. For the platform tag it states that "The platform tag is simply distutils.util.get_platform() with all hyphens - and periods . replaced with underscore _." This is unfortunate, since the output of distutils.util.get_platform()</a> isn't specified, so we need to reverse engineer. Looking only at 32-bit and 64-bit x86, we have either win_amd64</code> or win32</code> for windows. For linux, we have linux_i686</code> or linux_x86_64</code>, even though in practice we must use either manylinux1_i686</code> or manylinux1_x86_64</code> as desribed in the manylinux paragraph below. For mac the tag used by setuptools is macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64</code> for whatever reason. Examles: steinlaus-1.0.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl</code></pre>steinlaus-1.0.0-cp36-cp36m-manylinux1_x86_64.whl</code></pre>steinlaus-1.0.0-cp36-cp36m-win_amd64.whl</code></pre>Manylinux</a></h3> Libraries and binaries on Linux are traditionally (for the better or worse) dynamically linked to libraries in $LD_LIBRARY_PATH</code>, which are installed through the systems package manager. Native modules could require arbitrary versions of arbitrary libraries, but can't guarantee they are installed on the target machine, leading to linker errors when importing. To avoid such incompatibilities, PEP 513</a> specifies a target manylinux1</code> which contains only a set of old versions of libraries that can be found on basically every Linux. (This is an extremely short summary of the rational in the PEP) Wheels for the manylinux1</code> target must be in the manylinux1 docker container</a>. This container is based on CentOS 5, i.e. some very old Linux. Using this docker image is the only officially blessed way to build for the linux target in general. Pypi only accept wheels with the manylinux1</code> tag and rejects those with a linux</code> tag. A slightly more modern target, manylinux2010</code> is currently being working on as a successor for manylinux1 (PEP 571 - The manylinux2010 Platform Tag</a>, tracking issue</a>). manylinux is accompanied by a tool called auditwheel</a> that checks the library and then "awards" the manylinux1 tag. Afaik this is not checked by pypi, so it's possible to lie about that check. By default rust only links very few system libraries, which are a subset of the manylinux1 target. This means that pyo3-pack only needs to check that the constraints are met and we can otherwise totally skip the whole ancient-docker-mess. The internals of a (binary) wheel</a></h2> While there alternative ways to install python packages, using wheels with pip is (for good reasons) the officially blessed one, so for pyo3-pack I've only looked into into building wheels. They are specified in PEP 427</a>. Wheels are generally just zip files with a .whl</code> extension. They come in two flavors: sdist and bdist. bdist ("built distribution") wheels are pre-built packages. Their installation is mostly just unpacking the archive. They specify the compatible python version(s), an abi and a platform. sdist ("source distribution") wheels contain all the sources including your setup.py</code> or pyproject.toml</code>, so for installing them, they need to be built first. Every wheel contains a {distribution}-{version}.dist-info</code> folder with the following files inside it, where {distribution}</code> is again the name with the underscore-escapes. WHEEL</code>: Wheel-Version: 1.0 Generator: pyo3-pack ({version}) Root-Is-Purelib: false Tag: {python tag}-{abi tag}-{platform tag}</code></pre></li> METADATA</code>: This file contains the metadata as described above. Since metadata 2.1, you can (and want to) put the description in the body of the file, separated from the key value pairs by a newline. The only required keys are Metadata-Version</code>, Name</code> and Version</code>. Metadata-Version: 2.1 Name: {name} Version: {version} Summary: {summary or UNKNOWN} {description / content of readme.md}</code></pre></li> RECORD</code>: This file contains checksums and sizes of all files. Each line contains a file, a hash and the size of the file in bytes separated by commas like the following: path/to/file,sha256=HASH-AS-URLSAFE-BASE64-NOPAD,1234</code> The only exception is the record file itself, for which hash and size are left blank: {name}-{version}.dist-info/RECORD,,</code> The exact format is described in PEP 376</a>, while PEP 427</a> adds that the hasing algorithm must be "sha256 or better". </li> entry_points.txt</code>: This file isn't specified in any PEP, but in the Entry points specification</a>. It contains sections with key-value pairs in the ini format. While there's more it can do, the interesting part is a section called console_scripts</code>. This section lists function which should be exposed as shell commands. The keys are the commands, while the value specifies which function to call. Pip will create the scripts which are small wrappers around the functions when installing the package. The functions have the structure some.module.path:object.attr</code>. E.g. poetry defines [console_scripts] poetry=poetry.console:main</code></pre> which pip translates to #!/usr/bin/python3 # -*- coding: utf-8 -*- import re import sys from poetry.console import main if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit(main())</code></pre></li> top_level.txt</code>: Setuptools also add this file which contains only the name of your package. This is part of the (PEP-less) egg format, the predecessor of wheels, as described in The Internal Structure of Python Eggs</a>. This file is not documented and not needed for wheels and therefore not added by other packagers such as poetry. (Interestingly enough, the wheel</a> repository, which adds the bdist_whl</code> command to setuptools, does not even contain the string top_level.txt</code>.) </li> </ul> For actual package you have two options: If you only need to package one shared library, you put it at the top level of the zip. The shared library must be named according to the rules describe above, while basename must be the name of the module. Example: . ├── get_fourtytwo-1.6.8.dist-info │ ├── METADATA │ ├── RECORD │ └── WHEEL └── get_fourtytwo.cpython-36m-x86_64-linux-gnu.so</code></pre> For any wheels containing python files, whether they have native components or not, the top level module is a python module. This means a directory at the top level with the name of the module and a __init__.py</code> inside that direct ry. Inside this directory the same rules as for any other python project apply. Native modules work the same way as pure python single file module, only that the filenames end with .so</code> or .pyd</code> instead of .py</code>. Take a look at numpy's wheels for a complex, real world scenario. Example: . ├── get_fourtytwo │ ├── __init__.py │ ├── native_fourtytwo.cpython-36m-x86_64-linux-gnu.so │ └── python_fourtytwo.py └── get_fourtytwo-1.6.8.dist-info ├── METADATA ├── RECORD └── WHEEL</code></pre> ... where __init__.py</code> contains from .native_fourtytwo import native_class from .python_fourtytwo import some_class</code></pre> Besides the presented wheel 1.0 format, PEP 491</a> defining a "Wheel 1.9" format also exists. It is officially in draft status, but it seems completely abandoned, with no mention neither on the mailing list nor in the relevant github repos. The PEP doesn't explain why version 1.9 should follow version 1.0. Note that you can lie to pypi about the metadata. E.g. I actually ran into a case, where a .tar.gz was uploaded as 3.0.{date}, while the installed package identified itself as 3.0.dev0, which didn't exist on pypi. This effectively broke pip freeze. Source distributions</a></h2> Source distribution, sdist for short, are special source archives that can be build and installed with pip. They are used e.g. when there are no wheels for the current platform/abi and as base for building debian or fedora packages. While they existed for a longer time, they are formally specified in PEP 517</a>. This PEP differentiates between a source tree, which would be the git repository, and a source distribution, which this paragraph is about. A source distribution is a .tar.gz archive. It is explicitly stated that zip archives are not allowed anymore, even though it mentions lxml-3.4.4.zip</code> as an example in the beginning. The filename is {name}-{version}.tar.gz</code>. The archive contains one folder, which is named {name}-{version}</code>. This folder contains the required source, a setup.py and/or a pyproject.toml, and a file called PKG-INFO</code> which identical to the METADATA</code> file in wheels. Example: foobar-0.11.2/ ├── foobar │ ├── __init__.py │ └── main.py ├── LICENSE ├── PKG-INFO ├── pyproject.toml └── setup.py</code></pre> If the archive contains a pyproject.toml with a [build-system]</code> section that specifies a list of packages required for building as requires</code> and the path to a build backend object in build-backend</code>, this backend should be called by pip to build the source distribution into a wheel. pip 10.0.1, which is the latest version as of this writing, refuses to install such wheels stating "This version of pip does not implement PEP 517 so it cannot build a wheel without 'setuptools' and 'wheel'.". We can therefore skip any further details about the build backend because we can't use it yet anyway. Without a pyproject.toml with those entries, pip executes the setup.py</code> in the directory, meaning that currently the way to support source distributions is to use setuptools, which is exactely what I wanted to avoid. This means no sdist in custom packagers for now. As a side note, both flit and poetry already implement the PEP 517 interface (buildapy.py in flit</a> and api.py in poetry</a>) and add a pyproject.toml to the archive. But as they omit the [build-system]</code>, pip instead uses the setup.py they also create. Finding python interpreters</a></h2> It's convenient for building and essential for testing to find the installed python versions. For linux and mac, you can check which python binaries are in PATH</code> and then use the snippet from above to get the version and abiflags. (Or you can use a fixed list of 2.7 and 3.5+ and just try each because there's no good library to work with PATH</code> yet). For windows, every python version is just called python.exe</code>. Fortunately, there is a launcher called py</code>. With -0</code> (but not --list</code>, even if the help says otherwise) it will list all known versions, which you can launch with py -{version}</code>. It's then easy to get the path of the actual interpreter with py -{version} -c "import sys; print(sys.executable)"</code>. Contemporary legacy uploading</a></h2> Now that we've got our wheel built, we also want to publish it, i.e. upload it to pypi (which is now powered by a software called warehouse). It turns out that the api to upload packages is called the "legacy api", even though there's no new api for uploads (there is a json api, but it only supports reading package metadata). The upload part of the legacy api had no documentation other than "use twine", so I read through the source of poetry uploader, warehouse's endpoint and warehouse's tests to figure out how to use that api. Eventually I wrote a pull request</a> to warehouse documenting that api</a>. Errors</a></h2> As mentioned in the preface, all the information presented here is assembled from many different sources of varying qualitity and up-to-dateness, with some parts being reverse-engineered. So if you find any errors or missing parts, please ping me (konstin@mailbox.org, konstin on github, @konstinx on twitter). Meine Stadt Transparent, Teil 2: Die Technik 2017-12-20T12:00:00+02:00 Seit fast vier Monaten entwickeln Tobias Hößl</a> und ich Meine Stadt Transparent (Demo</a>, GitHub</a>). Im ersten Teil</a> ging es darum, wie es dazu kam und warum wir dieses Projekt machen. In diesen Teil geht es um den momentanen Status und die technische Details. Ziele</a></h3> Wie in Teil 1 beschrieben gibt es bereits genügend kommerzielle Systeme, die sich hauptsächlich an Verwaltungsmitarbeiter und Stadträte richten, aber für Bürger kaum benutzbar sind. Wir wollen die Lücke eines bürgerfreundlichen Systems füllen. Anders gesagt wollen wir Zugänglichkeit für alle schaffen. Verständliche Sprache: Existierende Systeme sind oft eine Mischung aus unverständlichen Abkürzungen und Wortmonstern wie „Beschlussvollzugkontrolle", „Ratsinformationssystem" und „Anliegenmanagement", die die meisten Nutzer erst mal googlen müssen. Das macht Thema „Stadtpolitik" nur noch unsexier als es sowieso schon ist. Wir vermeiden deswegen Abkürzung, Fachbegriffe und Techspeak soweit möglich. Googlebarkeit: Informationen, die man durch die Suche nicht findet, existieren nicht. Natürlich kann man auch versteckte Daten finden</a>, aber das ist Zeitaufwendig und scheitert oft auch einfach daran, dass man nicht weiß, was überhaupt existiert</a>. Likability: Politik ist eines dieser Themen, die zwar sehr wichtig aber auch sehr langweilig sind. Deshalb ist es umso wichtiger, dass man eine Website dazu auch benutzen möchte. Technische Perfektion ist schön und gut, aber wenn eine Webseite so interessant ist wie ein Ladebalken wird sich kaum einer freiwillig dort aufhalten. Das heißt nicht, dass wir Seite mit Achievements, Kalendarsprüche und Animationen vollstopfen, aber ein paar Katzenbilder</a> und Eastereggs dürfen es schon sein. Barrierefreiheit: Politik betrifft alle Menschen. Leider werden die Dokumente in der Regel nur als pdf veröffentlicht, woran wir aber im Moment leider nichts ändern können. Wir versuchen deshalb zumindest die Seite an sich durch html-aria-tags und Linter möglichst Barrierefrei zu halten. Nachhaltigkeit: Wenn Software nicht so gebaut ist, dass sich gut weiterentwickeln und damit an neue technische Entwicklungen anpassen lässt, veraltet sie und verliert dadurch nach und nach ihre Nutzbarkeit. Likability durch cat content (Symbolbild) Frameworks und Bibliotheken</a></h3> Um diese Ziele umzusetzen brauchen wir einen ganzen Stapel an Technik. Das Backend ist in pyhton 3 mit django geschrieben. Als Datenbank verwenden wir Mariadb bzw. teilweise sqlite und für die Suche elasticsearch. Synchronisiert werden die Daten über django-elasticsearch-dsl, als Server läuft gunicorn hinter einem nignx. Für das Fontend nutzen wir Bootstrap 4 in django-Templates, dazu etwas javascript in einer npm-es6-babel-webpack-pipeline. Wir hatten am Anfang überlegt, ein Javascript-Framework zu verwenden, uns dann aber dagegen entschieden. Mit Angular hatten wir beide bereits Erfahrung, wollten es aber aber aus unterschiedlichen Gründen nicht verwenden, genauso wenig verwenden wie vue.js noch React. Bei den meisten anderen Frameworks muss man davon ausgehen, dass sie in der schnelllebigen Javascript-Welt untergehen und damit nicht für ein langfristiges Projekt taugen, wie der Stack Overflow Blog sehr anschaulich gezeigt hat</a>. (Immer noch aktuelle Leseempfehlung dazu: How it feels to learn JavaScript in 2016</a>). Ich persönlich würde am liebsten elm</a> oder rust/webassembly</a> verwenden, leider sind beide aber nicht annähernd ausgereift genug. Nach 3 Monaten mit Bootstrap bin ich mit unserer Entscheidung immernoch glücklich; Die Dokumentation von Bootstrap ist exzellent, die fertigen Klassen nehmen einem eine Menge Arbeit ab und für zwei nicht-Designer sieht die Seite nicht schlecht aus. Das Design kann später über Themes gut anpasst werden, was z.B. praktisch ist, um die Seite an verschiedene Städte anzupassen. Für Javascript-Bibliotheken brauchen wir außerdem keine Framework-Anbindung. Die Live Demo</a></h3> Wir haben eine Live-Demo unter meine-stadt-transparent.de</a>, die echte Daten der Stadt Jülich anzeigt. Dank des großartigen Github Auto Deploy</a> ist die Demo immer auf dem aktuellen Stand. Das heißt aber auch, dass ein falscher Commit die Seite zerschießen kann (und momentan auch darf). Der Datenimport</a></h3> Die Seite ist so gebaut, dass man mit etwas eigenem Python-Code prinzipiell beliebige Daten importieren kann. Damit das Projekt aber nicht nur einen hypothetischen Zweck sondern auch eine realen praktische Nutzen hat, haben wir einen Importer für OParl-Schnittstellen geschrieben. OParl</a> spezifiziert dabei eine API, mit der Daten aus deutschen Ratsinformationssystemen (RIS) in einem einheitlichen json-Format über eine REST-Schnittstelle exportiert werden können. OParl 1.0 wurde zwar komplett ehrenamtlich entwickelt, wird aber mittlerweile von vier großen RIS-Herstellern angeboten bzw. entwickelt. Durch Open.NRW gibt es mittlerweile eine offizielle OParl-Unterstützung in 21 Kommunen mit Sternberg SD.NET</a>. Eine der 21 Kommunen, die Kleinstadt Jülich, nutzen wir für unsere Demo-Seite. Den Importer zu schreiben war komplexer als gedacht und wir hatten kompliziertere Bugs</a> und andere Probleme</a>. Eines der gößeren Probleme war und ist die Integration von liboparl</a>, da die Integration von gnome-Bibliotheken wie liboparl in Python deutlich schlechter funktioniert als erwartet. (Bedeutet in der Praxis, dass wir python-gobject/gi verwenden müssen, das aber nur mit System-Python und einem symbolischen Link in die virtualenv funktioniert. Das undokumentierte Über-pip-Installieren geht zwar theoretisch, in der Praxis dann aber doch nicht</a>) Die Installation</a></h2> Meine Stadt Transparent soll sich möglichst einfach aufsetzen lassen. Dafür haben wir eine Schnellstart-Anleitung im readme, die mittels docker compose alle benötigten Dienste (im Moment mariadb, elasticsearch und django) einrichtet, startet und verbindet. Wegen schlechten</a> Managments</a> bei Docker und dem Datenimport ist immernoch etwas Handarbeit dabei, die wir aber so gering wie möglich halten. Ausblick</a></h2> Wir haben noch etwa zwei Monate bis zum Demoday am 28. Februar. Die Hauptaufgaben bis dahin sind ein OParl-Export, eine Änderungshistorie für alle wichtigen Objekte sowie ein paar interne Umbauten. Meine Stadt Transparent, Teil 1: Was bisher geschah 2017-12-20T00:00:00+00:00 Seit fast vier Monaten entwickeln Tobias Hößl</a> und ich Meine Stadt Transparent (Demo</a>, GitHub</a>). In diesem ersten Blogpost geht es darum, wie es dazu kam und warum wir dieses Projekt machen. Im zweiten Teil wird es dann um das eigentliche Projekt und dessen technische Details gehen. Stadträte, Beezirksausschüsse und Gemeindräte produzieren seit langem große Mengen Papier. Der Münchner Stadtrat und die 25 Bezirksauschüsse allein haben bespielweise im Jahr 2015 wurden über 15.000 Dokumente mit mehr als 64.000 Seiten veröffentlicht.1</a> Um der großen Anzahl an Dokumenten Herr zu werden hat man in den 2000ern einen großen Teil der Stadtratsverwaltung digitalisiert. Dazu wurden spezielle Webportale gebaut, die Ratsinformationssysteme genannt werden. Während Städte besonders in der Anfangszeit auf Eigenentwicklungen oder Spezialanfertigungen gesetzt haben, haben sich mittlerweile fertige Systeme und eine handvoll Anbieter etabliert. Durch solche fertigen Systeme setzen mittlerweile auch immer mehr kleine Kommunen oder Kommunalverbände ein Ratsinformationssystem ein. Deren öffentliche Oberfläche ist oft die einzige Möglichkeit, an wichtige Dokumente zu kommen. Die Angelenheiten des eigenen Stadt- oder Gemeinderats betreffen eine mehr, als man oft erwartet. Dort wird z.B. jegliche Bauvorhaben, Schulen und Kitas, den ÖPNV sowie Unterstützung für eine Vielzahl von Vereinen und Initiativen entschieden. Leider sind die Webseiten als Informationsquellen für Bürger nicht zu gebrauchen: Die Oberflächen sind völlig veraltet, voll mit bürokratischen Fachbegriffen und außerdem voller technischer Fehler. Das größte Problem ist jedoch meisten das Fehlen einer Suche, mit der man etwas findet. Durch diese Probleme werden viele wichtige öffentliche Dokumente für die Öffentlichkeit faktisch unzugänglich. Das Problem haben Leute in verschiedenen Städten erkannt und versucht, auf eigenen Faust und fast immer ohne Unterstützung der Verwaltungen bessere Webseiten zu bauen. Diese Arbeit fand zu großen Teilen unter dem Dach der Open Knowledge Labs</a> statt. Leider sind die meisten dieser Projekte nie fertig geworden und irgendwann eingeschlafen. Zwei Projekte waren jedoch erfolgreich und existieren bis heute.2</a> Das eine ist Politk bei Uns</a>, welches ursprünglich 2012 als Plattform für das Ruhrgebiet veröffentlicht wurde.3</a> Mittlerweile kann man unter anderem die Dokumente für Köln, Bochum und die Berliner Bezirke durchsuchen. Das Hauptziel ist es, möglichst viele Städte durchsuchbar und damit für Bürger (insbesondere auch über google und co.) zugänglich zu machen. Die offiziellen Daten werden automatisiert aufbereitet, indem z.B. der Text aus pdfs extrahiert wird und Adressen erkannt werden. (Von Politik bei Uns wird mittlerweile Version 2</a> entwickelt. Aber das ist eine eigene Geschichte.) Das andere Projekt ist München Transparent</a>. Angefangen hat es als eine bessere Oberfläche für das münchner Ratsinformationssystem</a>, die wie Politik bei Uns die Daten aufbereitet und durchsuchbar macht. Dort werden aber nicht nur nicht alle Daten des offiziellen Systems abgebildet, sondern auch zusätzliche Informationen wie Verordnungen, verständliche Erklärungen und Geodaten. Der Clou des Systems sind jedoch die Benachrichtigungen, die Nutzer per E-Mail informieren, wenn es in seiner Umgebung oder zu von ihm abonnierten Themen Neuigkeiten gibt. Der größte Teil wurde Tobias Hößl entwickelt, erst unter dem Namen "OpenRIS", dann als "Ratsinformant". Ende 2014 bin ich zu dem Projekt gestoßen und habe die Seite erweitert und verbessert. Im Februar 2015 haben wir die Seite dann als München Transparent richtig veröffentlicht. Mittlerweile benutzen sogar Angestellte der Stadt und einige Stadträte München Transparent statt des hauseigenen Systems.4</a> Mit der Stadt arbeiten wir auch seit längerem gut zusammen, wofür wir sehr dankbar sind.5</a> Verteilte Zuständigkeiten und langwierige Prozesse machen aber natürlich in der Stadtverwaltung vieles komplizierter machen als bei einem Hobbyprojekt. </a></td> </a></td> </tr> </a></td> </a></td> </tr> </table> Links ist München Transparent, recht das offizielle münchner Ratsinformationssystem, jeweils mit Links zu den ensprechenden Seiten. Beide Projekte nutzen sogenannte Scraper, um die Informationen der offiziellen Seiten auszulesen. Das sind mehr oder weniger kleine Programme, die jede Seite der Oberfläche abrufen, die interessanten Daten extrahieren und die Ergebnisse in einer Datenbank speichern. Aus dieser Datenbank wird dann wiederum die neue Webseite erstellt. Durch Scraping kommt man zwar an die benötigten Daten, aber eigentlich ist es ziemlich unsinnige Sache: Man steckt viel Arbeit in ein Programm, das im Endeffekt die Datenbank hinter einer Webseite rekonstruiert, in dem es jede Seite dieser Webseite aufruft. Viel einfacher wäre es, wenn man die Daten direkt in maschinenlesbarer Form bekommen könnte, am besten noch für jede Stadt im gleichen Format. Aus diesem Grund ist OParl</a> entstanden. OParl ist eine maschinenlesbare Schnittstelle, die man mit geringem Aufwand in existierende Ratsinformationssysteme einbauen kann. Dazu werden die Daten verschiedener Städte auf eine gemeinsame Darstellung gebracht und es ist festgelegt, wie man diese Daten effizient kopieren kann. Über München Transparent wurden wir irgendwann auf OParl aufmerksam, dass damals noch eine ziemliche Baustelle mit immer wieder verschobenen Veröffentlichungsterminen war. Da ich eine solche Schnittstelle für München Transparent haben wollte (und weil ich damals noch zu viel Zeit und hatte), fing ich an mih bei OParl zu beteiligen. Im Juli dieses Jahres haben wir die erste Stabile Version 1.0 veröffentlicht, die im Moment von mehreren großen Anbietern in ihre Ratsinformationssysteme eingebaut wird. Politik bei Uns, München Transparent und 21 Kommunen in Nordrhein-Westfalen haben die Schnittstelle bereits. Eine Version 1.1 mit Fehlerkorrekturen und eine englische Übersetzung sind in Arbeit. Bei München Transparent haben wir verschiedene Anfragen bekommen, ob wir sowas wie München Transparent denn nicht auch für andere Städte machen könnten, schließlich ist die Oberfläche schon fertig und man bräuchte nur noch die Daten einer anderen Stadt dahinter zu setzen. Leider sind aber viele Teile des Programmcodes und unseres Datenmodells stark auf München angepasst. Dazu kommen andere technische Probleme, so ist das Framework, die technische Grundlage der Webseite, veraltet, und es fehlt an Tests zur Qualitätssicherung und der Code hat sehr viele Altlasten. Unserer Oberfläche schadet das glücklicherweise nicht, aber wir können sie dadurch nicht einfach auf andere Städte übertragen. Außerdem verhindern diese Altlasten langfristig eine vernünftige Weiterentwicklung. Deshalb hatte ich schon länger die Idee, das ganze mit den bisherigen Erfahrungen komplett neu zu schreiben. Mit einem normalen Alltag hat man nur leider nicht die Zeit dafür, und so blieb das lange nichts als eine schöne Idee. Das hat sich jedoch durch den Prototype Fund geändert. Der Prototype Fund</a> ist eine Initiative der Open Knowledge Foundation Deutschland</a>, bei der sich Projekte um eine Förderung von (damals) bis zu 30.000€ Euro über 6 Monate bewerben können. Das Geld dazu kommt vom Bundesministerium für Bildung und Forschung. Bei der zweiten Runde im März 2017 habe ich mich mit der Idee beworben</a>, ein neues nutzerfreundliches Ratsinformationssystem zu entwickeln, Arbeitstitel "Open Source Ratsinformationssystem". Dieses soll sich im Gegensatz zu München Transparent in ganz Deutschland einsetzen lässt und auf einem soliden technischen Unterbau stehen. Daraus ist mittlerweile das Projekt "Meine Stadt Transparent" geworden, an dem ich zusammen mit Tobias Hößl seit fast vier Monaten arbeite. Darüber, wie es zur Zeit bei dem Projekt steht und was noch geplant ist, werde ich im zweiten Teil schreiben. Die Zahlen stammen aus der Datenbank von München Transparent ↩</a> </li> Es gibt natürlich auch noch andere Projekte, die erfolgreich ihr Ratsinformationssystem bereichern, wie z.B. Dresden Ratsinfo</a>. Ich meine in diesem Fall aber nur solche, die eine weitestgehend vollständige, bürgernutzbare Oberfläche bauen. ↩</a> </li> OpenRuhr – offene Daten für das Ruhrgebiet</a> ↩</a> </li> In Anträgen und Anfragen wird oft auf andere Dokumente bezug genommen. Dabei wird dann auch gerne München Transparent verlinkt, wie eine Meta-Suche</a> auf München Transparent zeigt. ↩</a> </li> Dank eines Stadtratsantrags</a> hat diese Zusammenarbeit auch ein politisches Mandat. ↩</a> </li> </ol> </section>