Don't take it for granted
There's currently over 1500 packages in our registries. And there are probably about 10k license files. I'm not joking. On average, per source tarball, I'm seeing between 5 and 30 license files as the average. There are some packages with over 2000 license declarations. The amount of legal text in Open Source Software is immense.
If you think that 99% of these are just 10-20 licenses, you're sorely mistaken, too. I've seen about a dozen variants of the GNU Public License due to combinations and exceptions.
Making sense of it
The right thing to do isn't the easy thing. The more complete you want to cover your record of the licenses used, the harder you are making it for yourself. So, there's a trade off to putting in more effort - you'll uncover more complex and bizarre license combinations if your scanner code improves. It's an unthankful and endless job - especially if you try to do this manually.
First pass: checksums
I've been here. I've written before about what we did in Clear Linux OS in a different post. Essentially we follow the logic path. Logic says that we should be able to identify direct copies of most licenses based on a file checksum. This is most generally true for lots of packages, because almost all of them have a LICENSE or COPYING file somewhere easy to find, usually top-level.
But alas, that's just the start of your troubles. There are plenty of these that are just slightly different because they include statements like Copyright 2026 by sofar <...> and now the checksum method fails.
Second pass: patterns
There's a rust crate called askalono that is specifically designed to recognize license texts and does this by pattern. Things like This file is licensed and Permission is hereby granted are great pattern triggers and askalono does a reasonably good job of spotting them. But it's not really very good at being precise. However, this method does allow us to really quickly spot what files are carrying a license declaration, and what files are not. And that matters: reducing the problem space quickly is good. We want that. Even if it means we are left with a large amount of work to do.
The third pass
Sadly, here's where the fun really stops. There's no way around it but someone just has to make a call. This is where it hurts: Every package can potentially mean hours and hours of reading time to go through all these licenses. And that's just untenable.
So, the manual labour needs an aid. First step is to try and mechanically match the license declaration to all the known SPDX license identifiers, and see which one scores best. This is still an automated thing that is deterministic - we just assign a score based on how much of the license is matched in the texts. It's an okay method. But already it starts to show problems and misses the bar on things like BSD 2 vs 3 clause.
So the next pass is to throw an LLM at it and give it the context and knowledge to determine the intent of the text and match it to an SPDX identifier, or better, an SPDX declaration - which can have multiple SPDX identifiers in a combination, using words like AND and OR and even WITH in case of special linking exceptions.
We trained our own model in Clear Linux OS - at least Arjan was the one who built this out. It was very good - especially given that this was, oh, 10+ years ago? I wrote before how this would make the chore of sifting through new license files manageable.
Nowadays we have plenty of options here, so license-api which is the component that implements license scanning, has an interface for any LLM, including local models to go and fetch outstanding license texts to review, refer to the SPDX identifiers and provide a license-opinion to the designer. These opinions can then be found in the dispatch web interface for the reviewer to either accept or change.
In the above screenshot, you can see the license opinion filed easily spotted the dual license of this file, and correctly determined that there are two licenses at play here. Neither should be omitted. The statement also reads that both licenses apply, not one OR the other, and thus the correct SPDX declaration must be an AND combination.
And now it works
Sure, it'll cost tokens. But it's doing a meaningful and low-token task that otherwise would make me have to fully read all these texts by hand, and then still have to find the proper reference files (right pane of screenshot) to match. Having the LLM present an opinion that I can visually bless or reject moves the cost around. My time is valuable. Doing this in a local small footprint model is notably doable today. Speed isn't a factor either. The model can just chug through the work list one at a time.
The agent that services these requests can just run as an isolated task with a permission token, talking over a socket. So the agent could entirely run in a container for isolation purposes if that is needed.
Anyone who wants to play with this extension and try it with local models once everything is released should come and talk and we can do some experiments with smaller models to drive this. I'd be interested in seeing how low it can go and still be saving review time.
--sofar (aka Auke)