in_pub #

Fork of unpub by Innim.

in_pub is a self-hosted private Dart Pub server for Enterprise, with a simple web interface to search and view packages information.

Screenshots #

Usage #

Command Line #

pub global activate in_pub
in_pub --database mongodb://localhost:27017/dart_pub # Replace this with production database uri

in_pub use mongodb as meta information store and file system as package(tarball) store by default.

Dart API is also available for further customization.

Dart API #

import 'package:mongo_dart/mongo_dart.dart';
import 'package:in_pub/in_pub.dart' as in_pub;

main(List<String> args) async {
  final db = Db('mongodb://localhost:27017/dart_pub');
  await db.open(); // make sure the MongoDB connection opened

  final app = in_pub.App(
    metaStore: in_pub.MongoStore(db),
    packageStore: in_pub.FileStore('./unpub-packages'),
  );

  final server = await app.serve('0.0.0.0', 4000);
  print('Serving at http://${server.address.host}:${server.port}');
}

Options #

Sign-in required #

With --auth, browsing the repository requires signing in through an OpenID Connect provider. Any spec-compliant provider works — nothing here is tied to a particular one; everything except the issuer url is read from the provider's discovery document.

By default only the web UI is affected: dart pub get and dart pub publish keep working exactly as before, so turning this on breaks nothing for existing consumers. Closing the pub client's side too is a second, separate step — see Making the pub client private too.

1. Register a client with your provider

Create a confidential client (one with a secret — this server keeps it, never the browser) using the authorization code flow, and note down its client id and client secret.

Two of these are worth being careful about.

The redirect uri must match byte for byte. It is derived from --auth-public-url by appending /auth/callback: with --auth-public-url https://pub.example.org the callback is https://pub.example.org/auth/callback. A mismatch in scheme, host, port, path or trailing slash makes the provider reject the sign-in.

--auth-public-url must name the root of a host or subdomain — a path prefix such as https://example.org/pub is refused at startup. The built web UI carries <base href="/">, so it was never actually reachable under a prefix; only the redirect uri honoured one, which made the setup look half-working rather than wrong.

offline_access is not optional. It is what makes the provider issue a refresh token, and the refresh token is the only way this server can keep asking whether an account still exists. Without one it refuses to create a session rather than quietly issuing one that would outlive a revocation. Some providers only issue refresh tokens when the scope is explicitly permitted for the client.

If the provider can restrict which claims a client sees, allow at least sub, email, name and groups. Group membership is read from the userinfo endpoint, not the id token, so it is enough for those claims to be available there.

2. Run the server

export INPUB_AUTH_CLIENT_SECRET=...   # not on the command line: argv is world-readable
export INPUB_AUTH_SESSION_SECRET=...  # a long random string; without it, restarting signs everyone out

in_pub --auth \
  --auth-issuer https://id.example.org \
  --auth-client-id in-pub \
  --auth-public-url https://pub.example.org \
  --auth-allowed-groups developers \
  --auth-admin-groups pubadmins \
  --auth-trusted-proxies 127.0.0.1

Two that are easy to miss:

• --auth-session-secret. Without it a random key is generated at startup, so every restart signs everyone out.
• --auth-trusted-proxies. Behind a reverse proxy, without it X-Forwarded-For is ignored and every session records the proxy's address. It is ignored on purpose when the peer is not listed: otherwise any client could dictate its own recorded address, and both the session binding and the audit trail would be fiction.

The remaining options are timings — session lifetime, rotation, revalidation intervals — and all have workable defaults. Run in_pub --help for the list.

What it does

• Sessions live on the server, so they can be ended: from the Sessions tab of your account for your own, and at /auth/admin for anyone else's if you are in an admin group. Blocking someone there is independent of the provider — it keeps them out of this repository while their account elsewhere is untouched.
• A stolen cookie is detected. The session secret is re-issued every few minutes; once the browser has used the new one, a request still carrying the old one means two clients hold the session, and it is ended for both. A client that merely missed a cookie update is handed a working one instead of being locked out.
• Revocation follows the provider. Every few minutes each signed-in user is re-checked by refreshing their grant and reading their profile. A disabled or deleted account, or one removed from --auth-allowed-groups, loses every session within --auth-revalidate-interval. A provider outage is not treated as a revocation — nobody is blocked for it — but after --auth-revalidate-hard unverified users stop being served, so downtime cannot be used to keep a revoked account alive. The same bound applies when it is this server that cannot ask — a rotated INPUB_AUTH_SESSION_SECRET leaves the stored provider credentials unreadable — except that there it is counted in attempts rather than elapsed time: tokens keep working for --auth-revalidate-max-failures checks, and then stop until the account signs in through the browser again, which is what restores this server's ability to check it.

Making the pub client private too

By default only the web UI is gated: dart pub get and dart pub publish keep working for anyone who can reach the server. --auth-protect-pub-api closes that as well, so package metadata and the tarballs themselves need a token.

Turn it on after handing out tokens — the moment it is on, every consumer that has not run dart pub token add stops resolving dependencies.

Getting a token. Sign in, open your account from the header, and switch to the Tokens tab. The value is shown once and never again; only a hash of it is kept. Then:

dart pub token add https://pub.example.org
# paste the token when prompted

For CI, keep it out of the process list and the shell history:

export PUB_TOKEN=...   # from your CI secret store
dart pub token add https://pub.example.org --env-var PUB_TOKEN

Two things pub is strict about:

• https is required. dart pub token add refuses a plain-http server (localhost aside), so private mode needs TLS in front.
• The url must match what is in the pubspec. pub sends the token only to the exact prefix it was registered under, so the hosted: url in every consuming pubspec.yaml, --auth-public-url and the url given to dart pub token add all have to agree — scheme, host, port and path.

Tokens for automation. A personal token belongs to the person who made it and stops working the moment their account is blocked or disabled upstream — which is usually what you want, and a nasty surprise for a release pipeline when they leave. Administrators can issue a service token on the same page: it belongs to no account and survives staff changes, at the cost of being revocable only here. Give each one a name that says where it runs.

Publishing. Uploads have always authenticated; they now accept a token from this server as well as the original Google credential. The Google one authorises a publish only — it says nothing about group membership, and until its owner has signed in here it corresponds to no account on this server, so it cannot be used to read private packages. Whichever is used, the address it resolves to is what gets recorded as the package's uploader.

Every identity here has to be a real email address. An account's address, a service token's, and each entry on a package's uploader list are all held to the same rule: an @, and a domain with at least one dot in it. A bare name is refused, and so is a single-label intranet domain — dart pub uploader add ops@intranet does not work, and neither does a service token carrying ci@internal. That is deliberate: an uploader entry is an identity matched against publishing credentials, and a value nothing can deliver to is one no person can hold.

It is worth checking before you turn --auth-protect-pub-api on. If your provider reports dotless addresses — some AD and LDAP directories do — the people it reports them for cannot create a token at all, because the token would publish as something that is not an address, and with the pub API closed a token is the only credential there is. Put a real address on those directory entries.

One limit applies to a Google credential that corresponds to no account here: with --auth on it cannot create a new package. Publishing further versions of a package it already uploads is unaffected — that is bounded by the package's uploader list — but a new name is bounded by nothing, and a repository whose owner asked for authentication should not accept one from anybody holding any Google account. Sign in through the web interface once, or publish with a token from it.

The unpub_auth tool in this repository is that older path, and is now legacy: it obtains a Google credential, which still works — including against a server running without --auth at all, which is what it was written for. Nothing has to be migrated in a hurry, but new setups should use the tokens above. Once everyone has, turn the old path off with --no-google-auth.

When sign-in does not work

Start the server with --verbose. It then logs the whole exchange with the provider — the discovery document and the endpoints it named, every request to the token endpoint with its parameters and chosen client authentication, the provider's own error responses verbatim, and which claims userinfo returned. Secrets, tokens and authorization codes are never printed; the configured client secret is only ever described (its length, whether it has stray whitespace).

The startup summary alone catches most of it: it prints the exact redirect uri being sent, the scopes being requested, and whether the provider could be reached at all.

Common answers, and what they mean:

Trying it locally

Authentication cannot be exercised through make dev-web at all: it serves the UI from a different origin than the API, and the browser client does not send cookies cross-origin, so every request arrives unauthenticated. Adding that origin to --auth-dev-origins does not change this — it decides who may read what needs no session, not whether cookies are sent. Test against a server built with make build, over http, with --auth-insecure-cookie and --auth-public-url http://localhost:4000.

API documentation #

in_pub can generate dartdoc API documentation for hosted packages on demand and serve it at /documentation/<name>/<version>/ (this is the "API reference" link on each package page). The first request for a version extracts its tarball, runs dart pub get + dart doc, caches the result under the store directory, and serves it; later requests come straight from the cache. While docs are being built for the first time a progress page is shown that opens the documentation automatically once it is ready.

Enable it by passing a docStore:

final app = in_pub.App(
  metaStore: in_pub.MongoStore(db),
  packageStore: in_pub.FileStore('./unpub-packages'),
  docStore: in_pub.DocStore('./unpub-docs'),
);

Notes:

• With --auth on, documentation needs a credential — a browser session or a bearer token — whether or not --auth-protect-pub-api is passed. A token lets a CI job or a docs mirror read it without a browser.
• A Dart SDK must be available on the server host (to run dart pub get and dart doc). The executable defaults to dart on PATH; override it with in_pub.DocStore('./unpub-docs', dartExecutable: '/path/to/dart').
• To document packages that depend on the Flutter SDK (sdk: flutter), the configured Dart must be the one bundled with Flutter (<flutter>/bin/cache/dart-sdk/bin/dart), not a standalone Dart SDK. A standalone dart cannot locate the Flutter SDK, so dart pub get fails with "Flutter users should use flutter pub instead of dart pub" and dart doc cannot resolve dart:ui. Flutter's bundled Dart handles both Flutter and plain Dart packages, so pointing dartExecutable at it (or running the server on a host/image that has Flutter on PATH) is the simplest setup.
• Generation resolves the package's dependencies, so the host needs network access and reachability of any private dependencies hosted on this server.
• If docStore is not configured the /documentation/... route is disabled and the "API reference" link is hidden in the web UI.

From the command line docs are enabled by default; use --no-docs to disable them and --dart-executable to point at a specific Dart SDK.

Health check #

GET /health is the endpoint to point a monitor at. It answers 200 while the server can reach its metadata store and 503 when it cannot, so a check that reads nothing but the status code is already right. The answer is never cached.

{ "status": "ok", "checks": { "database": { "status": "ok", "latencyMs": 4 } } }

The check is a real query, not a look at the driver's connection state — a database that has stopped answering keeps a socket that still calls itself open. A store that does not answer within 5 seconds counts as unreachable, and the failing check names the error by type; the full error, connection string and all, goes to the server log.

{
  "status": "error",
  "checks": { "database": { "status": "error", "error": "TimeoutException" } }
}

The route is public and stays that way with --auth on, so that a monitor needs no credential. It is the same answer for every caller: nothing in it describes what this repository holds. A package count would have been a number anybody could poll every few seconds, and a count read on a schedule is a publication feed for a repository whose whole point is that outsiders cannot watch one.

Usage behind reverse-proxy #

Using in_pub behind reverse proxy(nginx or another), ensure you have necessary headers

proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

# Workaround for: 
# Asynchronous error HttpException: 
# Trying to set 'Transfer-Encoding: Chunked' on HTTP 1.0 headers
proxy_http_version 1.1;

Moving the repository to https #

Authentication needs https — dart pub token add refuses a plain-http url — but pub treats a package's repository url as part of its identity, so a version published years ago carrying

dependencies:
  innim_lib:
    hosted: http://pub.example.org

asks for a different innim_lib than an application that asks for the same package on https://pub.example.org. The solver reports the two as an unsatisfiable conflict rather than as one package, which would leave only three ways forward: republish every old version, override them in every consumer, or serve the old metadata under the new address.

The server can do the third, when asked. Switched on, it looks at every repository API answer it sends over https: a dependency naming the same address over plain http is served as https instead — and it does the same to the pubspec.yaml inside the archive on its way out.

Both halves are needed, and the second is not obvious. dart pub reads a hosted package's dependencies from the version listing only while that package is not yet in the local cache; once it has been extracted into $PUB_CACHE/hosted/<host>/<package>-<version>/, every later solve reads that copy instead. Rewriting the metadata alone therefore fixes exactly one pub get per cleared cache and then the conflict comes back.

dart pub global run in_pub --proxy-origin https://pub.example.org \
  --legacy-hosted-url-rewrite

It is off by default, and deliberately so: it makes the repository answer with something other than what was published, and it reports each rewrite at FINE, where the default log level shows nothing. On by default, a mistaken rewrite would be silent; off by default, a missing one is a pub get that fails with the conflict above and a README section naming the flag. A repository that never moved has nothing to gain from it either way.

With it on, --verbose shows what is being changed:

Pub compatibility rewrite: package=innim_iap_google_apple version=1.0.0
dependency=innim_lib from=http://pub.example.org to=https://pub.example.org

Nothing stored changes: the archive on disk keeps the pubspec.yaml it was published with, no version number moves, and switching the layer off restores the previous answers exactly — there is nothing to migrate back. Inside the archive only that one url differs; comments, quoting, key order, file modes and timestamps are all carried through, and the output is a deterministic function of the input, so the bytes a client receives do not move between requests or restarts. It also does not excuse a bad publish: new versions should name the https address, and one that already does is served untouched, without being unpacked at all.

What consumers have to do once #

Because the archive a client downloads now differs from the one it downloaded before, its content hash differs too, and a cache already holding the old copy is not repaired — pub does not re-download what it already has. Each consumer needs one:

dart pub cache clean

The next pub get reports that the hash in pubspec.lock is out of date, updates it, and succeeds. pubspec.lock does not need deleting, and it does not have to be done again.

Only this repository's own address is rewritten. A dependency on another hosted repository, on git, path or an sdk, is served as published, and the match is on the parsed address rather than on the text, so http://pub.example.org.attacker.test is a different host and stays one.

The address rewritten to is the one the server is answering on, which behind a TLS-terminating proxy means --proxy-origin has to be set — the same setting the archive urls already depend on. Without it the server sees its own plain-http address, finds no https counterpart, and rewrites nothing.

A package store that hands out its own download urls (supportsDownloadUrl, as an object store would) is a redirect: the bytes never pass through this server, so the pubspec inside cannot be rewritten and the layer can only fix the first resolution. The server says so, once per package, when it happens. The built-in FileStore streams through and is unaffected.

Rewriting happens on the way out, and this server holds no metadata cache, so there is nothing to invalidate. The pub client does keep one, at $PUB_CACHE/hosted/<host>/.cache/<package>-versions.json; a consumer that had already fetched a listing may need dart pub cache clean before the new answer is used.

Each distinct rewrite is reported once per run, not on every pub get.

Package validator #

Naming conflicts is a common issue for private registry. A reasonable solution is to add prefix to reduce conflict probability.

With uploadValidator you could check if uploaded package is valid.

var app = in_pub.App(
  // ...
  uploadValidator: (Map<String, dynamic> pubspec, String uploaderEmail) {
    // Only allow packages with some specified prefixes to be uploaded
    var prefix = 'my_awesome_prefix_';
    var name = pubspec['name'] as String;
    if (!name.startsWith(prefix)) {
      throw 'Package name should starts with $prefix';
    }

    // Also, you can check if uploader email is valid
    if (!uploaderEmail.endsWith('@your-company.com')) {
      throw 'Uploader email invalid';
    }
  }
);

Customize meta and package store #

in_pub is designed to be extensible. It is quite easy to customize your own meta store and package store.

import 'package:in_pub/in_pub.dart' as in_pub;

class MyAwesomeMetaStore extends in_pub.MetaStore {
  // Implement methods of MetaStore abstract class
  // ...
}

class MyAwesomePackageStore extends in_pub.PackageStore {
  // Implement methods of PackageStore abstract class
  // ...
}

// Then use it
var app = in_pub.App(
  metaStore: MyAwesomeMetaStore(),
  packageStore: MyAwesomePackageStore(),
);

Badges #

URL	Badge
/badge/v/{package_name}
/badge/d/{package_name}

Alternatives #

• pub-dev: Source code of pub.flutter-io.cn, which should be deployed at Google Cloud Platform.
• pub_server: An alpha version of pub server provided by Dart team.

Credits #

• pub-dev: Web page styles are mostly imported from https://pub.flutter-io.cn directly.
• shields: Badges generation.

License #

MIT