Charm healthchecks - any suggestions?

framework
1

I’ve been wondering what we could do to assist the charm ecosystem. If one visits the charm store today, it can feel like visiting a ghost town. It’s impossible to know what’s reliable. We know that experienced community members stick to charms/bundles that are maintained by Canonical, e.g. anything under the banner of Charmed OpenStack and Charmed Kubernetes.

The conventional approach is to add ratings. But rating systems are flawed.

I’m not convinced that the move the CharmHub will address this issue. In fact, I believe that removing namespacing will make things worse. Today, I know that if I see a charm written by someone I trust, I can probably trust their next charm.

We don’t have infinite resources… but perhaps we can brainstorm some ideas? How practical would it be to set up a GitHub repository that deploys models via, for example, GitHub Actions?

As someone deploying charms, I want to know that the juju deploy command works and, with the requisite relations/config in place, the application gets to the ready state without error. This should be as simple as a shell script.

We don’t need a perfect solution, just a gradual improvement from where we are now.

Any thoughts? @jamesbeedy, @ec0, @erik-lonroth, @magicaltrout, @chris.macnaughton, @szeestraten

4 Likes
2

Some things that matters:

  • last update not to far in history
  • supports a few distros
  • has a good documentation with guides for deployment.
  • has documentation for all actions
  • has a single repo in git hub rather than two, one in bzr and a similar in gh.
  • has a nice icon
  • info on supported clouds
  • info on system requirements
  • good info on all configuration options

If this could be evaluated partially automatically as part of the charm push, the developer could get rated based on these criteria…

Combine that with user added ratings and it would be interesting

2 Likes
3

To expand on @erik-lonroth’s ideas, it could be interesting to have documentation / links to the CI / test process for charms. As an example, the OpenStack charms could refer to the upstream review / CI process in documentation, or even to our CI.

With smaller (charming) organizations, the charm page could contain links to the charm’s TravisCI to allow users to see what is tested, and how.

2 Likes
4

the move the CharmHub

There are some features that might help.

The store already features “verified” publishers, who might have a recognizable name, and get a green check mark next to their name, e.g. Mozilla: Install Firefox for Linux using the Snap Store | Snapcraft

In addition, we plan for the charm details page to have a section listing “other charms or bundles from the same publisher.”

I recognize these don’t solve all worries, but help somewhat.

3 Likes
5

Thanks for posting @tartley. I was secretly hoping that a member or two from the implementation team would be tempted to post.

2 Likes
6

Also! The store features categories (comparable to “tags” in the current charm world). Within each category, there is a curated selection of “featured” snaps. These are the dozen or so most popular and well-maintained snaps within that category, and are quite visible on the store website by clicking a category name. I don’t know how carefully they are curated, but certainly updates to the selection occur every couple of weeks or so.

7

Thanx @tartley.

I do however want to emphasize the value of adding in means to encourage community members to work up quality on their charms.

This can be done through a reward system / gameification where charms with some evaluated score system gets promoted and the author might even be rewarded with perks and badges.

I realize this takes work, but it would help grow the community and raise quality at the same time.

This had been discussed alot in various threads during the last year.

4 Likes
8

I think that we should be more proactive on removing or deprecating charms which use out of support distro releases.

For example, do a search on “precise”:

Or “quantal”

or “Artful”

The only downside to doing this is that it will discourage developers from developing for non-LTS releases, but on the upside, we get less zombie charms on the store.

I also think charms should at minimum support a currently supported LTS release as well, since, for example, charms like Mediawiki get old and forgotten, destined to stay on Trusty forever. Those charms should be updated or deprecated. It might be a lot of work to support the latest LTS, say, but I would have a lot more confidence in charms if I saw it support Xenial, Bionic or Focal, over Precise or Trusty.

Maybe we have a CI system that force overrides the series attribute on deploy to the latest LTS, and developers get pinged if the tests fail, and if it works, the developer gets pinged a friendly message to say consider adding the latest LTS release to the supported series in yaml.

The goal of all of this would be to have a tight collection of charms on the charmstore which are high quality and run on recent Ubuntu series, over questionable quality on older series with lots of quantity.

1 Like
9

Really useful input @mruffell.

Over time, the goal should be to migrate from an Ubuntu-only mindset. When Juju plays nicely with others, it will become increasingly considered by others.

1 Like
10

@timClicks have a look here on the example ‘QA’ of a Nextcloud community plugin.

Features:

  • Last updated
  • Review
  • Instruction
  • Donations/get payed/contracts
  • Contact, docs and links.

https://apps.nextcloud.com/apps/phonetrack

… on the topic of community engagement, quality process of charms in charmstore.

1 Like
11

This is an excellent feature.

1 Like
12

It really is. Allowing for income of work, is so good because its benefitial for the individual community member, encourages quality and without nessecarily conflicting with the larger community if implemented right.

I would love to see a placeholder for, let’s say, allowing for payments or donations in established crypto currency.

Come to think of it, I will add that to all my charms in the future =D

13

As far as providing an indication of how “healthy” a charm is, I feel some of the existing suggestions here, if automated, could combine to form an automated “charm grade” that could be calculated and published (even something as simple as a traffic light rating).

Summarising aboive, and adding some additional thoughts, from my perspective, the following (which could be useful factors to make up an automatic “charm health” rating) would be -

  • How recently the latest charm version was published
  • Average time between releases
  • Does it support the latest LTS? This is a big indicator of freshness for me.
  • Does it have a README, and is it over 1K in size?
  • If we can get a good linkage between the charm store and sites like GitHub and GitLab, etc, does it have a link to a valid repo?
  • Does it have tests (this could look for travis, GitLab, buildkite, etc. metadata)
  • Does it provide relations for its services
2 Likes
14

I have been wondering whether the author could suggest bundle(s) that are expected to work. If the bundle can be deployed within LXD, the relation(s) provided by the charm are given some form of green light.

15

I think this is a good idea, and could also take the form of the charm having functional testing. A lot of the charms I work on have a sample bundle as part of the tests which can be tested for each release, and those tests passing would be a good indicator of health. In many cases where functional testing is present, it is trivial to add extra charms to the deployed Juju model during functional testing, and via python-libjuju, orchestrate the relation of the charm under test and external services such as haproxy or MySQL - and validate the result as part of the test run.

1 Like