Measuring Localization Time (in CI)

As we all know, measuring things is a good way to get concrete information. Now that Firefox CI is fully on Taskcluster this was a good opportunity to measure and see what we can learn about timing in localization tasks.

The code that generated the following charts and data is viewable in my Github repo, though as of this writing the code is rough and was modified manually to help generate the related graphs in various configurations. I’m working on adjusting them to be more general purpose and have better actual testing around logic.

One of the first things I looked at was per-task duration for the actual nightly-l10n task on beta/release by Gecko version per platform.  [Note: Android is missing on these graphs because we do not do Android Single Locale on Beta/Release anymore, so we don’t have metrics to compare] (graphs and takeaways after the jump)

Continue reading

Posted in Mozilla | Leave a comment

Find our footing on python best practices, of yesteryear.

In the beginning there was fire buildbot. This was Wed, 13 Feb 2008 for the first commit in the repository buildbot-configs.

For context, at this time:

In picking buildbot as our tool we were improving vastly on the decade old technology we had at the time (tinderbox) which was also written in oft-confusing and not-as-shiny perl (we love to hate it now, but it was a good language) [see relevant: image of then-new cutting edge technology but strung together in clunky ways]

As such, we at Mozilla Release Engineering, while just starting to realize the benefits of CI for tests in our main products (like Firefox), were not accustomed to it.

We were writing our buildbot-related code in 3 main repositories at the time (buildbot-configs, buildbotcustom, and tools) all of which we still use today.

Fast forward 5 years and you would have seen a some common antipatterns in large codebases… (over 203k lines of code! ) It was hard to even read most code, let alone hack on it. Each patch was requiring lots of headspace. And we would consistently break things with patches that were not well tested. (even when we tried)

It was at a workweek here in 2013 that catlee got our group agreement on trying to improve that situation by continually running autopep8 over the codebase until there was no (or few) changes with each pass.

Thus began our first, attempt, at bringing our processes to what we call our modern practices.

This reduced, in buildbotcustom and tools alone our pep8 error rate from ~7,139 to ~1,999. (In contrast our current rate for those two repos is ~1485).

(NOTE: This is a good contributor piece, to drive pep8 errors/warnings down to 0 for any of our repos, such as these. We can then make our current tests fail if pep8 fails. Though newer repos started with pep8 compliance, older ones did not. See List of Repositories to pick some if you want to try. — Its not glorious work, but makes everyone more productive once its done.)

The one agreement we decided where pep8 wasn’t for us was line length, we have had many cases where a single line (or even url) barely fits in 80 characters for legit reasons, and felt that arbitrarily limiting variable names or depth just to satisfy that restriction was going to reduce readability. Therefore we generally use –max-line-length of ~159 when validating against pep8. (The above numbers do not account for –max-line-length)

Around this time we had also setup an internal only jenkins instance as a test for validating at least pep8 and its trends, we have since found jenkins to not be suitable for what we wanted.

Stay tuned to this blog for more history and how we arrived at some best practices that most don’t take for granted these days.

Posted in Mozilla | 2 Comments

Release Engineering does a lot…

Hey Everyone,

I spent a few minutes a week over the last month or two working on compiling a list of Release Engineering work areas. Included in that list is identifying which repositories we “own” and work in, as well as where these repositories are mirrored. (We have copies in hg.m.o git.m.o and github, some exclusively in their home).

While we transition to a more uniform and modern design style and philosphy.

My major takeaway here is we have A LOT of things that we do. (this list is explicitly excluding repositories that are obsolete and unused)

So without further ado, I present our page ReleaseEngineering/Repositories

repositoriesYou’ll notice a few things about this, we have a column for Mirrors, and RoR (Repository of Record), “Committable Location” was requested by Hal and is explicitly for cases where “Where we consider our important location the RoR, it may not necessarily be where we allow commits to”

The other interesting thing is we have automatic population of travis and coveralls urls/status icons. This is for free using some magic wiki templates I did.

The other piece of note here, is the table is generated by a list of pages, using “SemanticMediaWiki” so the links to the repositories can be populated with things like “where are the docs” “what applications use this repo”, “who are suitable reviewers” etc. (all those are TODO on the releng side so far).

I’m hoping to be putting together a blog post at some point about how I chose to do much of this with mediawiki, however in the meantime should any team at Mozilla find this enticing and wish to have one for themselves, much of the work I did here can be easily replicated for your team, even if you don’t need/like the multiple repo location magic of our table. I can help get you setup to add your own repos to the mix.

Remember the only fields that are necessary is a repo name, the repo location, and owner(s). The last field can even be automatically filled in by a form on your page (see the end of Release Engineerings page for an example of that form)

Reach out to me on IRC or E-mail (information is on my mozillians profile) if you desire this for your team and we can talk. If you don’t have a need for your team, you can stare at all the stuff Releng is doing and remember to thank one of us next time you see us. (or inquire about what we do, point contributors our way, we’re a friendly group, I promise.)

Posted in Mozilla | 2 Comments

Firefox Launches Developer Editon (Minor Papercut Issues)

So, as you may have heard, Firefox is launching a dev edition.

This post does not attempt to elaborate on that specifically too much, but it’s more to identify some issues I hit in early testing and the solutions to them.


While I do admire the changes of the Developer Edition Theme, I’m a guy who likes to stick with “what I know” more than a drastic change like that. What I didn’t realize was that this is possible out of the box in developer edition.

After the Tour you get, you’ll want to open the Customize panel and then deselect “Use Firefox Developer Edition Theme” (see the following image — arrow added) and that will get you back to what you know.



As a longtime user, I had “Old Firefox Sync” enabled; this was the one that very few users enabled and even fewer used it across devices.

Firefox Developer Edition, however, creates a new profile (so you can use it alongside whatever Firefox version you want) and supports setting up only the “New” sync features. Due to creating a new profile, it also leaves you without history or saved passwords.

To sync my old profile with developer edition, I had to:

  1. Unlink my Desktop Firefox from old sync
  2. Unlink my Android Firefox from old sync
  3. Create a new sync account
  4. Link my old Firefox profile with new sync
  5. Link my Android with new sync
  6. Link Dev Edition with new sync
  7. Profit

Now other than steps 6 and 7 (yea, how DO I profit?) this is all covered quite well in a SuMo article on the subject. I will happily help guide people through this process, especially in the near future, as I’ve just gone through it!

(Special Thanks to Erik for helping to copy-edit this post)

Posted in Mozilla | 1 Comment

Keeping track of MQ patchsets…

Hey Everyone!

First some brief Background, Mozilla Releng has our code in a *lot* of repos, most being in Mercurial (a few other needs are in git or svn, but those are very rare relatively). I also do work for SeaMonkey which has needs with m-c, m-i, m-*, c-c, c-* etc. And needs with l10n

I personally manage all my patches with MQ. Which presents a problem for me, “keeping track of it all”. I used to try keeping open bugs, but thats hard with releng because while a bug may be open, we tend to have a good handful of patches attached to it, for various repos, and they need to land in certain orders sometimes.

Other ways I’ve tried to cope have been with landing as soon as the review comes in and avoiding writing patches for parts that need to land later until the first parts are landed/deployed. I found that method encompasses unneeded end-to-end times on bugs, and unnecessary context-switching.

To curb that I wrote a mozilla-build (bash) script [in ~/.bash_profile ] that sets an alias `patchset` that I run, and it works!

It especially works because I keep my code in /c/Sources/hg/* some repos are multi-levels deep, so this code could/should be improved or at least edited for your uses, but without further ado, this is how I manage my patchset (again note, all my work is in Mercurial, I do convert my stuff over to git/etc as needed though):

Continue reading

Posted in Mozilla | Leave a comment

Android Mobile Marketshare, and how we (Mozilla) stack up. – Part 1

So, I should preface this with a few big caveats!

  • I am not a metrics guy, nor do I pretend to be.
  • This post will not give you absolute numbers of users.
  • This post is not meant to show any sense of penetration into the market
  • Ignores all things Firefox OS for the purposes herein.
  • I present this as an attempt at data only and no pre-judging of where we should go, nor what we should do.
  • I am explicitly trying to avoid color commentary here, and allowing the reader to draw their own conclusions based on the static data.¹

What this series will attempt to show is:

  • Where the current marketshare is on Android OS’s (with citations where possible)
  • Where our (Firefox for Android) userbase is
  • Where we invest in builds/tests (due to length, this will be in a Part 2 — Will link from here once published)
  • How what we do in Release Engineering correlates to our known market (based on these stats). (also in Part 2)

Now to the juicy bits!

Google’s own stats on Android Marketshare

Currently Android has a pretty healthy marketshare on the later OS’s, and the earlier ones are seeing very very diminishing returns.

Android Usage Share Pie Chart

Android Usage Share on Dec 3, 2013, from

Version Codename API Distribution
2.2 Froyo 8 1.6%
2.3.3 –
Gingerbread 10 24.1%
3.2 Honeycomb 13 0.1%
4.0.3 –
Ice Cream Sandwich 15 18.6%
4.1.x Jelly Bean 16 37.4%
4.2.x 17 12.9%
4.3 18 4.2%
4.4 KitKat 19 1.1%

This data was all from so feel free to see updated data as of whenever you are reading this.

Stats from Google Play for “Firefox for Android”

Before we begin with the data, I need to clarify something readers may not be aware of at first glance, “What is an Install

Install in this context is any currently active device which has Firefox installed on it. It does not actually indicate frequent use.

Installs by OS – Release (org.mozilla.firefox)

Android_play_org_moz_firefox_redactedYes, you see that right, this is 81.4% of our GA users on some version of Android 4.0+.

Installs by OS – Beta (org.mozilla.firefox_beta)

Android_play_org_moz_firefox_beta_redactedOur beta audience is pretty similar with 83.59% on an Android 4.x or higher.

Installs by Chipset

Selecting by Chipset is a bit harder, since to do so we have to take a factor of how Releng does our Play Store releases (different buildID’s to factor to different chipsets). I am doing this by a feature of the play store, namely “Export as CSV” which gives buildID infoexport_as_csv

So with that in mind, here is the data:

Arm V7 Arm V6 x86
Firefox Beta 96.19% 0.90% 2.91%
Firefox GA 98.61% 1.39% N/A

The caveats to note is that I only gathered data from today’s Google Play installs, and I aggregated all installs over all versions, even ones that are multiple years old. We also do not have x86 released officially yet, so we only have beta users using that version.

Coming in Part 2:

  • Where does Mozilla Invest build and test resources for Android?
  • How does this compare to Mozilla’s Testing Infrastructure?

¹ – This post as-is is indeed intended to be data without analysis/commentary. I don’t feel I’m greatly suited for the latter compared to other people possibly reading. In Part 2 I intend to show some correlaries-as-data to what we are doing inside Release Engineering as it compares to our users at large. I’m currently hoping to also write it devoid of any assertions/commentary.
The underlying reasoning here is to help spur thoughts and commentary in others in order to further our mission using data, while at the same time without inserting my own opinions or biases into what I am presenting with this 2-part series. I do not yet know if I will do a commentary piece referencing these posts or not, and I may do so, I just don’t yet plan to.

Posted in Mozilla | Leave a comment

Mozillians… Where are your summit pictures?

Hey Everyone!

I hope your #mozsummit was even half as good as my own!

I am looking for links to all photos taken at summit locations, selfishly looking for ones of me, but while I’m doing this I had heard from IRC that there was a suggestion of “one big album” of pictures on a by-location basis.

So I’m offering to do a few things:

  • Gather said list of pictures, be it albums or individual pics
  • Inform anyone I personally recognize from each picture that they are in them
  • Publish as a group-of-links and/or a real photo-album somewhere (with credits)

What I would need from you:

  • Link(s) you have to any summit photo’s you have (or ones snapped by others if you have their permission to share with me)
  • Which summit location you were at
  • Any requirements you have on publishing (Creative Commons License)
  • Any requirements you have on your own name being publically linked to said pictures (I’ll avoid doing this unless I get a direct “yea link me”)
  • If your photos are stored privately, how I’m able to access them (password, become your FB friend, etc.)

How to tell me:

  • Email: or — please use include “Summit2013” somewhere in the subject
  • This blogs comments
  • Facebook Message (if you are friends with me on Facebook)
  • Twitter: @mozCallek
  • Google+ : (Sorry but I _never_ check this, so please use one of the above methods, I do have an account there so I can check an album on it if thats where your photos are)

With time permitting I’ll have some sort of nice-looking web app to let you find yourself/others, if not I may just correlate all these to an individual blog post and list.

It was wonderful meeting all you Mozillians, and I have never been so proud of us as a community

EDIT: Seems I wasn’t clear enough, you do not need to e-mail me full collections of images, a link to anyplace already existing is perfectly fine.

Posted in Mozilla | 8 Comments

SeaMonkey 2.18 – Where are you?

Hey Everyone,

So SeaMonkey 2.18 was supposed to be out today, so where is it?

We have had a hardware error in the systems that allow us to reliably generate the release — Without these systems anything we create will be of unknown quality/stability.

In order to meet our own quality and stability requirements we are NOT releasing SeaMonkey 2.18.

While there is a chance we could have these systems back up in time to do an intermediate release (say something corresponding to a possible Gecko 21.0.1) we can not promise nor plan for it at this time.

We are actively working on repairing the system and its data, once that is complete we will go forth with a new BETA based on the SeaMonkey 2.19 train, and we expect to release SeaMonkey 2.19 on time, on June 25’th.

We thank you for your understanding. where to buy domain .

Posted in Mozilla | 2 Comments

Welcome Back — Linux32 Tests!

As many of you know, last week we turned off many of our linux32 tests for desktop Firefox builds.

Yesterday, we were given the confirmation that we are now able to turn those tests back on, and we have done so. You should now see Fedora32 tests on all your Inbound, Central, and Try pushes, as well as any twig/project-branches.

Any questions/issues join us in #releng on

Posted in Mozilla | 1 Comment

Blogging More…. About Mobile?

Hello Planet, Long time no see.

Since our last conversation a lot has changed with me (and Mozilla!) I was a contractor since March, and just got my official offer to come on as a “real” employee. I will be detailing my journey in the community to a full time employee through a series of upcoming blog posts. (now don’t get me wrong, I’m still working on SeaMonkey in my free time too)

To also mix it up, I intend to throw in many blog posts about the state of our Mobile Infrastructure/Tooling, and where we intend to go from here. However I need ideas on posts! So to get me started, I’d like to enlist you, planet readers, to send me questions/topics you would like to learn more about when it comes to Mobile Automation (in Release Engineering) that I could blog about for you. History Lessons, Future Direction, Current Setup, Whatever!

Send your Topics/Questions to me at I cannot promise I will address every question/topic requested, but I can promise to try.

This work so far has been both challenging and a thrill!

Until Next Time!

Posted in Mozilla | Leave a comment